Some random (and mildly fun) thoughts while helping with a recent conference:

Recent major AI conferences (NeurIPS, ICLR, CVPR) have all suffered from the issues of overwhelming submission volumes, and the reviewer pool inevitably expanded. The Organizers have then found themselves battling against LLM-generated reviews. ACs were asked to identify such irresponsible reviewers, authors were out-raged, and stronger rules were introduced to penalize them.

I started wondering: instead of battling against LLMs, can we instead fully embrace the idea of using LLMs to review papers? Can we spend our efforts on building a capable LLM reviewer, rather than continuing to review everything ourselves?

Imagine all the benefits of having a capable LLM review submissions. The workload would no longer be a concern. Turnaround time would be much shorter. Bias could be largely reduced. Of course, there would be issues to be addressed. But tackling those challenges feels far more interesting than reviewing papers one by one.

We should ask the team of SACs, ACs, and our top reviewers to spend time collectively building a capable LLM reviewer for the conference.

Major AI conferences have 100+ SACs and 1000+ ACs, all of them are top talents. If they are already contributing their time to the conference (for free!), we should make the best possible use of that effort!

Reviewing for a given year is largely a one-off effort. The know-how is not retained in a way that could be leveraged without relying on these experts. If, instead, we organized them each year to spend time improving the LLM reviewer, we would be building something long-lasting and potentially increasingly capable over time.

The exact implementation is not straightforward. As a baseline, we could ask each committee member to contribute an agent or model that represents their reviewing style and preferences. They could tune the model as much as they like to better reflect their opinions. The rebuttal process could provide valuable data for RLHF. We could also ask big tech to sponsor the GPU costs for this effort.

— arising from some involuntary procrastination during the review process 🐶