A recurring question on r/MachineLearning asks whether the papers that defined modern deep learning — the ones written between 2000 and 2021 — would survive peer review if they were submitted fresh today. It is not a nostalgia exercise. The answer tells you how fast the floor has risen, and that floor is the same one your production AI tools are now measured against.
We went back through what acceptance at NeurIPS and ICML actually required at three points in time, and the gap is wider than the "papers are longer now" complaint suggests.
What changed between 2014 and now
The headline number is volume. NeurIPS received fewer than 2,000 submissions in 2014. By the early 2020s it was taking in well over 12,000, and recent years have pushed past 15,000. ICML followed the same curve. Acceptance rates, meanwhile, barely moved — they have stayed in a roughly 20–26% band the entire time.
Hold those two facts together. The same slice of papers gets in, but the pool it is drawn from is seven to nine times larger. A paper that landed in the top 25% in 2014 is not competing against the same field in 2026.
The 2014 NeurIPS program committee ran an experiment that still matters here. They routed about 10% of submissions through two independent review committees and compared verdicts. The committees disagreed on the majority of the papers that either one chose to accept — a result close enough to a coin flip that it has been cited ever since as evidence of review noise. NeurIPS repeated the experiment in 2021 and the disagreement rate had not improved.
Here is the uncomfortable implication. If review is that noisy, a rising bar does not cleanly reject weak work. It widens the band where solid work gets turned away because one reviewer wanted one more experiment. The bar did not get sharper. It got higher and stayed blurry.
The four things reviewers now expect
Read a review from 2015 and a review from 2025 side by side and four demands separate them.
Ablations. A 2014 paper could introduce an architecture and show it beat a baseline. Today a reviewer expects you to remove each component and quantify what it contributed. "Does the method work?" became "which part of the method works, and by how much?"
Baselines. One comparison now reads as cherry-picking. Reviewers expect current strong methods, tuned with the same budget you gave your own model. A favorable comparison against a weak baseline is a near-automatic weakness flag.
Variance. Single-run results draw immediate fire. The expectation is multiple random seeds with error bars or confidence intervals. The 2018 paper "Deep Reinforcement Learning That Matters" made this concrete by showing RL results that flipped sign across seeds, and the lesson generalized well past RL.
Reproducibility. NeurIPS added a reproducibility checklist in 2019 and a broader-impact statement in 2020, later folded into a mandatory paper checklist. Releasing code moved from optional to effectively expected.
None of these four demands is wrong on its own. Ablations, strong baselines, and seed variance each make a result more trustworthy. The problem is cumulative: a paper that satisfies all four needs experiments a 2014 lab could not have afforded on its compute budget. The bar is not just higher — it is more expensive to clear.
So would a classic paper survive? The original GAN paper from 2014 ran about eight pages with a handful of experiments, and against the literal 2026 checklist it would look thin — few baselines, limited ablation, qualitative samples instead of quantified variance. AlexNet in 2012 was similar in scope. Both reshaped the field. Both would draw a "needs more experiments" review today.
That is not really a knock on modern review. It is a sign that an idea-dense, experiment-light paper now has nowhere to land, because the format has standardized around exhaustive empirical validation.
What this means for the tools you ship
The research bar is not an academic curiosity for a developer. It is the spec that production AI tooling gets held to.
When a paper claims a model is better, the field now asks for ablations, matched baselines, and variance. The same questions are reaching vendor benchmarks. "Our model scores higher" invites "higher than which baseline, tuned how, averaged over how many runs?" The skepticism that hardened in peer review is leaking into procurement and into the evaluation suites teams run before adopting a tool.
If you build or evaluate AI features, the practical move is to apply the reviewer's checklist before a reviewer — or a customer — applies it to you. Track which component of your system earns its keep. Keep an honest baseline. Run more than one seed. Treating your own work the way ICML treats a submission is now the cheap version of due diligence.
When you read a new model release, run the four-question filter: is there an ablation, are the baselines current and fairly tuned, are results averaged over seeds, and is there enough detail to reproduce the numbers? A release that fails three of the four is a marketing artifact, not a result.
Keeping that kind of structured record — papers, the claims they make, the baselines they were tested against, what actually reproduced — is its own small workflow problem.
The bar will keep rising. The honest read of the last decade is that it rose mostly by accumulating requirements, not by getting better at telling good ideas from bad ones. Build to the requirements — and keep your own judgment about which of them actually made the work better.
Originally published at pickuma.com. Subscribe to the RSS or follow @pickuma.bsky.social for new reviews.