What Machines Can Fix

One in three papers beating the median at a competitive workshop is remarkable for a system that did not exist two years ago — and the kind of result that will improve fast with better models and more compute. What interests Clune is not just that the system will improve. It is that it could do what human scientists consistently fail at.

“AI scientists do not have to bring our vices and our incentives to the lab bench with them,” he told me — meaning the human tendency to cut corners, bury inconvenient results and game the system for career advantage.

Clune ticked off the things an AI system would do better: pre-register hypotheses, report negative results, run the extra experiment that might undermine a clean finding. Human scientists skip these routinely — sometimes because career incentives punish the effort, but just as often because of limited time, limited funding, the sheer cognitive load of managing a research program and the simple fact that no individual can hold every methodological ideal in mind while also doing the science.

This is a clean and appealing argument. It is also, I think, slightly too simple. The deepest distortions in science come not from researchers who know they should pre-register but skip it. They come from subtler failures: choosing safe questions, following fashions, mistaking technical sophistication for insight. An AI trained on the existing literature would inherit those biases, not correct them.

What Could Go Wrong

Still, the opportunities are real — and so are the dangers. An AI iterating experiments without ethical constraints could pursue research no responsible scientist would touch.

“We do not want, for example, to do what some people are recklessly calling for, which is ask AI to go explore and discover and be curious and chase down anything that’s interesting,” Clune said, “because it might decide that it’s interesting to see if it can make COVID one hundred times more dangerous.” He argues these systems need the same kind of ethics and oversight we impose on human scientists — maybe even more.

A nearer risk is volume — and slop. If generating a paper costs a few dollars in compute rather than months of labor, the peer review system faces a flood it cannot absorb. Yamada noted that scientific conferences and journals have begun requiring AI-use disclosure, but the norms are being built in real time, with no consensus in sight.

Then there is the apprenticeship problem. If the tasks that train young scientists — designing experiments, analyzing data, writing up results — are handed to machines, the pipeline that produces the next generation could quietly break. Clune compared the situation to baseball’s farm leagues: institutions may need to fund training positions even when AI handles the actual work, “not because of the crowds they attract, but because eventually they’ll be the stars in Yankee Stadium.”

The analogy is appealing but fragile. Baseball teams fund farm leagues because every player is a potential revenue source. Universities do basic research for which there is no market — funded largely by government agencies like the National Science Foundation and the National Institutes of Health — and despite the tremendous contribution of that research to economic prosperity, no one has articulated how the funding model adapts when the apprentices are no longer needed.

What Science Is For

Carl Boettiger, a professor of environmental science at UC Berkeley, thinks the deeper issue is that the AI Scientist takes the wrong thing for granted. “Too much of the infrastructure being put into the design is around the process of science as it is today — peer review, writing papers,” he told me. “Too little is around the hard work of the science.”

This strikes me as the sharpest critique of the project — and it cuts deeper because the AI Scientist was built for machine learning research, a field where innovations actually do compound. In computer science, a useful result becomes a software dependency: thousands of developers import it and build on it the next day. In most other sciences, a useful result becomes a citation. Citations are rhetorical; dependencies are functional. The AI Scientist automates paper production, but in the fields where papers are the primary output, papers are not the bottleneck holding science back.

Clune and Boettiger differ on what science is for. Clune frames it as utilitarian calculation: “People are saying goodbye to their children right now on a hospital gurney because their child is dying of cancer or some other horrible disease.” If automating research cures diseases faster, the trade-offs are worth it. Boettiger frames it in human terms: “The number one thing we do is we train other human beings. And we sometimes do science more slowly — we produce papers more slowly to do that.”

Both are right, and what they are right about does not resolve the tension. The AI Scientist has no career to protect, no incentive to hide a failed experiment. It also has no stake in the outcome and no way to teach a student why a result matters. As Boettiger put it: “The future of AI in research will be built by us, the research community.” The tools are here. The question is whether scientists will shape what comes next or be shaped by it.