Solid evidence for claims about automated AI research has been mostly absent, the authors argue. Existing evaluations either test agents on narrow, verifiable tasks or submit AI-generated papers to peer review, a process the researchers call "overstretched, stochastic, and suffers from poor review quality."
A new way to test AI research ability
The researchers call their approach "Shadow Evaluation." An agent receives the core research question from an unpublished paper. The original authors, who spent months on the same question, then evaluate the result as conference reviewers would. Since the results aren't on the web yet, the agent can't fall back on training data.
The team partnered with the authors of two NeurIPS 2026 submissions. The first paper examines how personality traits of language models can be steered through their weights. The second develops a method called TabPFN that detects when a tabular prediction model hits deployment data that differs sharply from its training data and tanks its accuracy.
The main experiments used Claude Opus 4.8 with Extra-High Reasoning. Each agent got six days, $3,000 in API credits, a GPU budget, and full access to a virtual machine and the open web. The agent ran inside a scaffold, a software environment that orchestrates model calls and provides tools, letting the model delegate to subagents and monitor its own resource usage. It could also consult external AI review tools.
