Every AI-for-science system that claims to generate new ideas is ultimately graded by another language model asked the question “is this novel?” That grader — the novelty judge — has never been standardized, and a new study from the Allen Institute for AI suggests the grading itself is unstable enough to undermine published novelty gains.
The paper, Old Ideas, Novel Problems (arXiv 2610.02022), is a controlled study of judge design choices across six judges: GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6. The headline number is ugly: given the same idea pairs, one prompt told GPT-5.4 that reviewers had found exactly one idea novel, the other just asked which was novel. Accuracy fell from 93.5% to 40.9% — below chance — and the model changed its verdict on more than half the identical pairs.
Labels From Real Reviews
Rather than treating paper acceptance as a proxy for novelty, the authors mined ICLR 2026 peer reviews for passages where reviewers explicitly praised or disputed originality, keeping only papers with unanimous agreement at the extremes of their research area. That produced a 154-idea high-novelty pool and a 145-idea low-novelty pool (the Human-Only setting), plus a Human+Generated setting pairing those 154 human ideas with 154 ideas from a vanilla LLM generator given only a research area and an instruction to be novel. An annotator agreed with the mined sentiment polarity in 47 of 50 cases, and an expert reviewing 30 pairs preferred the human idea in every one.
Instability was worse on AI-generated ideas than human ones — the same prompt change moved Claude Opus 4.6’s Human-Only accuracy by 21 points, but the AI-generated setting swung far more. Comparing two generators (Claude Opus 4.5 vs Sonnet 4.5, GPT-5.4 vs GPT-5.1), 25–33% of verdicts flipped on average across prompt variants.
Buy More, Judge Worse
The counterintuitive part is that effort did not pay. An agentic judge given an extra-high reasoning budget and live iterative arXiv search scored 0.78 macro-F1, eleven points behind a plain Claude Opus 4.6 prompt at 0.89, while costing roughly 10x more per decision. And the cheapest setup won outright: a simple Claude Opus 4.6 prompt with low reasoning and majority voting over three calls hit 0.854 macro-F1 at $2.59, beating Idea Novelty Checker at 0.642 for $30.66 and Sakana AI’s AI Scientist novelty evaluator at 0.772 for $82.05.
The team ships the benchmark, dataset and code, plus a reusable pipeline for mining new peer-review sets as the literature grows.
Related: the paper on arXiv, the Novelty Judge Bench dataset and the evaluation code