Ai2 finds LLM novelty judges swing by over 50 points on small prompt tweaks

Allen Institute for AI has released Old Ideas, Novel Problems, a study and benchmark of how large language models judge whether a research idea is novel. Across six judges, Ai2 says changing a few words in a prompt can shift pairwise accuracy by more than 50 percentage points and sometimes below chance, and that retrieval and bigger reasoning budgets do not consistently help. In one GPT-5.4 test on human-versus-generated idea pairs, accuracy fell from 93.5% to 40.9% when the prompt stopped saying reviewers had found exactly one idea novel. Ai2 also says a cheap prompted Claude Opus 4.6 baseline beat two purpose-built novelty evaluators on its Human+Generated set. Code, data and a reusable pipeline are public.