Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints

arXiv cs.AI · Haoyaun Zhu, Jie Zhang · 2026-09-03

This preregistered study is a bombshell: LLM-as-judge evaluations on shared endpoints show Spearman correlation of only 0.400 on repeat attempts, fundamentally undermining the reliability of every leaderboard and eval pipeline that depends on them.

What it does

The authors conducted two preregistered audit campaigns on shared LLM evaluation endpoints, testing the fundamental assumption that identical requests produce consistent judgments. Across 52,988 audited request attempts, same-window repeat rankings agreed at only Spearman 0.400—far below the threshold needed for reliable measurement. They systematically tested whether the same request sent to the same model name produces the same output, and found that it does not, even when controlling for prompt, model, and timing.

Why it matters

This paper strikes at the heart of the LLM evaluation ecosystem. Model judges now gate training data selection, score generations, and drive leaderboards—yet the foundational assumption that these judges are reliable measurement instruments has never been rigorously tested in this way. If a judge gives different rankings for the same request, then every leaderboard, every automated eval pipeline, and every training data filtering system built on LLM judges is built on sand. This isn't a minor calibration issue; it's a fundamental reliability failure.

How it applies

Any practitioner using LLM-as-judge for evaluation, ranking, or data curation needs to immediately audit their evaluation pipelines for this instability. The paper's methodology—preregistered campaigns with fixed thresholds—is a template for how to properly audit eval systems. Practitioners should consider implementing deterministic evaluation protocols, using multiple judges with aggregation, or moving to more stable evaluation methods when consistency matters. This directly impacts anyone building RAG systems, automated benchmarks, or leaderboard-driven model selection.

Takeaways

• LLM-as-judge evaluations on shared endpoints are fundamentally unreliable, with Spearman correlation of only 0.400 on repeat attempts.
• The assumption that 'same request, same model name, same answer' is broken—shared endpoints introduce variability that undermines measurement.
• Every leaderboard, eval pipeline, and training data filter built on LLM judges needs to be re-evaluated in light of this instability.
• Preregistered auditing with fixed thresholds is the gold standard for validating evaluation instruments—most current eval systems fail this basic test.
• The paper doesn't just identify a problem; it provides a methodology for auditing eval reliability that should become standard practice.

Topics: LLM-evaluation, reliability, leaderboards, model-judges, eval-pipelines