Agentic verification improves long-horizon task performance over single rollouts

2026-10-05

VeriHarness increases model performance by using a generator as an agentic verifier to resolve disagreements and challenge consensus through environmental evidence and reusable skills.

March 25, 2025, marked the introduction of Gemini 2.5, which Google DeepMind describes as a thinking model. The experimental 2.5 Pro version currently holds the top spot on LMArena, which tracks human preferences. I work in AI and I build agentic AI systems, so I'm generally skeptical when "reasoning" is used as a marketing term for better post-training. It usually just means the model is better at mimicking the appearance of a logical chain.

The real friction in agentic systems isn't the initial generation, but the verification of long-horizon tasks. The VeriHarness paper addresses this by turning the LLM into a verifier equipped with a workspace and evidence tools. It uses a disagreement resolver to check competing claims and a consensus challenger to find omitted requirements. This is a pragmatic approach because consensus among multiple rollouts often hides errors rather than confirming truth.

The results from VeriHarness show that evidence-backed revision improves average performance over a single rollout by 6.4 points with Claude Opus 4.8 and 6.2 points with Gemini 3.5 Flash. These gains occurred across five long-horizon workspace benchmarks. I think the most interesting part is the claim that verification skills can self-improve from failure feedback. If a verifier can actually learn from its own mistakes without a human in the loop, we stop relying on static rubrics.

I don't know how these verification skills scale when the environmental evidence is ambiguous or contradictory. The paper mentions a pool of 26,000 rollouts produced at a cost of over $100,000, which is a significant spend just to establish a verification baseline. It suggests that the cost of ensuring an agent is correct is becoming a primary bottleneck in deployment.

Most "thinking" models are just doing internal chain-of-thought. VeriHarness is different because it pushes the verification into the environment. It doesn't just ask the model to think harder; it forces the model to find evidence. I'd argue that external verification is the only way to move past the plateau of self-correction where models just hallucinate a reason why their first wrong answer was actually right.

Google DeepMind plans to build thinking capabilities directly into all models in the Gemini 2.5 family. We are essentially trading generation speed for a higher probability of correctness, but we still lack a cheap, universal way to prove that correctness in real-time.