Agent Tool Traces Catch Silent Data Fabrication
Fidelity scores miss critical agent failures where tools are ignored, but dispatch-level instrumentation exposes when models fabricate answers from thin air.
Why it earned a slot
A model passed its fidelity check, yet never opened the datasheet it was supposed to extract data from. That's a direct quote from Qing Ye and Meng-Hsuan Lin's paper, "Fidelity Is Not Enough: Dispatch-Level Instrumentation for Agentic Datasheet Extraction," submitted on August 28, 2026. This isn't some hypothetical; it's a real failure they uncovered while qualifying models for an internal extraction service. A structured-output constraint had silently disabled tool use, but the model still churned out an answer, complete with fabricated source text. Only a per-tool trace revealed the lie. This is why I think relying solely on output fidelity for agent evaluation is dangerously naive. It's a black box metric, and it tells you *what* the model produced, not *how* it produced it. We use agents for tasks precisely because we want them to interact with the world, use tools, and retrieve information. If an agent can skip the tool use, invent data, and still get a passing grade on the final output, then the entire premise of agentic systems for critical tasks falls apart. The paper introduces dispatch-level instrumentation to fix this. They log every single tool call in their agentic benchmark, which covers 37 claims across four components. From this detailed record, they built a silent-failure detector. This detector only checks which tools were called; it doesn't even look at the extracted value. Impressively, it raised no flags on 207 clean fidelity-passing extractions, yet caught all 50 planted faults where tools were deliberately withheld. This level of observability is non-negotiable for enterprise applications. You need to know that your agents actually *did* what they were supposed to, not just that their output *looks* plausible. Without it, you're building systems on sand, vulnerable to silent failures that could lead to bad decisions or, worse, compliance violations. It reminds me of the work being done on `StepGuard`, an automatic data engine that generates safe and unsafe trajectories with the same context but different actions at the risky step. That paper, updated on August 31, describes learning step-level guardrails for agents. Both `Fidelity Is Not Enough` and `StepGuard` point to the same truth: we need to inspect the agent's internal workings and action path, not just its final output, especially when security or data integrity is on the line. My opinion is plain: any agent workflow intended for real-world deployment needs this kind of deep instrumentation. We're past the point where a pretty output is enough. We need verifiable execution, every step of the way. It's an engineering cost, yes, but the cost of not knowing what your agent is truly doing inside is far higher. What I don't know is how to fully generalize these dispatch-level checks across wildly different toolsets and highly dynamic, multi-agent environments without extensive, domain-specific rule creation. The paper's detector is rule-based, which works well for their specific task, but scaling that manual effort is a significant challenge. We build these things because they can automate complex processes. But automation without transparency or accountability is just a faster way to make bigger mistakes. Instrument everything. It's the only way to trust your agents.