Let Confidence Change, Not the Prediction: Prediction-Preserving Repair for Post-hoc Calibration

arXiv · Daehwan Kim, Haejun Chung, Ikbeom Jang · 2026-09-05

Let Confidence Change, Not the Prediction: Prediction-Preserving Repair for Post-hoc Calibration fixes overconfidence in models without ruining their accuracy—finally, a calibration method that doesn’t trade one problem for another.

What it does

This paper introduces a post-hoc calibration method that adjusts model confidence scores *without* altering the underlying predictions, preserving accuracy while improving calibration. It leverages a lightweight repair mechanism that learns to adjust confidence scores based on the model’s internal representations, rather than relying on external datasets or retraining. The approach is agnostic to the base model and can be applied to any classifier. Experiments show significant improvements in expected calibration error (ECE) and other calibration metrics across multiple benchmarks.

Why it matters

For AI practitioners deploying models in the wild, overconfidence is a silent killer. It leads to brittle systems that fail unpredictably when confidence doesn’t match reality. This paper offers a practical solution: calibrate the confidence without touching the predictions. It’s a game-changer for agentic systems that rely on uncertainty estimates for decision-making.

How it applies

If you’re building systems that depend on calibrated confidence scores (e.g., retrieval-augmented generation, agent routing, or autonomous decision-making), this method can be dropped into your pipeline without retraining. It’s especially useful for black-box models where post-hoc adjustments are the only option.

Takeaways

• Post-hoc calibration can improve confidence scores *without* sacrificing prediction accuracy.
• The repair mechanism is model-agnostic and can be applied to any classifier.
• Lightweight and computationally efficient, making it suitable for production systems.
• Overconfidence is a systemic issue in deployed models—this paper gives us a scalpel, not a sledgehammer.
• Calibration isn’t just about metrics—it’s about trust in agentic systems.

Topics: calibration, uncertainty estimation, post-hoc methods, reliability, agentic systems