Internal probes outperform text monitors in the tested deception evaluations
Caught in the Act reports high AUROC under specific tasks and access conditions, not a universal replacement for other monitoring.
Caught in the Act reports 98.8% AUROC on SHADE-Arena and 99.7% in its introspective setting. These are discrimination metrics for the paper’s evaluated deception conditions. They are not direct percentages of all malicious behavior caught in a deployed fleet.
Internal probes require the relevant model-state access. Text monitors use observable outputs and can infer suspicious behavior without an explicit admission of intent. The original article’s claim that a text monitor must wait for a confession misdescribed that comparison.
For adoption, first establish whether internal activations are available. Then compare operating points, false alarms and missed cases on the intended task. A high AUROC can motivate a probe experiment without establishing that monitoring is the only scalable defense or that a provider API exposes the required signals.
Editorial correction (2026-10-13 UTC): Restored the evaluation/access scope and removed the text-monitor strawman.