Beyond F1: Evaluating Coverage and Failure Recovery in AI Model Security Scanners

arXiv cs.AI · Qianlong Lan, Vinothini Pandurangan, Anuj Kaul, Indranil Sanyal · 2026-08-27

This is no longer on the current shelf — shelves rotate as new material clears the bar. The analysis below is unchanged. See what is featured now.

This paper tears down the myth that AI security scanners are as reliable as we think—exposing critical blind spots in coverage and failure recovery that could leave your models vulnerable.

What it does

The paper evaluates three popular AI model security scanners (ModelScan, ModelAudit, Fickling) using a controlled, artifact-backed benchmark with 170 synthetic Pickle and PyTorch artifacts across 145 specimen families. Unlike traditional metrics that focus on binary security judgments, it introduces a framework to assess *coverage* (how much of the attack surface is scanned) and *failure recovery* (how well the scanner recovers from errors). The study reveals that these tools often miss critical vulnerabilities or fail catastrophically, even when they report high accuracy.

Why it matters

For AI practitioners building production systems, this paper is a wake-up call: security scanners are not the silver bullet we’ve assumed. Blindly trusting them could expose your models to real-world exploits. It highlights the need for rigorous, artifact-backed evaluation of security tools—something that’s shockingly absent in the current ecosystem.

How it applies

If you’re using (or building) AI security scanners, this paper gives you a playbook to stress-test them. Replicate their benchmarking methodology to evaluate your own tools, and consider supplementing scanners with manual audits or ensemble approaches. For tool builders, it’s a roadmap to improve coverage and failure recovery—critical features for trustworthy AI systems.

The short version

Your AI security scanner is lying to you—and this paper proves it.

My take

Teams deploy scanners with 99% ‘accuracy’ scores, only to realize they’re missing entire classes of attacks because the tools weren’t designed for the artifacts they’re scanning. The security community has a habit of optimizing for false positives over false negatives—we’d rather flag 100 harmless files than miss one exploit. But in AI, a single missed vulnerability can cascade into a full-blown breach. This paper is a gut check: we need scanners that are *adversarially robust*, not just performant on clean data. And for builders? Build *your own* evaluation datasets. Don’t outsource your security posture to a vendor’s marketing.

How it connects

• This aligns with the broader trend of ‘shift-left’ security in AI: testing tools must evolve to meet the unique risks of model artifacts (e.g., weights, configs, data pipelines).
• It’s a cousin to the ‘reliability’ crisis in AI evals—where metrics like F1 hide critical failure modes in real-world deployment.
• Expect regulators and auditors to start demanding *artifact-specific* security proofs, not just generic ‘AI safety’ claims.

Bottom line

Audit your security scanners with artifact-backed benchmarks—then build redundancy into your pipeline. A single scanner is never enough.

Takeaways

• Security scanners for AI models (e.g., ModelScan, ModelAudit, Fickling) have critical blind spots in coverage and failure recovery, despite high reported accuracy.
• Artifact-backed benchmarks (like the 170 synthetic artifacts used here) are essential for evaluating security tools—don’t trust metrics alone.
• Failure recovery is a first-class concern: scanners must gracefully handle errors to avoid false negatives in production.
• The study’s benchmark is a reusable framework for auditing other security tools—publish it, fork it, or extend it.
• AI security is not just about detection; it’s about *trustworthy* detection. This paper forces us to confront that.

Topics: AI security, model safety, evaluation, scanners, robustness