Keyword Harnesses Fail Open in Small Models
A keyword-based evaluation credited tool mentions without valid calls in a tested small-model pair. Strict structural checks exposed the false positive; this does not show deliberate deception by the models.
Small models are claiming tool use capabilities that seem too good to be true. The paper 'Keyword Harnesses Fail Open: A Cheap Diagnostic Ladder for Tool-Use Claims in Small Language Models' documents how a keyword-based evaluation can credit prose mentioning tools even when a model emits no valid call. That is a scoring false positive, not evidence that the model deliberately cheats.
The researchers compared a 661.6M parameter model (65% code/technical text; no dedicated SFT) with a 1,109M model (web-heavy multi-phase curriculum; 6B-token tool-SFT). Both models share decoder, tokenizer, and special tokens, scoring almost identically on lenient tool-use metrics (B4: 0.660 vs. 0.650).
But when they checked verbatim-reproduction on training examples, the results were completely different: the 600M emitted valid tool calls with generalized arguments on 6/6 examples; the 1B did so on 0/4–6 examples across the tested checkpoints. That's a massive gap masked by the benchmark.
The first-token probe localized the 1B's failure to a missing prior (prob. 10-4--10-5 on <|tool_call|>), which was erased by its web-heavy training phase. But here's the interesting part: a targeted SFT recipe (diverse corpus, 5x higher learning rate, 2,202 steps, ~3.3 GPU-hours) repaired the 1B using three orders of magnitude fewer tokens than the failed phase.
On all 269 corpus rows, valid emission rose from 0.100 to 0.959 (600M: 0.926). On 238 unseen prompts, the repaired 1B passed 0.536 vs. the 600M's 0.428 (p=0.004). Embedding-drift checks showed the repair didn't move the trigger token's tied embedding (97.7% of the bf16 table remains bit-identical), meaning changes live in the surrounding network.
Both models over-trigger, rarely answering negative prompts without a call (0.09 for 600M, 0.17 for repaired 1B). The researchers propose a ladder of strict, cheap diagnostics that costs minutes of CPU time and should gate tool-use claims on small models.
The authors bound these findings: the 600M training run was frozen at 64% of its schedule, and the model pair is a natural experiment rather than a controlled data-composition ablation. The original repaired checkpoint was lost; the reported repair results come from a reproduction. Those limits matter when inferring why the recipe worked or how well it transfers.
What worries me is that this failure mode isn't just an academic curiosity. If we're evaluating small models for real-world tool use based on these flawed metrics, we might be deploying systems that can't actually perform the tasks we think they can. The diagnostic ladder proposed is cheap to implement - it's something we should be demanding from any model claiming tool use capability.