Every AI agent evaluation tells you the same frustrating half-truth: the model picked the wrong tool. What it rarely tells you is why. A new paper from researchers Atul Anand and Sourav Chattaraj finally builds a way to answer that question.
The answer involves canaries…
Think coal-mine canaries, but redesigned for software. These are diagnostic probe tools planted directly inside an agent’s Model Context Protocol (MCP) toolset, each one engineered to trigger one specific tool-selection weakness.
Give an agent a task, seed its available tools with a canary, and watch exactly which one it reaches for…
What canary tools actually catch
Anand and Chattaraj built six categories of canaries, each targeting a distinct way agents fool themselves. Together, they read like a taxonomy of every trap a busy, overconfident intern could walk into:
- Semantic decoys: tools with descriptions written to sound like a better fit than they are, testing whether an agent reads past the marketing copy.
- Parameter traps: the correct tool, called with the wrong parameters. The agent grabs the right hammer and swings it sideways anyway.
- Capability mirages: tools that promise more than they deliver, checking whether an agent takes a stated capability at face value.
- Prerequisite blindness: tools with hidden dependencies the agent needs to satisfy first, testing whether it questions its own assumptions.
- Temporal decoys: timing-sensitive traps that catch agents reaching for a tool that would have worked five minutes ago.
- Granularity traps: tools scoped too broadly or too narrowly for the job, testing whether an agent understands the actual size of the task in front of it.
The setup behind the numbers
The scale here matters. Eight models, 120 tasks, 8,640 total runs, plus a set of ablation studies layered on top. Human judges cross-checked the outcomes with a Cohen’s kappa of 0.75, a level of agreement most annotation teams would frame and hang on the wall.
The gap is bigger than most people would guess
- Susceptibility to canary traps (what the paper calls the canary susceptibility rate, or CSR) varied by a factor of roughly 36 across the eight models tested.
- Claude Opus 4.8 posted the lowest susceptibility of the group.
- Llama 3.1 8B posted the highest. That range alone is worth sitting with for a second.
Capability tier fails to predict safety, and procurement teams should care
Here is where the paper earns a spot in a decision-maker’s reading pile. The capability tier fails to predict safety on its own.
The most susceptible hosted model in the study sat in the middle of the pack, not at the bottom where a leaderboard would suggest.
Within a single provider’s own lineup, the cheaper model sometimes came out safer than the pricier one sitting above it.
The result holds up under scrutiny
A fair skeptic might assume canary tools work by planting an obvious giveaway phrase that any model eventually learns to spot. The researchers tested this directly by softening each canary’s giveaway phrase, and frontier-model CSR barely moved.
That is a strong signal that this measures genuine reasoning weakness rather than keyword spotting.
Susceptibility also correlates with real task failure, with a Spearman correlation of -0.34. Models that fall for more canaries complete fewer tasks correctly.
That links a lab measurement to something a business actually feels: agents that fail at their jobs while everyone assumes they are working fine.
What builders and buyers should do with this
The paper is a research contribution, but it reads like a checklist for anyone shipping agents into production right now:
- Plant your own probes before launch: build a handful of canary-style tools into your MCP toolset and watch what your agent actually reaches for under pressure, before a customer finds out for you.
- Treat benchmark tier as one signal among several: a model’s leaderboard rank says little about how it behaves when a tool description oversells itself.
- Give mid-tier models extra scrutiny: the paper found the most susceptible hosted model sitting mid-pack, exactly where teams tend to relax their attention.
- Track susceptibility alongside completion rate: given the correlation the researchers found, a rising CSR is an early warning worth watching before it shows up in your success metrics.
Where this leaves the industry
Tool selection has become one of the most consequential decisions an agent makes, repeated hundreds of times a day, mostly out of sight of the humans relying on the output.
Canary tools give the industry a way to measure that decision directly, rather than guessing at it from whatever broke downstream.
Anand and Chattaraj have handed builders a genuinely useful instrument. The smart move is pointing it at your own agents first.
Building agents that actually hold up in production?
Join 300+ AI builders and tech leaders at the Agentic AI Summit Los Angeles on August 26, where we’ll dig into agent architectures, MCP, governance, observability, and what it takes to keep AI systems on the rails.
