AIAI’s recent article, “System One models: why fast AI agents act wrong,” describes a failure agent builders will recognize. The System One model covers “read agent state well and still pick the wrong action.”
The fix described is plain: the authors repaired the failures by combining the models’ correct answers to factual questions in ordinary code.
That points to a rule for an agent that can change something outside its own context window. The model proposes, and a workflow checks the proposal and takes the action.
Anthropic’s guide to building effective agents defines workflows as “systems where LLMs and tools are orchestrated through predefined code paths.” Agents, in its words, are “systems where LLMs dynamically direct their own processes and tool usage.”
For most business processes I build a hybrid. A fixed workflow runs the steps someone can write down, and an agent handles the single step that calls for judgment, such as parsing a messy email.
The agent returns a choice
Give the agent the data it needs and a short list of allowed answers. Make it return one of them: a category, a label or a yes or no. The workflow owns the rest: the trigger, the lookups, the write and the log.
Take a refund request. The agent reads the customer’s email and the order record, then returns “refund,” “replacement” or “escalate.” If it returns “refund,” the workflow pulls the amount from the order record, compares it with the refund ceiling and issues the refund.
In this design a wrong answer surfaces at one known point. Rerun that step on the same input to see why the model chose as it did, and swap in a different model later without touching the rest of the process.
Prompt rules did not hold
I learned how far prompt rules go on a front-desk phone agent we run for a local service business. It can book new customers and send messages to the front desk. In one week of calls we reviewed in September, the agent told a caller that a message had already been sent.
The notification call had been cut off when the caller spoke, and nothing confirmed it went out. The prompt already told it to acknowledge only actual notification results and never to resend an unconfirmed one.
The backend was healthy that week, and all of its executions succeeded. The logs showed that the workflow ran. They said nothing about whether the agent chose the right action.
We tried the prompt first, and prompt-only versions failed in testing. One model announced actions it had not taken. Another retried an abandoned notification. Switching models did not fix it by itself.
In testing, a procedure around the write held up. The procedure forces the call and reads the receipt. The agent may confirm only when four receipt fields agree: a submitted flag, a submitted status, a message ID from the provider and the matching action.
Any other result takes a fixed uncertainty line. The first strict test exposed an edge case: a receipt with the ID absent rather than empty. We corrected the check, and three test cases then passed: a valid receipt, an interrupted submission and a missing ID.
In October we tested the same idea on dates and identity. The test bank held ten scenarios drawn from three weeks of the business’s real calls, run as text simulations with a simulated caller. On the live version, 17 of 20 runs had no defect.
In one simulated booking, the agent invented the caller’s details (“Caller Caller,” an example.com email address and a birth date of 1990-01-01) and read “next Tuesday” as November 3. The other booking used the business’s own email address. A simulated emergency callback carried an invented birth date.
A new customer record requires the caller’s own name, birth date and email, and the write tool refuses placeholders before it writes anything. The same ten scenarios ran 21 times with no defect.
The test is small: text only, simulated callers, two or three runs per scenario and no audio. Four runs on earlier versions of the fix failed. The booking skipped intake, or an emergency callback carried the name “Caller.”
Two rules, emergency routing and cancellations, still live in the prompt, on top of a business-hours status that code supplies.
Limit what the agent can touch
Start the agent with read-only access, and let it write only by asking the workflow, one action per request. OWASP’s entry on excessive agency lists excessive functionality, excessive permissions and excessive autonomy as root causes.
Its mitigations include implementing authorization in downstream systems rather than relying on the model to decide whether an action is allowed. A refund ceiling, a discount limit and a recipient list are rules the workflow checks.
Choosing which customer record to open is an action too. In a customer-history lookup we built for a client, the call platform supplies the caller’s phone number to the tool, so the model cannot choose which record opens. If two contacts share the number, the lookup returns no history.
Treat anything the agent reads, including emails and web pages, as untrusted input. Hidden instructions in that text can steer a model that holds broad permissions.
Write the handoff rules before launch. A person takes over in four cases: the customer asks for one, the agent’s answer falls outside the allowed format, the conversation returns to the same question a second time, or the case involves money, legal terms or health information.
Test the choice and price the run
Test the agent step before it touches a customer. Build a set of a few hundred real past cases, including the ugly ones, and write down the right outcome for each. Score the agent against the set, and score it again after every prompt or model change.
Then run in shadow mode: the agent makes real decisions on live traffic, and a reviewer approves each one before the workflow sends or writes anything. Automate the categories where the reviewer’s change rate stays low.
Price the run the same way. A fixed workflow costs what the automation tool charges. An agent also bills for model usage, which grows with the text it reads, the tools it calls and the loops it runs. A research agent that runs a search, opens five results and tries again can cost a multiple of one classification call.
Anthropic’s guide says autonomy “means higher costs, and the potential for compounding errors.” A bounded choice puts a ceiling on the cost of each run, because the workflow makes one model call where an open agent makes as many as it decides to.
Emre Benian runs an AI consulting & implementation company in Chicago for owner-led businesses.
