An agent that answers well in a demo can still refund the wrong order, call a write tool twice, or follow instructions hidden in a retrieved document. That is why quality, not cost, is now the main reason many teams hesitate to ship.
LangChain’s State of Agent Engineering report, a public survey of 1,340 professionals from 18 November to 2 December 2025, found that 57.3% of respondents already had agents in production. Quality was the top barrier to production, cited by 32%. Nearly 89% had some observability, but only 52.4% ran offline evaluations on test sets, and 37.3% ran online evaluations on live traffic. Among respondents with production agents, 22.8% still were not evaluating at all. The sample was self-selected and skewed toward technology firms (63%) and organizations with fewer than 100 people (49%), so treat it as a practitioner snapshot, not a census.
The gap is easy to explain. A trace tells you what happened. It does not tell you whether the run was acceptable, or whether the next ten runs will look the same.
The Last Message Is a Weak Test
Agents change things. They call tools, write records, and adapt mid-conversation. A polite closing line can hide a failed refund.
The tau-bench paper from Sierra researchers grades support-style agents by comparing the final database state with a known goal, not by scoring the chat. With native function calling, GPT-4o reached a pass^1 success rate of 61.2% on the retail domain and 35.2% on the airline domain. Consistency fell quickly: the chance of succeeding on all eight independent trials of the same retail task dropped below 25%. Those figures come from the paper’s 2024 model lineup, but the lesson still holds. One clean demo is not a release test.
Anthropic’s guidance on agent evals draws a useful distinction between pass@k (at least one success in k tries) and pass^k (success on every try). Customer-facing workflows should be judged on the second measure. Anthropic notes that a 75% per-trial success rate implies only about a 42% chance of three successes in a row.
If an agent can spend money, change records, or talk to customers, measure how often it succeeds and what it does when it fails.
Start With a Small Set You Can Trust
You do not need hundreds of cases to begin. Anthropic recommends 20 to 50 tasks drawn from real failures and the checks you already run by hand. Keep two groups. Start capability tasks hard so you can see improvement. Regression tasks should almost always pass, so a drop means something broke.
Write each case so two reviewers would agree on pass or fail. Include a known-good reference outcome. After every serious miss, add the failing input to the set.
Mock the Tools and Isolate Every Run
Pointing a test agent at live systems is a poor idea. Refunds, emails, and tickets will leak into the real world, and leftover files or cached data can make later trials look better or worse than they are. Anthropic describes agents picking up an unfair advantage from git history left by earlier trials.
Give each trial a clean sandbox. Stub or record responses for read tools. For writes, use a disposable copy of the database and assert the final state. Google Cloud’s agent evaluation docs separate offline test-case evaluation, which belongs in continuous integration, from online monitoring after release.
When you inspect the path, prefer flexible checks: required tools, forbidden tools, and argument limits. Google Cloud’s trajectory metrics include exact match, in-order match, and any-order match for a reason. An exact sequence of calls is often too strict, because a valid agent may take a slightly different route to the same correct state.
Grade With Code First, Then a Careful Judge
Code checks are fast, cheap, and repeatable: amounts, status fields, tool permissions, turn limits, and token use. Use a second model only for tone, groundedness or policy language, and calibrate it against human labels. In the LangChain survey, 59.8% of teams running evals used human review, and 53.3% used an LLM-as-judge. Neither replaces a database assertion.
Read failing transcripts. A low score can mean a broken grader or an ambiguous task, not a weak agent.
Attack the Agent Before Users Do
OWASP lists prompt injection as LLM01 in its 2025 Top 10 for LLM applications. Direct injections come from the user. Indirect ones hide in web pages, files, or retrieved documents. OWASP also recommends least privilege, human approval for high-risk actions, and regular adversarial testing.
Add cases that try to override policy, leak data, or trigger write tools from untrusted content. The test fails if the agent calls a privileged tool, skips a required confirmation, or follows hidden instructions.
Shadow First, Then Open the Gate
After offline tests pass, run the new version beside the current one without giving it authority to act, and compare traces. Then sample live traffic with online evaluators, and feed every serious miss back into the regression set.
Before you go live, confirm five things. The golden set covers failures you have already seen, plus injection and policy cases. Graders check state and permissions, not only the final sentence. Each critical task has been run more than once, with pass^k, cost, and latency recorded. A trial can run without touching production systems. A prompt, model, or tool change will fail continuous integration before it reaches users.
If any of those is missing, the workflow is not ready. Build and enforce the tests first.
