Research puts the gap between benchmark scores and real-world deployment performance at around 37%. The reason is unglamorous: benchmarks use clean inputs and controlled environments, while production agents meet ambiguous requests, flaky APIs, rate limits, unexpected data shapes and users actively trying to break them.
An evaluation suite that only measures final answers will not close that gap, because it cannot tell you which step went wrong.
Three levels, and why you need all three
- End-to-end. Treat the system as a black box: did the task complete correctly? Cheap to write, tells you *that* something broke.
- Trajectory. Score the plan, the reasoning steps, the tool calls, the retries and the handoffs. Tells you *where* it broke.
- Component. Test the retriever, the individual sub-agent, the tool schema in isolation. Tells you *why*.
Teams typically build end-to-end first and stop, then find they cannot act on a regression because the trace is opaque. Building trajectory scoring second is what makes the suite actionable.
Build the suite from real failures
The highest-value eval cases are not invented — they are the failures you already had. Every production incident becomes a permanent test case, which is the same discipline as a regression test after a bug fix.
# One eval case, derived from a real incident
id: refund-ambiguous-order-reference
input: "refund the order I placed yesterday"
context:
user_orders: 3 # ambiguity is the point
expect:
tool_calls_not_containing: ["issue_refund"] # must not guess
must_ask_clarifying_question: true
reason: "Three candidate orders. Guessing costs real money." What to measure
- Task completion — did it achieve the goal, judged against a rubric rather than string equality.
- Tool-call correctness — right tool, right arguments, and importantly *no* call when none was warranted.
- Groundedness — is every claim traceable to retrieved context, or did it fill a gap.
- Refusal behaviour — does it decline cleanly when it cannot answer, or produce something adjacent and wrong.
The metric teams forget
Most suites only score cases where the agent *should* act. Half your eval set should be cases where the correct behaviour is to do nothing, ask a question, or hand off to a human. An agent that never declines is not a capable agent, it is an unevaluated one.
Run evals where the failures are
Pre-deployment suites drift because production traffic is not your test set. Score a sample of live traces continuously and feed the failures back into the suite. This is the loop that keeps the gap closing rather than reopening after each model change.
A model change is a regression risk
Swapping the underlying model — or accepting a provider's silent upgrade — changes behaviour in ways your unit tests will not catch. Treat the model version as a pinned dependency and re-run the full suite on any change, exactly as you would for a major framework upgrade.
Frequently Asked Questions
How many eval cases are enough to start?
Twenty real cases beat two hundred invented ones. Start with every production failure you can remember, plus the ten most common successful requests as regression guards.
Can an LLM grade my agent?
For rubric-based judgements, yes, with the caveat that the judge needs its own validation against human labels. For deterministic checks — was this tool called, was this argument correct — use assertions, not a judge.
Where do evals live?
In the repository next to the code, running in CI. An eval suite that only runs manually stops running.
References
About Jishu Labs
Jishu Labs is a software development company founded in 2016. We build custom software, AI/ML systems, and full-stack web and mobile applications for clients, and we make eight AI tools for software teams.