Back to blog

Why does your agent still ship without real proof it works?

Thắng Đoàn
Thắng Đoàn

Your agent demos well, then quietly fails in production, and your test suite stays green. The gap exists because an agent eval is not a unit test, and most teams never build one.

Vibe checks are not a measurement

Unsuccessful products almost always share one root cause: a failure to create reliable evaluation systems. Hamel Husain has watched many LLM products succeed and fail, and says unsuccessful products almost always share one root cause: a failure to create reliable evaluation systems. He argues you need to measure success beyond vibe checks, which are useful but not enough. That means evals tailored to your users' tasks, not general model benchmarks.

  • Benchmarks like leaderboards measure the model, not your task
  • A vibe check catches one failure, an eval catches the class

Why unit tests cannot do this job

Without evals, it can be very difficult and time intensive to understand how different model versions might affect your use case. A unit test checks a fixed input against a fixed output. An agent takes many paths to an answer, so the same prompt can produce different valid responses. OpenAI's evals framework exists precisely because of this: without evals, it is very difficult and time intensive to understand how different model versions affect your use case. You are testing a distribution of behavior, not a single function.

  • You judge quality, not just correctness
  • Model updates shift behavior under your feet
  • Private evals let you test on your own workflow data

Simple agents, evaluable agents

The most successful implementations weren't using complex frameworks or specialized libraries. Anthropic worked with dozens of teams building agents and reports that the most successful implementations avoided complex frameworks. They used simple, composable patterns instead. Simpler agents are easier to reason about, and that makes their evals tractable. If you cannot describe what success looks like, you cannot test for it.

What this does not solve

Evals measure the cases you thought to encode. They will not catch harms, edge cases, or user contexts you never anticipated. Husain's own framing is that evals, logging, and the ability to change behavior must work together; any one alone stalls iteration. An eval suite is a living artifact, not a one-time certification.

Could your team describe, concretely, what a good agent outcome looks like for your top user task?
If a model vendor shipped a new version tomorrow, how would you know if your agent got worse?

Write down three real tasks from your users and sketch what a good versus bad agent answer looks like for each.

Sources

hamel.dev: Your AI Product Needs Evals

OpenAI Evals framework

Anthropic: Building Effective Agents

Share:

Recommended for you

Enjoyed this article?

Subscribe for new articles. No spam. Unsubscribe anytime.

By subscribing you agree to receive the newsletter. No spam, and you can unsubscribe anytime.