AI demos are optimized for a moment. A clean prompt goes in, a polished answer comes out, and everyone around the table can imagine the finished product. Production is different: people bring incomplete context, unfamiliar abbreviations, and sources that disagree.
That gap is why evaluation is not a final quality gate. It is part of product design. A useful evaluation system tells a team what good means, where the product fails, and which improvement is worth building next.
Start with the decision, not the model
Teams often begin with a convenient model metric. Those numbers can help, but they do not define the product outcome. A weather app is not useful because its sentences resemble a reference answer; it is useful when a person can decide whether to carry an umbrella.
For an AI assistant, write the user decision in plain language first:
A data scientist should be able to find the approved rule, understand its source, and decide whether it applies without asking a platform expert.
Translate the decision into observable signals
- Task success: Did the user reach the correct next step?
- Grounding: Can each important claim be traced to an approved source?
- Boundary behavior: Does the assistant say when evidence is missing?
- Effort: How many turns or corrections did the task require?
A single score hides trade-offs. A response can be fluent and wrong, accurate and unusable, or safe but needlessly evasive.
Build a small, living evaluation set
The first dataset does not need thousands of examples. It needs coverage of the decisions that could change the product. Start with cases collected from interviews, support questions, pilot sessions, and known failure modes.
- Map the core journeys.
- Add vague requests, conflicting sources, and missing context.
- Record why each case matters in a field such as
risk_reason. - Turn real production failures into regression cases.
Treat the set like a garden, not a museum. Prune duplicate cases and add new varieties when user behavior changes. The Markdown Guide is a useful reference when authoring new posts for this site.
Make every failure actionable
“The answer was bad” is not a diagnosis. A compact result should point toward an owner or intervention:
{
"case_id": "protocol-017",
"task_success": false,
"failure_mode": "retrieval.missing_source",
"severity": "high",
"next_owner": "search"
}
The category retrieval.missing_source is more valuable than a low aggregate score because it shows the team where to inspect the system.
Put evaluation into the product rhythm
An evaluation suite that only runs before launch will slowly become ceremonial. Connect it to decisions the team already makes, and keep the evidence beside each product change.
The purpose of evaluation is not to prove that an AI system is intelligent. It is to help a team make the next responsible decision.