AI Evaluation and Reliability — Knowing When to Deliver

One successful demonstration says little about the next user’s experience. Evaluation exposes failures before delivery and helps you tell what a change actually improved.

Building on AI Workflows, this chapter asks which approach—manual work, the current workflow, or a candidate—delivers acceptable results at an acceptable cost. Small samples and thresholds here are teaching choices, not deployment certification or representative claims about every user.

Define acceptable performance from the task

Instead of asking whether an answer is good, specify what the user needs to do next and which failures are unacceptable. A comparison of public event information must let readers trace claims, distinguish known from unknown information, and avoid presenting an old schedule as current.

Separate two layers:

• Conditions that must all pass: real sources, accurate consequential figures, no disclosure of restricted material, and no unauthorized actions. Elegant writing cannot offset a failure.
• Quality to compare: organization, readability, revision effort, and total time. Compare these after the first layer is satisfied.

Write the criteria before seeing the results. Relaxing them after a candidate fails makes the evaluation a defense of your preferred solution.

Build a small, varied test set

Sample actual tasks you intend to support, obtain permission to use materials, and remove unnecessary information. A first set might contain ten cases: four ordinary inputs, two missing information, two conflicting or outdated inputs, one tool failure, and one source containing operational instructions. This is a starting mix, not a standard; cover failures with different consequences.

Store each input, expected behavior, and judging rationale. Do not insert all test answers into the prompt and call the resulting score generalization. Reserve examples that have not informed changes for the adoption decision. Once you use that set to tune the prompt, replace it with fresh held-out cases.

Change one main factor per comparison

Give current and candidate versions the same inputs, sources, permissions, and budget. Record model identifiers, prompt and source versions, and run times. Changing the model, prompt, and knowledge base together makes improvements harder to attribute.

Repeat cases a few times to observe variation, particularly for tasks that act on external systems. Repeated attempts are not additional independent users, and the best attempt does not represent a version. Keep failures, timeouts, and human takeovers in the denominator.

Anthropic’s January 2026 evaluation article separates objects that should not be conflated: a task has fixed inputs and success criteria, a trial is one run of that task, a transcript contains the tool calls and intermediate interactions, an outcome is the environment’s final state, and an evaluation or agent harness runs, records, and grades the process. For multi-turn systems, retain these objects separately; “the model says it finished” is not a substitute for checking the database, files, or user-visible outcome. We reference these concepts; the examples, rules, and worksheet below are this guide’s own exercises. See Demystifying evals for AI agents.

Worked example: faster, but not ready for adoption

Suppose a team runs each of ten public-information cases once, producing these fictional results:

At an illustrative internal time value of RMB 60 per hour, batch cost is 120 ÷ 60 × 60 + 6 = RMB 126 for A and 100 ÷ 60 × 60 + 8 = RMB 108 for B. These are process costs only, excluding acquisition, taxes, and other business expenses. Including rework, cost per accepted result is RMB 12.6 and RMB 10.8 respectively.

B is faster and succeeds more often on first delivery, yet violates the no-invented-facts condition. The decision is do not expand use; fix and retest. This does not establish A as reliable: ten inputs with one attempt each remain limited evidence.

Keep first-pass acceptance separate from acceptance after human repair. Counting repaired answers as model successes inflates quality and hides maintenance costs.

Who grades, and how to handle disagreement

Use explicit rules for fields, calculations, and file state. Let people who understand the task judge meaning, usefulness, and expression. If a model helps score outputs, compare it against human-labeled examples first and check whether it favors longer, more confident answers.

Inspect each consequential error against the original materials, beyond the average score. When reviewers disagree, record the dispute, clarify the rubric, and reassess. Voting alone cannot fix ambiguous criteria. Accept valid differences in phrasing, and allow “the material is insufficient” to be the correct answer.

Turn failures into regression checks

After a fix, rerun both the triggering case and previously passing cases. For external services, preserve source and environment versions and explain which conditions cannot be reproduced fully.

Move into limited use

Old Scores Do Not Automatically Survive a Model Change

A passing result describes particular inputs, model, materials, memory settings, and tool permissions. This guide recommends recording them together. If a service does not expose a fixed version, retain its displayed model name, run date, and uncontrolled conditions instead of claiming complete reproducibility.

Maintain two sets with different purposes: regression cases retain known failures to prevent recurrence; held-out cases check situations that did not inform tuning. Once repeated inspection of the held-out set guides changes, move it into development or regression coverage and add fresh cases.

After changing models or tools, updating sources, enabling memory, or observing anomalies, run relevant cases in a test environment and inspect final files or system state. For retrieval, separately check whether expected material was found and whether it supports the conclusion. For multimodal tasks, separate recognition accuracy from interpretation. This helps locate the layer that needs repair.

Record the adoption date, owner, and last accepted version. If a mandatory condition fails, reduce use to the validated scope. If the old version cannot be restored, keep a manual fallback. Include maintenance in the cost; release is the beginning of continued observation.

Give Each Adoption an Exit

Before adoption, specify supported tasks, unsupported inputs, human takeover conditions, spending limits, and rollback. Continue collecting real failures during limited use and refresh the sample set. Rerun relevant checks after changes to the model, prompt, retrieval, or tools.

Keep a complete comparison in the AI Evaluation Record. If you do not have an actual task yet, return to Customer Discovery: rigorous evaluation cannot establish demand for an output nobody needs.