Building an AI Output Evaluation Rubric

Derive criteria from the job

Start with the output's intended decision and acceptance conditions. A grounded support reply needs policy correctness, evidence, actionability, appropriate tone, and correct escalation. A document extraction needs field accuracy, provenance, and valid structure. Avoid importing generic criteria that do not affect the actual use.

Make each criterion sufficiently independent. Factual support and instruction adherence can fail separately. Define what the reviewer inspects and what evidence is allowed. If a criterion cannot be observed consistently, narrow it or replace it with a direct check such as required-field presence or claim-to-source support.

Write anchored score levels

Use a small scale with concrete anchors, such as pass, minor issue, major issue, and critical failure. Describe each level in task terms. A minor issue may require a quick wording edit; a major issue changes meaning or omits a required step; a critical failure creates unacceptable harm or unauthorized action.

Add positive and negative examples from representative work. Examples should clarify boundaries without becoming the entire test. For objective fields, prefer exact checks over subjective scoring. Keep an explicit not-applicable option only where it has a defined use, and do not let reviewers use it to avoid difficult judgments.

Calibrate reviewers and graders

Have multiple reviewers score the same varied sample independently, then discuss disagreements. Revise ambiguous criteria and anchors. Record adjudicated outcomes for future calibration. Domain experts should define high-impact correctness standards even if general reviewers perform routine scoring.

Model-based graders can assist at scale, but validate them against human judgments and protect against shared blind spots. A grader should not merely reward polished language. Where feasible, use deterministic checks for JSON validity, exact labels, calculations, citations, and prohibited strings, then reserve human judgment for meaning and context.

Report useful results

Report pass rates and severity by criterion, input category, and known risk. Include examples of failures and reviewer disagreement. A single average can allow excellent style scores to obscure fabricated facts. Set release thresholds before viewing the comparison so teams do not move standards to favor a preferred model.

Maintain the rubric with the workflow. Add production failures as test cases, update outdated policy expectations, and rerun baselines after changes. Official evaluation guidance evolves, so review provider recommendations while keeping acceptance criteria owned by the people responsible for the task.

Create a rubric for policy answers

An HR assistant drafts answers from an approved handbook, and a staff member reviews each response before sending.

1. Define criteria for handbook support, direct answer, preserved exceptions, source location, privacy, and escalation.
2. Create four severity anchors and mark invented policy or exposure of personal data as critical failures.
3. Assemble cases with clear policy, ambiguous policy, conflicting sections, missing coverage, and malicious text inside a question.
4. Ask two policy owners to score the same drafts independently and adjudicate disagreements.
5. Set launch thresholds per critical criterion and report editing time separately from correctness.

Evaluation rubric worksheet

Create one row per criterion and link it to representative test cases.

• Criterion: observable property, reason it matters, evidence available, and responsible domain owner.
• Anchors: pass, minor, major, and critical descriptions with one boundary example each.
• Checks: deterministic test, human judgment, specialist review, or validated grader method.
• Coverage: normal, edge, adversarial, missing-information, and known production-failure cases.
• Decision rule: per-criterion threshold, critical-failure limit, disagreement process, and release owner.

Common mistakes

• Using 'quality' as one criterion without defining what a reviewer should observe.
• Averaging critical safety or factual failures together with minor style preferences.
• Changing scoring rules after seeing which configuration the team wants to select.

Try one

Sources

• OpenAI evaluation best practicesOfficial methods for defining criteria and testing model output.
• NIST AI RMF PlaybookNIST suggested actions for applying the AI Risk Management Framework.