Choose a model by defining the task, required modalities, quality threshold, latency, volume, privacy, tool use, and failure cost. Shortlist models that meet hard constraints, then test them on representative inputs. Published capability descriptions guide the shortlist; your evaluation should decide the workflow fit.
Who this is for: Product teams, developers, and operations leaders selecting a generative AI model for a defined workflow.
- Start with task and risk requirements instead of a general claim that one model is best.
- Use hard constraints to narrow candidates before spending time on comparative evaluation.
- Measure quality, cost, and latency on realistic end-to-end cases with the same review standard.
Describe the task precisely
Identify input types, output shape, languages, context size, tool needs, and whether the task is generation, extraction, classification, coding, or reasoning. State the expected traffic and deadline. A model suitable for an interactive support draft may differ from one used for overnight document processing.
Define consequence and review. If a person checks every draft before use, a lower individual-error cost may be acceptable. If output triggers an action, reliability and control requirements increase. Some tasks should remain deterministic software or specialist work rather than being assigned to a generative model.
Separate hard constraints from preferences
Hard constraints can include supported input modality, data residency, contractual terms, context capacity, structured output, required tool interface, or a maximum response time. Eliminate candidates that cannot satisfy them. Check current official model documentation because availability, limits, and features change.
Preferences include better prose, simpler integration, lower median latency, or lower expected cost. Weight them according to the workflow rather than treating every dimension equally. Document tradeoffs. A stronger model that requires slower review may cost more overall than a smaller model with predictable output.
Build a representative evaluation
Sample real task shapes while removing or protecting sensitive data. Include common inputs, long inputs, ambiguous cases, adversarial content, and known failures. Write expected answers or scoring rubrics before viewing model outputs. Use the same prompt, tools, and validation steps where capabilities permit a fair comparison.
Score the properties that matter: factual support, extraction accuracy, instruction adherence, unsafe behavior, valid structure, reviewer time, latency distribution, and actual token use. Repeat variable cases. Aggregate averages can hide a severe failure class, so inspect results by category and consequence.
Choose and monitor the system
Select the smallest or least costly system that meets the agreed threshold with acceptable risk, not automatically the highest-scoring model. Consider routing simple and difficult cases differently only if the extra complexity has a measurable benefit. Keep a fallback plan for outages, deprecations, or unexpected behavior changes.
Version prompts, model identifiers, evaluations, and approval decisions. Monitor production samples and rerun tests when the model, prompt, source data, or workflow changes. Current model catalogs are volatile; review official documentation quarterly and avoid hardcoding assumptions from a dated comparison article.
Select a model for support reply drafts
A support team needs draft replies grounded in approved help articles, with an agent approving every message.
- Define hard requirements for language coverage, retrieval input size, response format, privacy terms, and interactive latency.
- Create fifty anonymized cases spanning routine questions, missing policy, angry customers, and prompt injection inside messages.
- Score article grounding, prohibited promises, correct escalation, agent edit time, latency, and measured usage cost.
- Compare eligible candidates by failure category rather than overall preference scores.
- Pilot the selected model with mandatory approval and monitor edits plus unsafe drafts before expanding volume.
Model selection scorecard
Use one completed scorecard per candidate and retain the supporting evaluation results.
- Task fit: input, output, modality, language, tools, context, volume, and interaction pattern.
- Hard constraints: security, privacy, residency, contract, availability, integration, and response-time limits.
- Quality measures: task correctness, evidence, format, safety, edge cases, and human edit effort.
- Operational measures: latency distribution, token use, price basis, rate limits, failures, and fallback behavior.
- Decision record: threshold, tradeoffs, approver, chosen configuration, monitoring, and reevaluation trigger.
Common mistakes
- Selecting from a public leaderboard without testing the organization's actual task and review process.
- Comparing model prices per token while ignoring different output lengths and human correction time.
- Assuming a model remains the right choice after material changes to inputs, prompts, or provider features.
Try one
Two models meet accuracy needs. One is faster but fails more often on prompt injection; the other costs more but routes those cases safely. How should the team decide?
The team should weight injection failures by consequence, confirm the test reflects production inputs, and determine whether independent controls can reliably close the faster model's gap. A passing answer compares total workflow cost and risk, not speed alone, and documents a threshold plus approval owner. High-impact unsafe failures should not disappear inside an average score.
Sources
- OpenAI models documentationOfficial current model capability and availability reference.
- Gemini API models referenceOfficial current reference for available models and supported capabilities.