Control AI cost by measuring tokens and requests at the workflow level, shortening irrelevant context, limiting output to what the next step needs, caching stable material when supported, and routing tasks to appropriate models. Track cost per accepted result because retries and human corrections can outweigh a cheap request.
Who this is for: Developers, operators, and product owners responsible for generative AI usage, budgets, and service performance.
- Measure actual input, output, retry, and tool usage before deciding where optimization matters.
- Reduce repeated context and unnecessary generation without removing evidence required for correctness.
- Optimize total successful-work cost, including validation and review, rather than token price alone.
Build a usage baseline
Record model identifier, input tokens, output tokens, cached tokens when reported, request outcome, retry count, latency, and task type. Provider token accounting and prices change, so use current official documentation and billing records. Aggregate by workflow and customer action rather than looking only at a monthly total.
Define the useful unit: accepted support draft, processed invoice, completed analysis, or resolved question. Divide total model and review cost by successful units. A low-priced request that fails twice and needs extensive editing can be more expensive than a stronger first response.
Reduce input waste
Remove duplicate conversation turns, repeated instructions, boilerplate, and irrelevant documents. Retrieve the passages needed for the task rather than sending an entire collection. Summarize stable history carefully, retaining decisions and evidence. Keep a test set to ensure context reduction does not delete information needed for difficult cases.
Structure prompts so stable prefixes can benefit from provider caching features when available. Do not design around undocumented behavior. Reuse verified intermediate results, such as document extraction, instead of asking the model to rediscover them in every step. Protect cache keys and stored content according to data policy.
Control output and calls
Ask for the minimum output the next step uses. Classification may need a label and evidence, not a long essay. Set documented output limits and stop conditions, but test whether truncation creates invalid structures. Avoid requesting several alternatives when one reviewed draft is sufficient.
Eliminate serial calls that repeat the same reasoning. Combine compatible operations only when doing so does not make validation opaque. Use deterministic code for calculations, formatting, filtering, and schema checks. Batch asynchronous work through supported provider features when latency does not matter, after checking current terms and limits.
Route and govern spend
Use task evaluation to decide whether a smaller model handles routine cases and a more capable model handles exceptions. Routing adds complexity, so measure misrouting and fallback costs. Set per-workflow budgets, alerts, request limits, and circuit breakers for retry loops or sudden input growth.
Review cost changes with quality and safety metrics. An optimization that increases unsupported answers or reviewer time is not a saving. Recalculate after model, price, prompt, traffic, or acceptance-rate changes. Keep pricing values in maintained configuration rather than embedding them in permanent training material.
Lower the cost of meeting summaries
A workflow sends full transcripts plus company policy to a model three times to produce notes, actions, and an email.
- Measure token use, retries, acceptance rate, and editor minutes for each of the three calls.
- Remove repeated headers and retrieve only policy passages relevant to commitments in the meeting.
- Produce one structured evidence record containing decisions, actions, owners, dates, and supporting timestamps.
- Generate the notes and email from that validated record, using code for fixed formatting.
- Compare cost per accepted package and error rates against the original workflow before rollout.
AI cost control dashboard specification
Track these measures by workflow, model configuration, and reporting period.
- Demand: requests, users, task type, input size distribution, and accepted output count.
- Usage: input tokens, output tokens, cached tokens, tool calls, retries, and failed requests.
- Unit economics: provider cost, review time, infrastructure, and cost per accepted result.
- Quality guard: correctness, evidence, format validity, unsafe failures, and human rejection rate.
- Controls: budget, alert threshold, retry cap, output cap, fallback, and accountable owner.
Common mistakes
- Cutting source context aggressively and measuring savings without checking factual quality.
- Using the cheapest listed model while ignoring retries, correction time, and failed outcomes.
- Allowing automatic repair loops to run without a bounded retry count and spend alert.
Try one
A prompt costs less after removing reference documents, but reviewer rejection rises from rare to frequent. Is the change successful, and what should be measured next?
It is not successful on token savings alone because the workflow's accepted-result rate and review cost worsened. A strong answer restores or retrieves necessary evidence, calculates total cost per accepted output, and compares factual-error and correction-time changes. Evaluation should reward preserving task quality while finding targeted context reduction rather than treating fewer tokens as the final objective.
Sources
- Gemini API token counting guideOfficial explanation of token counting and request context measurement.
- OpenAI latency optimization guideOfficial techniques for reducing requests, generated tokens, and response time.