Designing a Production AI Agent Architecture

Separate responsibilities

The user-facing application authenticates the person and captures the goal. An orchestrator loads durable task state, selects the allowed model decision, and coordinates bounded steps. A model gateway applies approved configuration and records usage. Tool adapters validate typed arguments, authorization, timeouts, and idempotency before calling external systems. No model output directly becomes a privileged operation.

A knowledge service manages ingestion, permissions, retrieval, provenance, and source versions. A policy layer decides which tools, data, and approvals are allowed for the user and workflow state. An approval service binds a reviewer to an exact proposal. Observability links these components. The boundaries can begin in one application, but their contracts should remain explicit.

Model state and execution

Represent work as durable tasks with named states, attempts, budgets, pending approvals, evidence, and verified outputs. Conversation history may inform a decision, but it is not the authoritative transaction log. Use a queue for asynchronous work and protect each task against concurrent execution. Checkpoint completed steps so a process restart does not repeat successful side effects.

Design every write for ambiguous outcomes. A timeout may occur after the target system accepted an action. Use idempotency keys, status lookup, and reconciliation instead of blind retries. Apply maximum steps, elapsed time, token or cost budget, and repeated-observation stops. Route exhausted or contradictory work to an operator with the trace and current state intact.

Secure data and capabilities

Propagate user and tenant identity through retrieval and tool execution. Issue short-lived, scoped credentials where possible and isolate development, test, and production. Minimize data sent to models and external services. Classify sources as trusted instructions or untrusted content, and prevent retrieved pages, MCP resources, emails, and tool results from changing policy.

Expose capabilities by workflow state rather than giving the agent one permanent tool catalog. Separate low-risk reads, reversible drafts, approval-bound writes, and prohibited operations. Store secrets outside model context. Review network destinations, logs, caches, and backups as part of the data path. A service boundary is useful only when its access controls are enforced.

Plan releases and recovery

Version prompts, model settings, tool schemas, policies, retrieval preparation, and evaluation sets. Run offline cases and shadow traffic before enabling new actions. Roll out by workflow or tenant with flags that can disable individual tools or return the system to recommendation-only mode. Database and external API changes need compatibility and rollback plans.

Operate with traces, outcome metrics, cost limits, alerts, and runbooks. Test dependency outages, revoked permissions, stale approvals, queue duplication, model changes, and operator handoff. Conduct capacity planning around rate limits and downstream bottlenecks. Platform and protocol details evolve, so confirm current OpenAI and MCP documentation quarterly while keeping core contracts portable and provider-aware.

Architect an invoice exception assistant

A finance team wants an agent to investigate invoice mismatches, prepare corrections, and route approved changes to its accounting system.

1. Authenticate the operator and create a durable task containing invoice identity, tenant, state, budget, and source references.
2. Retrieve approved purchase and receiving records through scoped read adapters, then let the model classify the mismatch from typed evidence.
3. Validate totals deterministically and create a versioned correction proposal without exposing the accounting write tool during investigation.
4. Bind finance approval to the exact account, amount, and fields, then execute through an idempotent adapter that returns verified ledger state.
5. Trace every transition, test timeout-after-success and stale-approval cases, and provide flags for read-only fallback and tool shutdown.

Production agent architecture map

Document these contracts before a workflow receives production credentials or autonomous execution.

• Entry and identity: user, tenant, goal, authentication, authorization, and request limits.
• Orchestration and state: states, transitions, checkpoints, concurrency, budgets, queues, and stop rules.
• Knowledge and models: sources, permissions, provenance, configuration, context limits, and versions.
• Tools and approvals: schemas, credentials, validation, idempotency, protected proposals, and results.
• Operations: evaluations, release flags, traces, metrics, alerts, fallbacks, rollback, and incident ownership.

Common mistakes

• Putting workflow state only in chat history and losing authoritative progress when a process retries or restarts.
• Allowing model-generated arguments to call external systems without typed validation, authorization, and idempotency.
• Shipping one inseparable agent deployment with no way to disable a risky tool while preserving lower-risk service.

Try one

Sources

• OpenAI agents guideOfficial guidance for building agent workflows with current OpenAI platform capabilities.
• MCP architecture overviewOfficial description of MCP hosts, clients, servers, and protocol layers.