Managing Production Configuration

Inventory configuration by purpose

List service endpoints, resource identifiers, timeouts, concurrency, feature controls, log levels, policy thresholds, and credentials. Mark which values differ by environment and which should remain identical. Configuration that changes application behavior deserves the same ownership and review as code, even when an operator can edit it through a console.

Define a schema with type, allowed range, format, requirement, default, sensitivity, and compatibility. Parse values once at startup or at a controlled reload boundary. Fail safely on missing critical configuration with an error that names the key but not its value. Defaults should be conservative and should never silently point a development process at production.

Choose authoritative storage

Ordinary reviewed settings may live in environment-specific configuration repositories, deployment manifests, or managed parameter stores. Secrets belong in a system designed for confidential storage, access control, encryption, rotation, and audit. Avoid duplicating the same value across several sources because operators cannot tell which copy governs a running instance.

Parameter and secret paths should encode application and environment clearly without placing secret material in names. Use workload identity and least privilege for retrieval. Restrict write access more tightly than read access, and separate production from lower environments. Back up or replicate configuration metadata according to recovery needs without making uncontrolled secret copies.

Release settings deliberately

Create changes through a reviewed path that shows old value, new value, environment, affected services, validator result, owner, and reason. Test in a lower environment with representative constraints. For a high-risk timeout, routing rule, or feature control, use staged exposure and define a metric-based stop condition.

Understand propagation. Some applications read at startup, some poll, and some receive notifications. Mixed versions can create inconsistent behavior. Tie immutable configuration versions or hashes to deployment records and expose a safe version identifier in diagnostics. Never print the resolved secret value merely to prove that loading worked.

Audit, rotate, and recover

Record who changed a value, through which workflow, and when each service adopted it. Alert on out-of-band edits and reconcile them through the source of truth. Periodically review unused keys, stale environment overrides, broad access, and values with no owner. Remove them only after confirming consumers and rollback requirements.

For secrets, test rotation with overlapping credentials where supported and revoke old access after all consumers move. For ordinary configuration, retain a known-good prior version and a rapid deployment path. Include configuration and secret dependencies in disaster recovery exercises because restored code cannot operate without them.

Change a production request timeout

A team wants to increase an API timeout from five to twenty seconds after reports of failed exports.

1. Confirm whether timeout is the cause by tracing export latency, downstream limits, retries, and resource use rather than changing it immediately.
2. Update the typed setting in the authoritative production configuration proposal with range validation and a documented prior value.
3. Apply it first to a small instance group, verify adoption through a version hash, and watch success, latency, concurrency, and saturation.
4. Expand only if the user outcome improves within limits; otherwise restore the prior version and investigate the slow operation.

Production configuration catalog

Keep one catalog entry per setting or cohesive configuration document.

• Key, purpose, service, owner, type, range, default, required status, and sensitivity.
• Authoritative source, environment path, reader identity, writer role, encryption, and audit log.
• Load timing, propagation method, compatibility, version identifier, and consumer inventory.
• Validation, test environment, rollout, indicators, stop condition, prior value, and recovery method.
• Last change, last access review, rotation schedule, retention, disaster recovery, and removal criteria.

Common mistakes

• Changing a production console value without updating the source that will overwrite it on the next deployment.
• Giving every service access to an entire secret hierarchy because individual policies take more effort.
• Using a default production endpoint when an environment-specific variable is missing.

Try one

Sources

• AWS Systems Manager Parameter StoreOfficial AWS guidance for hierarchical configuration data and controlled parameter access.
• AWS Secrets Manager documentationOfficial AWS introduction to storing, retrieving, rotating, and auditing application secrets.