Troubleshoot slowness by defining who is affected and when, comparing latency distributions with a healthy baseline, tracing requests across the path, and checking demand, errors, and saturation at each dependency. Change one reversible variable only after evidence supports a hypothesis, then verify the user-facing result.
Who this is for: Developers and on-call operators diagnosing intermittent or sustained latency in a distributed cloud application.
- Measure latency by journey, percentile, location, and release rather than relying on one average.
- Follow request timing across queues and dependencies to locate where waiting time accumulates.
- Preserve evidence and prefer reversible mitigation before restarts, scaling, or configuration changes.
Bound the symptom first
Record the affected user journey, start time, regions, tenants, request types, frequency, and exact experience. Distinguish slow responses from connection failure, client rendering delay, retries, and queued background completion. Compare median and tail latency with a known healthy period. Averages can remain normal while a meaningful minority waits far too long.
Check recent deployments, configuration, traffic, data size, provider events, certificate changes, and dependency releases without assuming correlation is cause. Create a shared incident timeline. If impact is severe, assign an incident lead and communication owner so investigators can focus without making overlapping changes.
Follow the request path
Break total time into DNS, connection, TLS, load balancer, application queueing, application work, cache, database, external calls, and response transfer. Use traces where available and correlate them with structured logs and metrics. Compare a slow request with a similar fast request to identify the span where duration diverges.
For asynchronous work, include broker delay, oldest-message age, consumer processing, retries, and dead-letter behavior. Verify clocks and trace propagation before trusting cross-service timing. When tracing is sampled, retain slow and failed examples through an approved method rather than enabling unrestricted verbose logging in production.
Test saturation and contention hypotheses
Inspect demand, errors, latency, and saturation for each constrained resource. CPU may be low while threads, connection pools, storage operations, memory, locks, file descriptors, or external quotas are exhausted. Database plans can change with data distribution, and one expensive query can block unrelated work. Queueing often causes tail latency before utilization appears fully saturated.
Form a falsifiable statement such as 'checkout waits for the database pool because active connections equal the pool limit during slow traces.' Identify evidence that would disprove it. Use read-only queries and dashboards first. Reproduce with production-like data in a safe environment when possible, and avoid running unbounded diagnostic queries against the impaired database.
Mitigate, verify, and learn
Choose the lowest-risk reversible mitigation supported by evidence: route around an unhealthy zone, pause a costly background job, revert a release, reduce a feature's exposure, or add bounded capacity below dependency limits. Restarts can temporarily hide leaks and erase process evidence, so capture useful diagnostics and understand customer consequences before using them.
Verify the original user journey, latency distribution, errors, backlog, saturation, and dependent systems after mitigation. Continue long enough to cover the triggering load pattern. Then fix the underlying defect, add a regression or load test, improve telemetry, and document the decision. Remove temporary capacity and debug settings deliberately once evidence shows they are no longer needed.
Diagnose slow product search
Search p95 latency rose from 400 milliseconds to six seconds after a catalog import, while average CPU stayed below 30 percent.
- Segment latency by query type, release, zone, and catalog size, then compare slow and fast traces from the same period.
- Locate waiting time in database spans and inspect pool occupancy, query duration, storage operations, locks, and execution plans with bounded read-only tools.
- Confirm that imported data changed one query plan and reproduce it against a safe copy before selecting an index or query fix.
- Deploy the fix progressively, compare tail latency and write cost, and retain a plan-regression test using representative catalog statistics.
Latency investigation worksheet
Use this worksheet to keep incident diagnosis evidence-based and changes coordinated.
- Journey, impact, start time, scope, percentiles, healthy baseline, and incident roles.
- Request segments, trace examples, queue time, dependencies, retries, and telemetry confidence.
- Demand, error, latency, saturation, resource limits, quotas, data changes, and release markers.
- Hypothesis, supporting evidence, disconfirming check, safe reproduction, and decision owner.
- Mitigation, blast radius, rollback, user verification, observation period, root fix, and follow-up test.
Common mistakes
- Scaling application instances because users report slowness without checking the saturated dependency.
- Restarting every process at once and losing the evidence needed to distinguish leak, load, and dependency failure.
- Using average latency as proof of recovery while tail requests and queue age remain high.
Try one
An API is slow, CPU is low, and adding instances makes the database error rate worse. What should the next investigation focus on?
A strong answer stops indiscriminate scaling and examines traces, database connection pools, active sessions, query latency, locks, storage, retries, and quotas. It compares slow and fast requests and forms a disprovable bottleneck hypothesis. Any mitigation stays below database limits, preserves evidence, and is verified against user-facing tail latency.
Sources
- Amazon CloudWatch documentationOfficial AWS overview of metrics, alarms, dashboards, logs, and application monitoring.
- Kubernetes running Pod debuggingOfficial Kubernetes workflow for inspecting and diagnosing running application Pods.