When I troubleshoot a slow serverless chain, I stop looking for one magic slow service and start tracing the whole request path. I recommend proving where time is spent before changing memory, code, or database settings.
Architecture under test
Amazon ECS services on AWS Fargate
Data stores
Common factors
In this shape, performance usually degrades in five places. First, the edge can add time through authorization, request mapping, payload size, or throttling. Second, the ALB can reveal target health, connection, and backend timing problems. Third, Fargate tasks can run hot on CPU or memory, or scale slower than the incoming burst. Fourth, service to service calls can multiply latency through retries, synchronous fan out, missing timeouts, and no connection reuse. Fifth, the data layer can dominate everything through PostgreSQL lock waits, slow query plans, connection pool starvation, DynamoDB throttling, or a hot key.
Do not start by tuning random knobs. Start with the request path and ask which hop owns the p95 or p99 time.
Trace the problem
When I talk about "Trace the problem", I am checking whether Finding Performance Bottlenecks in a Serverless Microservice Chain makes ownership, failure handling, or rollback clearer.
Put a trace ID on the request at the edge and carry it through every downstream call. For HTTP APIs, use API Gateway CloudWatch metrics to separate total latency from integration latency. For ALB, enable access logs and inspect request processing, target processing, and response processing time. ALB can add the X-Amzn-Trace-Id header, but it does not send data to X-Ray or appear as a service map node. That means the services themselves must emit spans.
Instrument Service A, B, and C with OpenTelemetry or an X-Ray compatible path. Each span should include route, downstream host, target service, database operation, status code, retry count, and timeout outcome. A useful trace is not a pretty waterfall. It is a receipt that says where the request waited.
| Layer | What commonly hurts | Evidence to collect | Fix direction |
|---|---|---|---|
| API Gateway | Auth work, route mapping, payload size, throttling. | Latency, IntegrationLatency, 4xx, 5xx, count by route. |
Trim payloads, cache safe reads, remove slow mapping, tune limits. |
| ALB | Unhealthy targets, backend wait, connection setup, large responses. | Access log timing fields and target status codes. | Fix target health, keep connections warm, right size response payloads. |
| Fargate services | CPU saturation, memory pressure, thread pool waits, cold scaling. | Container Insights CPU, memory, network, task count, restart count. | Right size tasks, set autoscaling on useful signals, cap concurrency. |
| PostgreSQL | Slow plans, locks, connection pool exhaustion, chatty transactions. | Database load by waits, SQL, hosts, and users. | Add indexes, tune queries, shorten transactions, size the pool per task. |
| DynamoDB | Hot keys, throttled reads or writes, over broad reads. | Successful request latency, consumed capacity, read and write throttle events. | Fix key design, reduce item size, batch carefully, adjust capacity mode. |
Profile before fixing
My recommendation in "Profile before fixing" is to write the operational cost beside the architecture.
Traces tell you where the time went. Profiles tell you why the slow span is slow. For Service A and B, profile CPU, allocation, thread blocking, and database client waits during the same load window. For Java services, a short JFR recording during the p99 window is often enough to show lock contention, allocation pressure, blocked threads, or excessive serialization. For PostgreSQL, pair the service span with the SQL text, execution plan, wait event, and connection pool state. For DynamoDB, compare the trace span with table metrics so a slow read is not mistaken for application CPU.
Fix, rerun, prove
What I learnt around "Fix, rerun, prove" is that a clean diagram is not enough if the failure path is vague.
The proof is not "the code looks better." The proof is the same traffic pattern producing better numbers. Freeze the baseline first: request rate, payload mix, region, task count, database size, and sample window. Then fix one constraint at a time and rerun.
| Change | Before | After | Proof signal |
|---|---|---|---|
| Add missing PostgreSQL index and reduce transaction scope | Service B p95: 620 ms | Service B p95: 180 ms | RDS wait time moved away from read I/O and lock waits. |
| Set service timeouts and remove duplicate retry layer | Chain p99: 2.8 s | Chain p99: 1.1 s | Trace waterfall no longer shows retry amplification. |
| Fix DynamoDB partition key for inventory reads | Throttle spikes during peak | No throttles in same load window | DynamoDB throttle events flat while successful request latency holds. |
Distributed performance work ends when the trace, the profile, and the infrastructure metrics agree. If only one chart improves, keep digging. If the same load shows lower p95, lower p99, no hidden throttling, no new error rate, and lower database wait, the fix is real.
Fargate request chain example
I use "Fargate request chain example" to keep Finding Performance Bottlenecks in a Serverless Microservice Chain grounded in a real system, because abstract patterns are too easy to agree with and too hard to operate.
A common cloud incident is a checkout chain where API Gateway looks healthy, but p99 latency jumps when Service B waits on PostgreSQL locks and Service C hits a hot DynamoDB key. The fix starts by proving the slow span, not by guessing which service looks suspicious.
Trace correlation code
When I show "Trace correlation code", I want the code in Finding Performance Bottlenecks in a Serverless Microservice Chain to reveal the production decision, not just the syntax.
Add trace attributes at the service boundary so slow spans can be grouped:
Span span = Span.current();
span.setAttribute("service", "pricing-api");
span.setAttribute("orderId", orderId);
span.setAttribute("db.system", "postgresql");These span attributes make traces searchable by service, order, and database system. Without that context, distributed latency becomes guesswork.