DynamoDB Tail Latency: Cold Starts, Hot Partitions, and the Spikes Nobody Sees at p50

When people say DynamoDB cold start, I first separate client cold paths, network cold paths, and partition pressure. I recommend looking at tail latency and throttling reasons before blaming the database.

Here is the shape I would use for a checkout service. The service reads cart state, writes an order reservation, updates an inventory record, and reads a payment routing policy. Each request is small. Each operation looks harmless. Then a launch email goes out, one partition key becomes hot, a few containers create fresh clients, retry logic gets excited, and the user sees a two second checkout while the table dashboard still looks mostly green.

The trap in the phrase cold start

When I talk about "The trap in the phrase cold start", I am checking whether DynamoDB Tail Latency makes ownership, failure handling, or rollback clearer.

A cold request often pays for work the warm path already did: DNS, TLS, credentials, first connection, class loading, HTTP client pool setup, and the first SDK call. AWS documents that DynamoDB latency has to be split into service side latency and client side latency. That distinction matters because a slow user request can contain a fast DynamoDB service call plus slow client setup or retries.

Where the spike hides

My recommendation in "Where the spike hides" is to write the operational cost beside the architecture.

DynamoDB throttling is not one condition. AWS exposes different throttling reasons, and each points to a different fix. KeyRangeThroughputExceeded means a partition key range is hot. ProvisionedThroughputExceeded points to table or index provisioned capacity. AccountLimitExceeded means account level limits. MaxOnDemandThroughputExceeded means your configured on demand cap is doing exactly what you asked it to do.

The local secondary index bill

A local secondary index is useful when you need an alternate sort key under the same partition key, and you need strong consistency. The limits are the part teams forget: only five local secondary indexes, they must be created with the table, they share the base table partition key, and the item collection for one partition key value cannot exceed 10 GB when the table has a local secondary index.

PutItemResponse response = ddb.putItem(request -> request
    .tableName("Orders")
    .item(orderItem)
    .returnItemCollectionMetrics(ReturnItemCollectionMetrics.SIZE));

response.itemCollectionMetrics().forEach((table, metrics) -> {
    List<Double> estimate = metrics.sizeEstimateRangeGB();
    double upperEstimateGb = estimate.isEmpty() ? 0.0 : estimate.get(estimate.size() - 1);

    if (upperEstimateGb > 8.0) {
        log.warn("Order aggregate is close to the LSI item collection limit",
            kv("partitionKey", orderItem.get("customerId").s()),
            kv("estimatedGb", upperEstimateGb));
    }
});

This write asks DynamoDB to return item collection metrics, then warns before the partition key approaches the 10 GB local secondary index limit. The production decision is simple: split or remodel the entity before the table starts rejecting writes.

Trace the whole request

What I learnt around "Trace the whole request" is that a clean diagram is not enough if the failure path is vague.

For a browser or mobile request, use CloudWatch RUM for client timing, carry the trace context through API Gateway or the service entry point, instrument the backend with OpenTelemetry or AWS Distro for OpenTelemetry, and export traces to X-Ray. AWS has put the old X-Ray SDKs and daemon on a support timeline, so new instrumentation should prefer OpenTelemetry even when X-Ray remains the trace view.

Span span = tracer.spanBuilder("dynamodb.GetItem Cart")
    .setSpanKind(SpanKind.CLIENT)
    .setAttribute("db.system", "dynamodb")
    .setAttribute("aws.operation", "GetItem")
    .setAttribute("aws.dynamodb.table", "Cart")
    .startSpan();

try (Scope scope = span.makeCurrent()) {
    GetItemResponse response = ddb.getItem(request -> request
        .tableName("Cart")
        .key(Map.of("cartId", AttributeValue.fromS(cartId)))
        .returnConsumedCapacity(ReturnConsumedCapacity.TOTAL));

    span.setAttribute("aws.request_id", response.responseMetadata().requestId());
    span.setAttribute("aws.dynamodb.consumed_capacity",
        response.consumedCapacity().capacityUnits());
    return response.item();
} catch (DynamoDbException error) {
    span.recordException(error);
    span.setStatus(StatusCode.ERROR);
    span.setAttribute("aws.request_id", error.requestId());
    span.setAttribute("aws.error_code", error.awsErrorDetails().errorCode());
    throw error;
} finally {
    span.end();
}

OpenTelemetry instrumentation can create AWS SDK spans automatically, but this wrapper shows the fields I still want searchable: table, operation, request id, consumed capacity, and AWS error code. Exporting these spans to X-Ray makes the service map useful without pretending X-Ray is a packet level network sniffer.

Stop proxying DynamoDB

A custom DynamoDB proxy looks attractive when teams want private traffic, central logging, or shared credentials. It usually creates a new tail latency problem: one more hop, one more connection pool, one more retry layer, and one more place where head of line blocking can hide. For workloads inside a VPC in the same Region, start with a DynamoDB gateway VPC endpoint. If the access pattern needs AWS PrivateLink features such as private connectivity through an interface endpoint, use the DynamoDB interface endpoint instead of building a proxy.

DynamoDbClient ddb = DynamoDbClient.builder()
    .region(Region.US_EAST_1)
    .overrideConfiguration(config -> config
        .apiCallAttemptTimeout(Duration.ofMillis(180))
        .apiCallTimeout(Duration.ofMillis(500)))
    .build();

GetItemResponse response = ddb.getItem(request -> request
    .tableName("Cart")
    .key(Map.of("cartId", AttributeValue.fromS(cartId)))
    .consistentRead(false)
    .returnConsumedCapacity(ReturnConsumedCapacity.TOTAL));

The client is created once and reused. The attempt timeout bounds one try, while the total call timeout bounds the whole operation including retries. This prevents one slow DynamoDB path from quietly consuming the full API request budget.

What DynamoDB is great at

I use "What DynamoDB is great at" to test whether the pattern helps on a bad production day, not only in a design review.

DynamoDB is a strong fit for high scale key value and document access where the question is known before the table is designed: get cart by id, get account balance by account id, write idempotency record by request id, read session by token, update inventory by sku and location. The operational upside is real: no server fleet to patch, single digit millisecond service latency for well designed access patterns, global tables when multi Region replication is needed, streams for change capture, TTL for cleanup, and capacity modes that avoid many database administration chores.

What makes it hurt

The way I apply "What makes it hurt" is to make the tradeoff explicit before the implementation spreads.

The pain starts when the access pattern is not known, when one key gets too hot, when teams expect joins, when a local secondary index becomes a permanent table design decision, when a global secondary index throttles base table writes, when large items inflate cost and latency, or when retries are left to turn small throttles into large user waits. DynamoDB is excellent at predictable access. It is not a magic shield against poor cardinality, unbounded fan out, or vague query requirements.

The fix I would prove

• Put one trace id on the user request and carry it into every DynamoDB call.
• Log table, index, operation, partition key shape, request id, consumed capacity, retry count, and error code.
• Compare app latency with DynamoDB SuccessfulRequestLatency. If only the app is slow, look at client setup, retries, and network path.
• Check hot key metrics and Contributor Insights before increasing capacity blindly.
• Use gateway or interface VPC endpoints for private DynamoDB access. Remove unnecessary proxies from the hot path.
• Run the same load again and publish p50, p99, p99.9, p99.99, Max, throttle count, retry count, and error rate before calling the incident fixed.

The cleanest DynamoDB postmortem is not "we increased capacity." It is "we proved where the time was spent, changed the access path or client behavior, and reran the test until the tail moved." That is the difference between treating DynamoDB as a black box and operating it like a production dependency.