I use event streams when the business needs a durable timeline of what happened, not just a current row. I recommend designing the broker, schema, processors, lake storage, and serving projections as one governed path.
Facts in the order they happened
Event streams are most useful when consumers can replay the same facts and arrive at the same story.
How event streams drive reaction and replay
Event streams are most useful when consumers can replay the same facts and arrive at the same story.
Storage patterns I would expect
When I talk about "Storage patterns I would expect", I am checking whether Event Stream Data Architecture can be traced back, trusted, and used by someone making a decision.
Kafka, Kinesis, Event Hubs, Pub/Sub, Flink, Spark Structured Streaming, Druid, Pinot, ClickHouse, BigQuery, OpenSearch.
Useful cloud services
| Provider | Typical services |
|---|---|
| AWS | Kinesis, MSK, Glue Streaming, Lambda, OpenSearch, Redshift. |
| Azure | Event Hubs, Stream Analytics, Data Explorer, Synapse. |
| Google Cloud | Pub/Sub, Dataflow, BigQuery, Bigtable. |
Public example worth knowing
I use "Public example worth knowing" to keep Event Stream Data Architecture grounded in a real system, because abstract patterns are too easy to agree with and too hard to operate.
LinkedIn Kafka is the reference example. Kafka was created at LinkedIn for data pipeline needs and became the industry default mental model for durable, replayable event streams.
A business example
I use "A business example" to keep Event Stream Data Architecture grounded in a real system, because abstract patterns are too easy to agree with and too hard to operate.
A checkout system emits order placed, payment captured, item packed, and shipment created. Stream jobs update customer search, fraud features, warehouse facts, and operational alerts independently.
Operational code example
When I show "Operational code example", I want the code in Event Stream Data Architecture to reveal the production decision, not just the syntax.
void handle(OrderEvent event) {
if (processedEvents.exists(event.eventId())) return;
if (event.schemaVersion() < 3) {
deadLetter.publish(event, "unsupported schema");
return;
}
orderProjection.apply(event.orderId(), event.eventType(), event.occurredAtEpochMs());
processedEvents.save(event.eventId());
}The handler ignores duplicate events, rejects old schemas, updates the projection, and records the event id. That is the minimum production shape for safe stream consumption.
What I would check first
- Use event ids for idempotency.
- Treat schema evolution as part of the API.
- Design for replay before the first incident.