I use event streams when the business needs a durable timeline of what happened, not just a current row. I recommend designing the broker, schema, processors, lake storage, and serving projections as one governed path.
Facts in the order they happened
Event streams are most useful when consumers can replay the same facts and arrive at the same story.
How event streams drive reaction and replay
Event streams are most useful when consumers can replay the same facts and arrive at the same story.
Storage patterns I would expect
I judge this data by whether it supports a real decision at the moment someone needs it.
Kafka, Kinesis, Event Hubs, Pub/Sub, Flink, Spark Structured Streaming, Druid, Pinot, ClickHouse, BigQuery, OpenSearch.
Useful cloud services
| Provider | Typical services |
|---|---|
| AWS | Kinesis, MSK, Glue Streaming, Lambda, OpenSearch, Redshift. |
| Azure | Event Hubs, Stream Analytics, Data Explorer, Synapse. |
| Google Cloud | Pub/Sub, Dataflow, BigQuery, Bigtable. |
Public example worth knowing
The point here is to move from concept to operation: who uses it, what breaks, and what decision changes.
LinkedIn Kafka is the reference example. Kafka was created at LinkedIn for data pipeline needs and became the industry default mental model for durable, replayable event streams.
A business example
The point here is to move from concept to operation: who uses it, what breaks, and what decision changes.
A checkout system emits order placed, payment captured, item packed, and shipment created. Stream jobs update customer search, fraud features, warehouse facts, and operational alerts independently.
Operational code example
Good example code should reveal the production rule the system is depending on.
void handle(OrderEvent event) {
if (processedEvents.exists(event.eventId())) return;
if (event.schemaVersion() < 3) {
deadLetter.publish(event, "unsupported schema");
return;
}
orderProjection.apply(event.orderId(), event.eventType(), event.occurredAtEpochMs());
processedEvents.save(event.eventId());
}The handler ignores duplicate events, rejects old schemas, updates the projection, and records the event id. That is the minimum production shape for safe stream consumption.
What I would check first
- Use event ids for idempotency.
- Treat schema evolution as part of the API.
- Design for replay before the first incident.