When I work with time series data, I start with the question window: last five minutes, last day, or last year. I recommend separating recent fast reads from long term rollups before cost gets out of control.
The answer changes over time
Time series data is about shape over time, not a single value in isolation.
How time turns readings into decisions
Time series data is about shape over time, not a single value in isolation.
Typical systems of record
Storage is only the start. I want to see evidence, ownership, and the decision the data supports.
Timestream, InfluxDB, TimescaleDB, Prometheus with Thanos or Cortex, Azure Data Explorer, Bigtable, BigQuery, ClickHouse, Druid.
Provider options in practice
| Provider | Typical services |
|---|---|
| AWS | Kinesis, Timestream, CloudWatch, Managed Prometheus, Redshift. |
| Azure | Event Hubs, Azure Data Explorer, Azure Monitor, Synapse. |
| Google Cloud | Pub/Sub, Dataflow, Cloud Monitoring, Bigtable, BigQuery. |
Real company pattern
My recommendation in this time series pattern is to keep raw windows and rollups connected, because people will ask both what happened now and what changed over months.
Uber real time infrastructure is a public example of time sensitive data at scale: trip, driver, marketplace, and operational signals need fast processing because the business changes minute by minute.
Real world operating example
I use the real workflow as the test. If it falls apart there, the pattern is still theory.
A payments API records latency, error count, queue depth, and authorization rate every minute. Recent data powers alerts; hourly rollups support capacity planning; long term data supports reliability reviews.
Small implementation example
The idea has to work inside an operating system with owners, alerts, and confused users.
with windows as (
select bin(time, 5m) as window_start,
service,
percentile(latency_ms, 95) as p95_latency
from api_metrics
where time > ago(30m)
group by bin(time, 5m), service
)
select service, count(*) as breach_windows
from windows
where p95_latency > 400
group by service
having count(*) >= 3;The query looks for repeated p95 latency breaches across five-minute windows. One spike is noise; three bad windows is an operational signal.
Operational traps
- Validate event time and arrival time separately.
- Define retention before the bill defines it for you.
- Roll up old data but preserve enough detail for incidents.