Derived Data Architecture: Metrics, Features, and Reusable Results

When I build derived data, I care less about the calculation looking clever and more about whether I can reproduce it. I recommend treating metrics, aggregates, and features as products with lineage, tests, and owners.

Where computed answers live

When I talk about "Where computed answers live", I am checking whether Derived Data Architecture can be traced back, trusted, and used by someone making a decision.

Warehouses, lakehouses, dbt models, feature stores, Redis, DynamoDB, Cassandra, Bigtable, Pinot, Druid, ClickHouse, vector indexes when features support similarity.

Cloud metric and feature services

Reusable metric pattern

My recommendation in "Reusable metric pattern" is to keep the raw source close enough that the answer can be explained later.

Feature stores at Uber and Airbnb are public examples of derived data becoming infrastructure. The lesson is that ML features and business metrics need definitions, ownership, offline history, and online serving.

Daily feature example

I use "Daily feature example" to keep Derived Data Architecture grounded in a real system, because abstract patterns are too easy to agree with and too hard to operate.

A risk platform derives failed login count, device age, recent chargeback count, and customer tenure. The same features are used for model training and low latency checkout risk scoring.

Certified metric code

When I show "Certified metric code", I want the code in Derived Data Architecture to reveal the production decision, not just the syntax.

insert into features.customer_login_risk
select customer_id,
       count_if(event_type = 'FAILED_LOGIN') as failed_logins_24h,
       max(event_time) as last_event_time,
       case when count_if(event_type = 'FAILED_LOGIN') >= 5
            then 'STEP_UP_AUTH'
            else 'ALLOW'
       end as decision,
       'login_risk_v3' as feature_version
from login_events
where event_time >= :as_of - interval '24 hours'
  and event_time < :as_of
group by customer_id;

This turns raw login events into a versioned risk feature and a concrete decision, so downstream services reuse the same logic instead of recalculating it differently.

Derived data traps

• Do not let metric definitions fork by dashboard.
• Test derived data like application code.
• Keep point in time correctness for ML features.

Related metric reads

• Everything Data: The Asset Behind Modern Software
• YCSB: A Useful Benchmark for Data Stores
• Finding Performance Bottlenecks in a Serverless Microservice Chain