"We need real-time analytics" is one of the most common requests we hear — and one of the most expensive to implement without clarity. Before choosing technology, pin down what decision the freshness actually enables.
Define freshness by decision
- Sub-second: fraud checks, live pricing, in-app personalization
- Seconds to a minute: operational dashboards, logistics tracking
- Minutes to an hour: marketing performance, inventory views
- Daily: finance, planning and most executive reporting
Most organizations find that only a handful of use cases need true streaming. Everything else can run on micro-batches at a fraction of the cost and complexity.
A reference architecture
Capture changes from operational databases using CDC and publish events to Kafka. Use Flink or Spark Structured Streaming for stateful processing — joins, aggregations, deduplication. Land results both in a real-time OLAP store such as ClickHouse for low-latency queries, and in the lakehouse for history and reprocessing.
Design for replay
Streaming pipelines will have bugs. Retain enough event history to reprocess, make transformations deterministic, and make sinks idempotent. A pipeline you can safely replay is a pipeline you can safely change.
Watch the operational cost
Streaming systems run 24/7. Budget for on-call, schema management, consumer lag monitoring and backpressure handling. Managed services often pay for themselves here.
Real-time data is only valuable if someone — or something — acts on it in real time.