Some decisions need fresh data: fraud checks, operational dashboards, live personalisation.
Components
- Event sources: applications, devices, change data capture.
- Streaming platform: transports events.
- Stream processing: filters, enriches and aggregates in motion.
- Serving layer: real-time analytical databases or key-value stores for fast queries.
Patterns
- Streaming aggregation: maintain running counts and windows.
- Lambda-style: combine batch and streaming results.
- Streaming-first: process everything as streams, replaying for history.
Challenges
- Out-of-order and late events.
- Exactly-once processing semantics.
- Higher operational complexity and cost.
- Testing streaming logic.
Is Real-Time Needed?
Ask how fresh data must be for the decision. Many "real-time" requirements are satisfied by updates every few minutes, which simple micro-batch processing handles.
Start Narrow
Build real-time pipelines for specific use cases with clear value, not everything at once.