Apache Kafka is a widely used platform for streaming events between systems.
Core Concepts
- Event: a record of something that happened — an order placed, a sensor reading.
- Topic: a named stream of events.
- Partition: topics are split into partitions for parallelism; order is guaranteed within a partition.
- Producer: writes events.
- Consumer: reads events; consumer groups share partitions to scale.
- Retention: events are stored for a configured period, so consumers can replay them.
Uses
- Real-time analytics and dashboards.
- Feeding data warehouses and lakes.
- Event-driven microservices.
- Change data capture from databases.
- Real-time features for machine learning.
Considerations
- Operational complexity; managed services reduce it.
- Schema management with registries to prevent breaking changes.
- Delivery guarantees: at-least-once is common, so consumers should handle duplicates.
When Not to Use It
If data is only needed daily, batch processing is simpler and cheaper.