Apache Kafka vs Apache Storm

Question

Apache Kafka: Distributed messaging system
Apache Storm: Real Time Message Processing

How we can use both technologies in a real-time data pipeline for processing

Answer 1

Kafka and Storm have a slightly different purpose:

Kafka is a distributed message broker which can handle big amount of messages per second. It uses publish-subscribe paradigm and relies on topics and partitions. Kafka uses Zookeeper to share and save state between brokers. So Kafka is basically responsible for transferring messages from one machine to another.

Storm is a scalable, fault-tolerant, real-time analytic system (think like Hadoop in realtime). It consumes data from sources (Spouts) and passes it to pipeline (Bolts). You can combine them in the topology. So Storm is basically a computation unit (aggregation, machine learning).

---

But you can use them together: for example your application uses kafka to send data to other servers which uses storm to make some computation on it.