Apache Kafka Setup
If you have ever used LinkedIn, Uber, PayPal, Netflix, Twitter, Pinterest etc., then congratulations! you have experienced Apache Kafka service. Yes, these all big firms are using this open source system to tackle the complexity of their messaging model. Kafka is capable of tolerating millions of messages per secind and in a very efficient way. And it's very easy to set up Apache Kafka on a Windows machine.
Ingredients
- Download Java
- Setup Java environment
- Download Apache Kafka
Prerequisite
- Download and unzip Apache Kafka library.
- Configure Apache Kafka.
- Set environment variables for Kafka.
- Open system properties.
- Click "Environment Variables" and add new "System Variable" as follow.
- Click "New" and create "KAFKA_HOME" variable
- Add "%KAFKA_HOME%\bin" in "Path" variable.
Recipe
To set up the basic Kafka messaging model, we need to understand a couple of terms. This article gives you an extremely basic overview of Apache Kafka.
A couple of the most important parts of Apache Kafka.
- Zookeeper
- Kafka Server
- Topic (Channel)
- Producer (Sender)
- Consumer (Receiver)
Start Zookeeper Server
zookeeper-server.bat ../../config/zookeeper.properties
Start Kafka Server
kafka-server.bat ../../config/server.properties
Create Topic
kafka-topics.bat --create --topic KafkaDemo --zookeeper localhost:2181 --replication-factor 1 --partitions 1
We can see an entry of created topics in the running Kafka server.
Create Producer (Sender)
kafka-console-producer.bat --broker-list localhost:9092 --topic KafkaDemo
Create Consumer(Receiver)
kafka-console-consumer.bat --zookeeper localhost:2181 --topic KafkaDemo --from-beginning
Write a message in Producer console, hit enter and we will receive it in the consumer console.
And here, we are done with an extremely basic Apache Kafka messaging setup. I have not covered Partition, Broker, Cluster and many more topics in this blog. The intention was just to explain how quickly we can set up the Kafka messaging system.