Introduction
In this article, we are going to learn about Apache Kafka and its overview.
Agenda
- What is Kafka
- Why Kafka
- Kafka Components
- Component Architecture
- Kafka APIs
- ZooKeeper
- Real World Examples
What is Kafka
Kafka is used to build distributed real-time data and streaming applications. It has a few features such as scalability, fault-tolerant, and event replay, capable of persisting messages for a certain duration.
Why Kafka
Kafka is often used to generate real-time metrics, log aggregation solutions, and stream processing. Mostly used to understand the customer journeys, so that we can recommend offers based on the customer click stream data.
Kafka Components
- Cluster
- Broker
- Producer
- Consumer
- Consumer Group
- Replication
- Topic
- Partition
- Offset
Component Architecture
![Kafka]()
Note: First Topic, then partition, and finally, offset. The offset is used to read the data from which location. Moreover, offset it incrementally by 1.
Kafka APIs
- Producer API => Allows an application to publish a stream of records into one or more topics
- Consumer API => Allows an application to consume or subscribe from one or more topics to process the stream record(s).
- Connector API => Reusable producers or consumers that connect Kafka topics to an existing database. For eg., If there are any changes in DB, trigger to Kafka and vice versa.
- Stream Processing API => Acts as a stream processor and basically acts as transforms input feed into desired output format feed.
ZooKeeper
It acts as a manager to coordinate all Kafka brokers. Its vital role is to notify consumers or producers if there are any issues or changes occurring in the Kafka broker, may be added new broker, or failure of the broker. so that the consumer or producer can decide to coordinate with the next available broker.
Real World Example
- Customer Analytics Data to recommend offers
- Black Boxing Day
- Understanding Customer Journey
- Credit Fraud Validation System
- Many more...
I hope, this article gives an idea about Kafka, Thanks for reading this, we will meet in the next session.