Apache Kafka is a system for passing messages between services through a shared, durable log. One service writes a message once, any number of others read it at their own pace, and nothing is lost if a reader is offline for a while.
The problem it solves
Picture an online shop. When a customer checks out, several parts of the system care: billing has to take the payment, the warehouse has to pack the order, email has to send a receipt, and analytics wants to count the sale.
The obvious design is for checkout to call each of them directly. That works with two services. With ten producers and ten consumers it becomes a web of connections where every service has to know the address, API and health of every other. If the email service is down when checkout calls it, the receipt is lost unless checkout adds its own retry logic. Adding a new consumer means changing the producer.
Kafka breaks that web. Producers write to Kafka and stop caring who reads. Consumers read from Kafka and never talk to the producers. Each side only needs to know one thing: the name of the topic.
Producers, topics and consumers
Three words cover most of Kafka:
- A producer is any program that writes messages, such as the checkout service.
- A topic is a named stream of related messages, such as
ordersorpayments. Producers write to a topic; they don't address a particular reader. - A consumer is any program that reads from one or more topics, such as the email service.
A message is a small record: an optional key, a value (often JSON), a timestamp and some headers. Kafka doesn't care what the value means. It stores bytes and hands them back.
The main difference from a classic message queue is what happens after a message is read. In a classic queue, a message is removed once a consumer acknowledges it. In Kafka, reading doesn't delete anything. Messages stay in the topic for a set retention period (seven days by default, configurable per topic), whether or not anyone has read them.
Wondering how this was made? How we make these