Kafka: From Zero to Topic with Producer and Consumer.

Introduction

The best way to introduce Kafka is by installing one. Currently I have a way to install a 3-node Kafka cluster using helm for kubernetes . In the future, I shall attempt to add steps for bare-metal and/or for other cloud vendors.

replicaCount: 3
nodeSelector: |
node-type: 2-cores
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
helm install -f ./config.yaml kafka bitnami/kafka -n ns1
To create a pod that you can use as a Kafka client run the following commands:kubectl run kafka-client --restart='Never' --image docker.io/bitnami/kafka:2.8.0-debian-10-r27 --namespace ns1 --command -- sleep infinity
kubectl exec --tty -i kafka-client --namespace ns1 -- bash
PRODUCER:
kafka-console-producer.sh \
--broker-list kafka-0.kafka-headless.ns1.svc.cluster.local:9092,kafka-1.kafka-headless.ns1.svc.cluster.local:9092,kafka-2.kafka-headless.ns1.svc.cluster.local:9092 \
--topic test
CONSUMER:
kafka-console-consumer.sh \
--bootstrap-server kafka.ns1.svc.cluster.local:9092 \
--topic test \
--from-beginning
kafka-zookeeper.ns1:2181
kafka-zookeeper.ns1:2888
kafka-zookeeper.ns1:3888

Commands

Once installed, here are some useful commands to navigate around kafka topics.

View topics

./kafka-topics.sh  --bootstrap-server kafka-headless.ns1.svc.cluster.local:9092 --list

Produce messages

./kafka-console-producer.sh --bootstrap-server kafka-0.kafka-headless.ns1.svc.cluster.local:9092,kafka-1.kafka-headless.ns1.svc.cluster.local:9092,kafka-2.kafka-headless.ns1.svc.cluster.local:9092 --topic test

Consume messages

./kafka-console-consumer.sh  --bootstrap-server kafka-headless.ns1.svc.cluster.local:9092 --topic test --group gp1

Describe topic

./kafka-topics.sh --describe --topic test --bootstrap-server kafka-headless.ns1.svc.cluster.local:9092
Topic: test     TopicId: PgZj2P-zSj2wPDLS_g6wyQ PartitionCount: 5       ReplicationFactor: 1    Configs: flush.ms=1000,segment.bytes=1073741824,retention.ms=86400000,flush.messages=10000,max.message.bytes=1000012,retention.bytes=1073741824
Topic: test Partition: 0 Leader: 1 Replicas: 1 Isr: 1
Topic: test Partition: 1 Leader: 2 Replicas: 2 Isr: 2
Topic: test Partition: 2 Leader: 0 Replicas: 0 Isr: 0
Topic: test Partition: 3 Leader: 1 Replicas: 1 Isr: 1
Topic: test Partition: 4 Leader: 2 Replicas: 2 Isr: 2

Create topic

One can create / alter a topic explicitly using the following

./kafka-topic.sh --create --topic test --bootstrap-server kafka-headless.ns1.svc.cluster.local:9092
./kafka-topics.sh --bootstrap-server kafka-headless.ns1.svc.cluster.local:9092 --alter --topic test --partitions 5
./kafka-configs.sh --bootstrap-server kafka-headless.ns1.svc.cluster.local:9092 --entity-type topics --entity-name test --alter --add-config retention.ms=86400000

Message Counts

In order to get message counts across partitions for a particular topic,

./kafka-run-class.sh kafka.tools.GetOffsetShell --topic test --broker-list kafka-headless.ns1.svc.cluster.local:9092
test:0:15
test:1:5
test:2:4
test:3:1
test:4:2

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store