Kafka: From Zero to Topic with Producer and Consumer.

Introduction

replicaCount: 3
nodeSelector: |
node-type: 2-cores
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
helm install -f ./config.yaml kafka bitnami/kafka -n ns1
To create a pod that you can use as a Kafka client run the following commands:kubectl run kafka-client --restart='Never' --image docker.io/bitnami/kafka:2.8.0-debian-10-r27 --namespace ns1 --command -- sleep infinity
kubectl exec --tty -i kafka-client --namespace ns1 -- bash
PRODUCER:
kafka-console-producer.sh \
--broker-list kafka-0.kafka-headless.ns1.svc.cluster.local:9092,kafka-1.kafka-headless.ns1.svc.cluster.local:9092,kafka-2.kafka-headless.ns1.svc.cluster.local:9092 \
--topic test
CONSUMER:
kafka-console-consumer.sh \
--bootstrap-server kafka.ns1.svc.cluster.local:9092 \
--topic test \
--from-beginning
kafka-zookeeper.ns1:2181
kafka-zookeeper.ns1:2888
kafka-zookeeper.ns1:3888

Commands

View topics

./kafka-topics.sh  --bootstrap-server kafka-headless.ns1.svc.cluster.local:9092 --list

Produce messages

./kafka-console-producer.sh --bootstrap-server kafka-0.kafka-headless.ns1.svc.cluster.local:9092,kafka-1.kafka-headless.ns1.svc.cluster.local:9092,kafka-2.kafka-headless.ns1.svc.cluster.local:9092 --topic test

Consume messages

./kafka-console-consumer.sh  --bootstrap-server kafka-headless.ns1.svc.cluster.local:9092 --topic test --group gp1

Describe topic

./kafka-topics.sh --describe --topic test --bootstrap-server kafka-headless.ns1.svc.cluster.local:9092
Topic: test     TopicId: PgZj2P-zSj2wPDLS_g6wyQ PartitionCount: 5       ReplicationFactor: 1    Configs: flush.ms=1000,segment.bytes=1073741824,retention.ms=86400000,flush.messages=10000,max.message.bytes=1000012,retention.bytes=1073741824
Topic: test Partition: 0 Leader: 1 Replicas: 1 Isr: 1
Topic: test Partition: 1 Leader: 2 Replicas: 2 Isr: 2
Topic: test Partition: 2 Leader: 0 Replicas: 0 Isr: 0
Topic: test Partition: 3 Leader: 1 Replicas: 1 Isr: 1
Topic: test Partition: 4 Leader: 2 Replicas: 2 Isr: 2

Create topic

./kafka-topic.sh --create --topic test --bootstrap-server kafka-headless.ns1.svc.cluster.local:9092
./kafka-topics.sh --bootstrap-server kafka-headless.ns1.svc.cluster.local:9092 --alter --topic test --partitions 5
./kafka-configs.sh --bootstrap-server kafka-headless.ns1.svc.cluster.local:9092 --entity-type topics --entity-name test --alter --add-config retention.ms=86400000

Message Counts

./kafka-run-class.sh kafka.tools.GetOffsetShell --topic test --broker-list kafka-headless.ns1.svc.cluster.local:9092
test:0:15
test:1:5
test:2:4
test:3:1
test:4:2

--

--

--

Listener and reader

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Why Hackathons Make You a Better Developer

10 Minute Beginner’s Guide to Snowflake Cloud Data Warehouse

Host Your Static Website in Azure as a Docker Container

DJ Rob Swift Workshop Archives

Hadoop days are over..long live big data

Episode 006: Home is where the App is.

STREAMING | 2021 IFBB Pro League/NPC Heart of Texas’ Livestream | Live_HD

Teddy Bear Museum — VR Experience Design

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Programmer

Programmer

Listener and reader

More from Medium

Migrating to AWS Managed Streaming Kafka (MSK)

Legacy to MSK

Helm incubator kafka setup with SSL auth

Mastering Apache Kafka on Kubernetes — Strimzi K8s operator

Simple authN and authZ on Kafka for Users