My own Apache Kafka Cheat Sheet

Apache Kafka is an open-source distributed event streaming platform used for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.

This cheat sheet includes the most common commands to install, deploy, administrate or operate an Apache Kafka cluster.

NOTE: Apache Kafka is the upstream project of Red Hat AMQ Streams, so these commands are also valid for this product.

Enjoy it !!!

Topics Management

Listing topics:

❯ ./bin/kafka-topics.sh --list \
  --bootstrap-server localhost:9092 \
  --exclude-internal

Creating topics:

❯ ./bin/kafka-topics.sh --create \
  --bootstrap-server localhost:9092 \
  --topic sample-topic --partitions 9 --replication-factor 3

Describing topics:

❯ ./bin/kafka-topics.sh --describe \
  --bootstrap-server localhost:9092 \
  --topic sample-topic

Deleting topics:

❯ ./bin/kafka-topics.sh --delete \
  --bootstrap-server localhost:9092 \
  --topic sample-topic	

Altering topics:

❯ ./bin/kafka-configs.sh --alter \
  --zookeeper localhost:2181 \
  --entity-type topics --entity-name sample-topic \
  --add-config min.insync.replicas=2

Topics Reassignment

Generate topics reassignment json file:

❯ ./bin/kafka-reassign-partitions.sh --generate \
  --bootstrap-server=localhost:9092 --zookeeper=localhost:2181 \
  --broker-list 0,1,2 --topics-to-move-json-file ./reassignment-topics.json

Execute topics reassignment:

❯ ./bin/kafka-reassign-partitions.sh --execute \
  --bootstrap-server=localhost:9092 --zookeeper=localhost:2181 \
  --broker-list 0,1,2 --topics-to-move-json-file ./reassignment-topics.json

Verify topics reassignment:

❯ ./bin/kafka-reassign-partitions.sh --verify \
  --bootstrap-server=localhost:9092 --zookeeper=localhost:2181 \
  --reassignment-json-file ./new-reassignment-topics.json

Sample Producers

Console Producer: This tool helps to read data from standard input and publish it to Kafka.

❯ ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic sample-topic

Performance Producer: This tool is used to verify the producer performance.

❯ ./bin/kafka-producer-perf-test.sh --topic sample-topic \
  --num-records 1000000 --record-size 1024 \
  --throughput -1 \
  --producer.config ./config/producer.properties --print-metrics

The producer.properties file will be similar to:

bootstrap.servers=localhost:9092
compression.type=none

In case the Apache Kafka requires SASL authentication with SCRAM-SHA-512, the following properties must be declared:

# SASL by plain connection
security.protocol=SASL_PLAINTEXT
# SASL by secured connection (SSL)
#security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username=user password=passw0rd;

NOTE: The latest semicolon in the sasl.jaas.config property is mandatory.

Sample command:

❯ ./bin/kafka-console-producer.sh --bootstrap-server localhost:9092 \
  --topic sample-topic \
  --producer-property security.protocol=SASL_PLAINTEXT \
  --producer-property sasl.mechanism=SCRAM-SHA-512 \
  --producer-property "sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username=user password=passw0rd;"

Sample Consumers

Console Consumer: This tool helps to read data from Kafka and show in console.

❯ ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 \
  --topic sample-topic --group group01 --from-beginning

Performance Consumer: This tool helps in performance test for consumer data.

❯ ./bin/kafka-consumer-perf-test.sh --broker-list localhost:9092 \
  --topic sample-topic \
  --group perf-consumer-01 --messages 1000000 --timeout 60000 \
  --reporting-interval 1000 --show-detailed-stats --threads 10 \
  --consumer.config ./config/consumer.properties

The consumer.properties file will be similar to:

bootstrap.servers=localhost:9092
group.id=consumer-group

In case the Apache Kafka requires SASL authentication with SCRAM-SHA-512, the following properties must be declared:

# SASL by plain connection
security.protocol=SASL_PLAINTEXT
# SASL by secured connection (SSL)
#security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username=user password=passw0rd;

Sample command:

❯ ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 \
  --topic sample-topic \
  --from-beginning \
  --max-messages 1 \
  --consumer-property security.protocol=SASL_PLAINTEXT \
  --consumer-property sasl.mechanism=SCRAM-SHA-512 \
  --consumer-property "sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username=user password=passw0rd;"

Consumer Group Management

Listing Consumer Groups:

❯ ./bin/kafka-consumer-groups.sh --list \
  --bootstrap-server localhost:9092 \
  --command-config ./config/consumer.properties

Describing a Consumer Group: This command gives a lot of information about a consumer group:

❯ ./bin/kafka-consumer-groups.sh --describe --group sample-consumer \
  --bootstrap-server localhost:9092 \
  --command-config ./config/consumer.properties

Reset Consumer Group

From time to time you need to reset the consumer groups to advance or rewind messages. The following commands help you to do it:

Advance to the latest offset of a consumer group:

❯ ./bin/kafka-consumer-groups.sh --reset-offsets --to-latest --execute \
  --topic sample-topic \ 
  --group sample-consumer \
  --bootstrap-server localhost:9092 \
  --command-config ./config/consumer.properties  

Rewind to the beginning:

❯ ./bin/kafka-consumer-groups.sh --reset-offsets --to-earliest --execute \
  --topic sample-topic \ 
  --group sample-consumer \
  --bootstrap-server localhost:9092 \
  --command-config ./config/consumer.properties  

Move to an specific offset (e.g: 100):

❯ ./bin/kafka-consumer-groups.sh --reset-offsets --to-offset 100 --execute \
  --topic sample-topic \ 
  --group sample-consumer \
  --bootstrap-server localhost:9092 \
  --command-config ./config/consumer.properties  