Tutorial: Kafka basic commands
Familiarize yourself with the basic commands of Kafka by creating a topic, producing messages, and consuming messages.
This tutorial assumes you are running a cluster based on TDP getting started, an easy-to-launch TDP environment for testing purposes. This deployment provides you with:
tdp_user, a user with the ability to
- An edge node accessible by SSH
Note: When using another TDP deployment than tdp-getting-started, some commands require some customization with your environment.
Before beginning the tutorial, connect to the cluster and authenticate yourself with
kinit using the following commands:
# Connect to edge-01.tdp vagrant ssh edge-01 # Switch user to tdp_user sudo su tdp_user # Authenticate the user with his Kerberos principal and password kinit -kt ~/tdp_user.keytab tdp_user@REALM.TDP
Use the following command to create a topic named
tdp-user-system-metrics with three partitions. Each partition will have three replicas, the
--command-config option is used to specify the client properties file for Kafka authentication.
# Create tdp-user-system-metrics topic /bin/kafka-topics.sh --create \ --topic tdp-user-system-metrics \ --replication-factor 3 \ --partitions 3 \ --command-config /etc/kafka/conf/client.properties
Print the description of your topic using the
# Describe tdp-user-system-metrics topic /bin/kafka-topics.sh --describe \ --topic tdp-user-system-metrics \ --command-config /etc/kafka/conf/client.properties
In this example, we will continuously capture system metrics every 5 seconds and send them to a Kafka topic called
To start producing the messages, run the following command:
while sleep 5; do ps -e -o pid,user,cmd,%cpu,%mem,vsz,rss,ni,state,etime --no-headers; done \ | /bin/kafka-console-producer.sh --topic tdp-user-system-metrics --producer.config /etc/kafka/conf/producer.properties
The command runs a loop every 5 seconds, utilizing the
ps command with the
-e option to list all processes. It captures crucial information such as the process ID (PID), user, command, CPU usage, memory usage, and other metrics. The header is excluded from the output by using the
--no-headers option. Finally, the captured metrics data is sent to the Kafka topic
tdp-user-system-metrics using the Kafka console producer and the provided producer configuration.
To begin consuming messages from the
tdp-user-system-metrics topic, open a new terminal and execute the following command:
# Start consuming messages /bin/kafka-console-consumer.sh --topic tdp-user-system-metrics \ --consumer.config /etc/kafka/conf/consumer.properties \ --from-beginning
To delete the topic:
# Delete tdp-user-system-metrics topic /bin/kafka-topics.sh --delete \ --topic tdp-user-system-metrics \ --command-config /etc/kafka/conf/client.properties