• Create kafka topic
• View all topic lists
• View the specified topic information
• Console to topic Production Data
• Console consumes topic data
• View the maximum (minimum) offset of a partition in topic
• Increase the number of topic partitions
• Delete topic, use cautiously, only delete metadata in zookeeper, message file must be deleted manually
• View topic Consumption Progress

 

Create kafka topic

bin/kafka-topics.sh --zookeeper node01:2181 --create --topic t_cdr --partitions 30  --replication-factor 2

Note: Partitions specify the number of topic partitions, replication-factor specifies the number of copies of each topic partition

• partitions partition number:
  • Partitions: Number of partitions, controlling how many log s topic will slice into. You can display the number of num.partitions configurations in broker(server.properties) if not specified
  • Although increasing the number of partitions can provide the throughput of the kafka cluster, too many partitions or too many partitions on a single server will increase the risk of unavailability and delay. Because of the number of partitions, it means opening more file handles, increasing point-to-point latency, and increasing the memory consumption of the client.
  • The number of partitions also limits the parallelism of consumer, that is, the number of threads in parallel consumer messages cannot be greater than the number of partitions.
  • The number of partitions also limits the number of partitions that producer sends messages to. If the partition is set to 1 when topic is created, it will be wrong for producer to specify the number of partitions to be 2 or more when sending messages through a custom partitioning method; in this case, alter-partitions can be used to increase the number of partitions.
• replication-factor copy
  • The replication factor control message is stored on several brokers (servers), generally equal to the number of brokers.
  • The number of default.replication.factor configurations in broker(server.properties) is used if no specified or non-existent topic production message is displayed at creation time

 

View all topic lists

bin/kafka-topics.sh --zookeeper node01:2181 --list

 

View the specified topic information

bin/kafka-topics.sh --zookeeper node01:2181 --describe --topic t_cdr

 

Console to topic Production Data

bin/kafka-console-producer.sh --broker-list node86:9092 --topic t_cdr

 

Console consumes topic data

bin/kafka-console-consumer.sh  --zookeeper node01:2181  --topic t_cdr --from-beginning

 

View the maximum (minimum) offset of a partition in topic

bin/kafka-run-class.sh kafka.tools.GetOffsetShell --topic hive-mdatabase-hostsltable  --time -1 --broker-list node86:9092 --partitions 0

Note: When time is - 1, it means maximum, and when time is - 2, it means minimum.

 

Increase the number of topic partitions

Adding 10 partitions to topic t_cdr

bin/kafka-topics.sh --zookeeper node01:2181  --alter --topic t_cdr --partitions 10

 

Delete topic, use cautiously, only delete metadata in zookeeper, message file must be deleted manually

bin/kafka-run-class.sh kafka.admin.DeleteTopicCommand --zookeeper node01:2181 --topic t_cdr

 

View topic Consumption Progress

This will show the offset of consumer group, which must be a parameter of -- group, not specified -- topic, default to all topics.

Displays the: Consumer Group, Topic, Partitions, Offset, logSize, Lag, Owner for the specified set of Topics and Consumer Group

bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker

required argument: [group] 
Option Description 
------ ----------- 
--broker-info Print broker info 
--group Consumer group. 
--help Print this message. 
--topic Comma-separated list of consumer 
   topics (all topics if absent). 
--zkconnect ZooKeeper connect string. (default: localhost:2181)

Example,

bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group pv

Group           Topic              Pid Offset   logSize    Lag    Owner 
pv              page_visits        0   21       21         0      none 
pv              page_visits        1   19       19         0      none 
pv              page_visits        2   20       20         0      none