kafka3.x learning

This article mainly learns and records some knowledge of kafka3.x.

---

---

I. Overview

Kafka is a distributed message queue based on publish/subscribe mode, which is mainly used in the field of big data real-time processing.

Two modes of message queues:

• Point-to-point mode:

A producer corresponds to a consumer, and the consumer actively pulls the message, and after confirming the receipt of the message, deletes the corresponding message.

• Publish and subscribe mode:

A producer can correspond to multiple consumers, and multiple consumers are independent of each other. Messages are classified according to topics. Consumers actively pull messages without deleting them, and other consumers can still consume the message again.

2. Kafka infrastructure

Architecture features:

• To facilitate expansion and improve throughput, a topic is divided into multiple partitions.
• In order to cooperate with the design of multiple partitions, the concept of consumer group (consumer group) is introduced.
• To improve availability, a replica mechanism is added to each topic partition, and there is a distinction between leafer and follower between replicas.
• Before Kafka2.8, it must rely on zk, which records the information of all nodes in the Kafka cluster, as well as the leader and follower in each topic partition.

3. Kafka-3.1.0 cluster installation

Download the kafka-3.1.0 compressed package, decompress it and complete the installation. Modify the following configuration files. After modification, distribute them to node-01, node-02, and node-03 machines to complete the installation of the kafka cluster:

// kafka每个broker的身份标识，必须唯一
broker.id=2
// kafka数据存放地址
log.dirs=/orkasgb/data/kafka
// 指定zookeeper的地址
zookeeper.connect=node-01:2181,node-02:2181,node-03:2181

// 安装好集群之后，分别执行以下命令启动kafka
$KAFKA_HOME/bin/kafka-server-start.sh -daemon $KAFKA_HOME/config/server.properties

4. Basic commands of Kafka

• kafka-topics.sh:

Basic parameters	illustrate
–bootstrap-server <String: server to connect to>	Specify the kafka cluster address, there can be multiple, required, the format is: node-01:9092, node-02:9092
–topic <String: topic>	Specify the topic of the link (topic)
–create	create topic
–delete	delete topic
–alter	To modify the theme, generally only the number of partitions can be modified
–list	list all topics
–describe	Describe the details of the current topic
–partitions <Integer: # of partitions>	Specify the number of partitions when creating a topic, required
–replication-factor <Integer: replication factor>	Specify the number of replicas when creating a topic, required
–config <String: name=value>	Specify the configuration file location

# 创建一个topic-orkasgb-test主题，分区数为2，副本数为3
[root@node-01 ~]# kafka-topics.sh --bootstrap-server node-01:9092,node-02:9092 --topic topic-orkasgb-test --create --partitions 2 --replication-factor 3
Created topic topic-orkasgb-test.
[root@node-01 ~]#

# 查看当前kafka集群中的topic
[root@node-01 ~]# kafka-topics.sh --bootstrap-server node-01:9092,node-02:9092 --list
__consumer_offsets
topic-orkasgb-test
[root@node-01 ~]# 

# 查看topic-orkasgb-test主题的详细信息
[root@node-01 ~]# kafka-topics.sh --bootstrap-server node-01:9092,node-02:9092 --topic topic-orkasgb-test --describe
Topic: topic-orkasgb-test       TopicId: k6OqKVFcQgCFrahkJOyWbQ PartitionCount: 2       ReplicationFactor: 3    Configs: segment.bytes=1073741824
        Topic: topic-orkasgb-test       Partition: 0    Leader: 2       Replicas: 2,1,3 Isr: 2,1,3
        Topic: topic-orkasgb-test       Partition: 1    Leader: 3       Replicas: 3,2,1 Isr: 3,2,1
[root@node-01 ~]# 

• kafka-console-producer.sh :

Basic parameters	illustrate
–bootstrap-server <String: server to connect to>	Specify the kafka cluster address, there can be multiple, required, the format is: node-01:9092, node-02:9092
–topic <String: topic>	Specify the topic of the link (topic)
–producer.config <String: config file>	Specify the configuration file location

# 创建一个消费者，将数据发送到topic-orkasgb-test主题中
[root@node-03 bin]# kafka-console-producer.sh --bootstrap-server node-01:9092,node-02:9092 --topic topic-orkasgb-test --producer.config /orkasgb/software/kafka-3.1.0/config/producer.properties
>hello kafka
>

• kafka-console-consumer.sh ：

Basic parameters	illustrate
–bootstrap-server <String: server to connect to>	Specify the kafka cluster address, there can be multiple, required, the format is: node-01:9092, node-02:9092
–topic <String: topic>	Specify the topic of the link (topic)
–consumer.config <String: config file>	Specify the configuration file location

# 创建一个消费者，用于消费发往topic-orkasgb-test主题中的数据
[root@node-01 ~]# kafka-console-consumer.sh --bootstrap-server node-01:9092,node-02:9092 --topic topic-orkasgb-test --consumer.config /orkasgb/software/kafka-3.1.0/config/consumer.properties
hello kafka

• kafka-consumer-groups.sh:

# 列出整个集群中的消费组
[root@node-02 bin]# kafka-consumer-groups.sh --bootstrap-server node-01:9092,node-02:9092 --list
test-consumer-group
[root@node-02 bin]# 
# 查看test-consumer-group属组中的主题消费情况
[root@node-02 bin]# kafka-consumer-groups.sh --bootstrap-server node-01:9092,node-02:9092 --group test-consumer-group  --describe

GROUP               TOPIC              PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                           HOST             CLIENT-ID
test-consumer-group topic-orkasgb-test 0          0               0               0               console-consumer-b234bb08-4374-498d-9cfe-192de9f9cdd3 /192.168.137.223 console-consumer
test-consumer-group topic-orkasgb-test 1          1               1               0               console-consumer-b234bb08-4374-498d-9cfe-192de9f9cdd3 /192.168.137.223 console-consumer
[root@node-02 bin]# 

5. Kafka producer sending process

• batch.size: used to control the data size, when the data size reaches this value, the Sender thread will be awakened to send data, the default is 16k.
• linger.ms: Data sending delay setting. When the data has not reached the batch.size, but has reached the delay time (linger.ms), the Sender thread will also be woken up to send data. The default is 0, no delay.
• ack response:
  • 0: After the producer sends data, it immediately returns a response to the client, regardless of whether the leader has placed the order.
  • 1: After the producer sends the data, it needs to wait for the leader to place the disk and the ack response is successful before returning the response to the client.
  • -1 (all): After the producer sends data, it needs to wait for both the leader and follower to be processed and the ack response is successful before returning a response to the client.

Code for sending messages asynchronously:

	@Test
	public void producter() throws ExecutionException, InterruptedException {
    
    
		// 创建生产者配置文件
		HashMap<String, Object> config = new HashMap<>();
		// 集群地址
		config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "node-01:9092,node-02:9092,node-03:9092");
		// KEY的序列化器
		config.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
		// VALUE的序列化器
		config.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);

		// 创建Producter对象
		DefaultKafkaProducerFactory<String, String> producerFactory = new DefaultKafkaProducerFactory<>(config);
		Producer<String, String> producer = producerFactory.createProducer();
		for (int i = 0; i < 5; i++) {
    
    
            // 同步发送数据
//			RecordMetadata recordMetadata = producer.send(new ProducerRecord<>("topic-orkasgb-test", "test" + i)).get();
//			System.out.println(recordMetadata.toString());
            			// 同步发送数据
			// RecordMetadata recordMetadata = producer.send(new ProducerRecord<>("topic-orkasgb-test", "test" + i)).get();
			// System.out.println(recordMetadata.toString());

			// 异步发送数据
			Future<RecordMetadata> send = producer.send(new ProducerRecord<>("topic-orkasgb-test", "哈哈哈哈" + i));
            producer.flush();
			System.out.println(send.toString());

			// 带回调函数的异步发送数据
//			Future<RecordMetadata> send = producer.send(new ProducerRecord<>("topic-orkasgb-test", "哈哈哈哈" + i), new Callback() {
    
    
//				@Override
//				public void onCompletion(RecordMetadata metadata, Exception exception) {
    
    
//
//				}
//			});
//			System.out.println(send.toString());
		}	
		producer.close();
	}

6. Producer partition strategy

The kafka producer has a default partition strategy. If the partition is specified in the record, the specified partition strategy will be used. If the partition is not specified but the key is specified when sending the message, the partition will be selected according to the hash of the key. If no partition or key is specified , choose a sticky partitioning strategy that changes only when the batch is full.

• Specify partition number: When specifying a partition, send the message directly to the specified partition. For example, if the specified partition is 0, the message will be sent directly to partition 0.
• The partition number is not specified but the key is specified: When the partition number is not specified but the key is specified, the partition number will be calculated based on the remainder of the hash value of the key and the partition number of the topic. For example, if the hash value of key1 is 4, the hash value of key2 is 5, and the partition number of topic is 2, then the message corresponding to key1 is directly sent to partition 0, and the message corresponding to key2 is directly sent to partition 1.
• No partition number or key specified: When neither the partition number nor the key is specified, Kafka adopts the sticky partition strategy internally, randomly selects a partition number, and sends all the messages of this batch to the partition as much as possible. Partition, until the bach.size of the partition is full or the innerger.ms time is up, another partition number will be randomly selected again (must be inconsistent with the partition number selected last time). For example, if the partition number selected last time is 1, then another partition number will be randomly selected again until the bach.size of the partition is full or the innerger.ms time is up. Until it is inconsistent with the last time.

Custom partitioner:

// 在生产者配置文件中指定自定义分区器
config.put(ProducerConfig.PARTITIONER_CLASS_CONFIG, UserPartition.class);

/**
 * 自定义分区器，继承Partitioner类，重写partition方法即可。
 */
public class UserPartition implements Partitioner {
    
    
	@Override
	public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) {
    
    
		// 如果消息中包含了orkasgb字符串，就将这条消息发送到0好分区，否则就发送到1号分区
		return StringUtils.contains(value.toString(), "orkasgb") ? 0 : 1;
	}

	@Override
	public void close() {
    
    

	}

	@Override
	public void configure(Map<String, ?> configs) {
    
    

	}
}

Seven, the principle of idempotency

Idempotency means that no matter how many pieces of duplicate data are sent by the product to the broker, the broker will only persist one piece of data. The standard for judging data duplication by idempotency is that data with the same key as <PID, Partiton, SeqNumber> can be considered as duplicate data. Among them, PID is that the kafka cluster is not started once, and a PID will be reassigned. Partiton refers to the partition number, and SeqNumber is a monotonically increasing sequence. According to this standard, idempotence can ensure that data in a single partition is not repeated, and idempotence (enable.idempotence) is enabled by default.

Eight, kafka transaction principle

• To start a transaction, idempotency must be enabled, and the bottom layer of a transaction is idempotence.
• _transaction_state-partition-Leader is a special topic for storing transaction information, with 50 partitions by default.
• The main transaction coordinator is elected inside the kafka cluster. The election method is: according to the hash value %50 of the transaction ID (transational.id), calculate which partition the transaction belongs to. The broker node where the partition leader is located is the transaction coordinator corresponding to the transaction ID (transational.id).
• When the product starts a transaction, it needs to specify a unique transaction ID. After the cluster is restarted, the unfinished transaction will continue to be processed according to this transaction ID.
• The product will request a product id from the Kafka cluster, which is mainly used for idempotency.
• The producter sends a message to the cluster to topicA-partition0 and sends a commit request.
• After the transaction coordinator receives the commit request, it persists the commit request into the transaction topic, and also sends a commit request to topicA-partition0, and judges that the message is processed successfully.
• After topicA-partition0 returns a successfully processed message, the transaction coordinator will persist the message successfully processed information to _transnnaction_state-partition-Leader.

9. Data order/disorder

• The Kafka data is in order to ensure the internal order of a single partition.
  1. Before kafka1.x, to ensure the internal order of the data single partition, you only need to set max.in.flight.requests.per.connection=1.
  2. After kafka1.x, it is necessary to ensure the internal order of the data single partition. If idempotence is not enabled, you only need to set max.in.flight.requests.per.connection=1. If idempotence is enabled, you need to set max.in.flight.requests.per.connection<=5, (the default is 5). Because after idempotence is turned on, the kafka cluster will cache 5 (max.in.flight.requests.per.connection) requests. Under normal circumstances, the requests are sent to the kafka cluster in order, but if the order is disordered, the That is to say, the data is out of order, then the kafak cluster will wait until all 5 requests are sent, and the kafka cluster will reorder the 5 requests internally. When sorting, the monotonically increasing feature of SeqNumber in idempotence will be used, so Kafka can ensure the internal order of the data single partition.

10. Working principle of kafka broker

1. After the broker is started, it will go to zookeeper to register information, and generate a /kafka/brokers/ids information in zookeeper, which stores all broker information.
2. Each topic replica corresponds to a controller, so a controller boss must be selected, and finally the controller boss monitors the changes of each broker node, and will assist the broker to elect a new topic replica leader in the future. The way to elect the controller boss is to pre-register the /kafka/controller node in zookeeper. Whoever registers first will be the controller boss.
3. The controller boss uploads the broker information in the cluster to zookeeper, and generates /kafka/brokers/topics/[corresponding topic]/partitions/[corresponding partition number]/state, which stores the copy of the topic under the partition Information, including leader and surviving follower information (isr).
4. If the leader in the partition of a topic in the node hangs up, and the controller boss listens to this change, it will update /kafka/brokers/topics/[corresponding topic]/partitions/[corresponding partition number] /state information, and start re-electing the leader. The election rule is, based on the order of nodes in ar, the node existing in isr is the new leader. For example, the leader is 1, ar[1,3,2], is[1,3,2], if the leader is 1 and dies, then the new leader will be 3.

11. Service/retirement of nodes

• Commission a new node:

  1. Create a new kafka node or clone a kafka node from the previous machine (note that if it is a cloned machine, you need to delete the data and log directories of kafka, and modify the broker.id), and start it.
  2. Create a xxxx.json file to load balance the original theme.

  {
        
        
      "topics" : [
          {
        
        "topic" : "xxxx"}
      ],
      "version" : 1
  }
  

  1. Generate a load balancing plan according to xxxx.json.

  kafka-reassign-partitions.sh --bootstrap-server node-01:9092 --topics-to-move-json-file xxxx.json --broker-list "0,1,2,3" --generate
  

  1. Save the generated load balancing plan as xx.json file.
  2. Execute the load balancing plan according to the xx.json file.

  kafka-reassign-partitions.sh --bootstrap-server node-01:9092 --reassignment-json-file xx.json --execute
  

  1. Verify that the load balancing plan is executed.

  kafka-reassign-partitions.sh --bootstrap-server node-01:9092 --reassignment-json-file xx.json --verify
  

• Decommission an old node:

The idea is to transfer the data on the old node to other nodes and delete the node. The method is similar to the way of serving the new node. When generating the load balancing plan, set --broker-list "0,1,2, The broker.id to be decommissioned in 3" is canceled, and the other steps are executed in order to transfer the data to other nodes.

12. Kafka troubleshooting

Follower Troubleshooting

• LEO (Log End Offset): The last offset of each copy, that is, the latest offset + 1.
• HW: Smallest LEO among all replicas.

Follower troubleshooting steps:

1. Remove the failed Follower node from the isr.
2. During this period, the leader and other followers continue to receive data.
3. After the follower recovers, start to restore data, first load data from the local disk, read its last HW, the data higher than the last HW will be deleted, and request synchronization data from the leader, wait until the data recovery reaches When the new HW is installed, it can be added to the ISR.

Leader fault handling

• LEO (Log End Offset): The last offset of each copy, that is, the latest offset + 1.
• HW: Smallest LEO among all replicas.

Leader fault handling steps:

1. Remove the failed Leader node from the isr.
2. Elect a new leader from other followers.
3. Other followers must maintain data consistency with the new leader, and the data higher than the leader must be deleted. Therefore, there will be a problem of data loss in this way, because kafka guarantees data consistency, but cannot guarantee that data will not be lost or repeated.

Thirteen, kafka file storage mechanism

Kafka's data storage

The data storage of Kafka is stored according to partitions. The naming format of each partition directory is topic+partition number. There are three types of files in each partition directory, .log, .index, and .timeindex. Because the kafka production end continuously appends data to the .log file, .index is used to record the index of the data, and the .timeindex file is mainly used to delete expired data. Kafka's default data expiration time is 7 days.

.index is a sparse index, and the relative offset is stored in the file. Every time about 4kb (log.index.interval.bytes) of data is written in the .log file, an index will be written to the .index file, which ensures The file does not take up too much space.

# 查看kafka的.log文件
[root@node-02 topic-orkasgb-test-1]# kafka-run-class.sh kafka.tools.DumpLogSegments --files ./00000000000000000000.log
Dumping ./00000000000000000000.log
Starting offset: 0
baseOffset: 0 lastOffset: 0 count: 1 baseSequence: -1 lastSequence: -1 producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 0 isTransactional: false isControl: false position: 0 CreateTime: 1649067089350 size: 79 magic: 2 compresscodec: none crc: 384684249 isvalid: true
[root@node-02 topic-orkasgb-test-1]#

Manually adjust the partition copy storage policy

1. Create partition copy storage policy plan xxxx.json.

{
    
    
  "partitions" : [
  	{
    
    "topic" : "xxx","partition": 0,"replicas": [1,3]},
    {
    
    "topic" : "xxx","partition": 1,"replicas": [2,3]},
    {
    
    "topic" : "xxx","partition": 2,"replicas": [1,2]}
  ],
  "version" : 1
}

1. Execute the load balancing plan according to the xx.json file.

kafka-reassign-partitions.sh --bootstrap-server node-01:9092 --reassignment-json-file xxxx.json --execute

1. Verify that the load balancing plan is executed.

kafka-reassign-partitions.sh --bootstrap-server node-01:9092 --reassignment-json-file xxxx.json --verify

Log Cleanup Policy

File cleanup in Kafka, controlled by the following parameters:

• log.retention.hours: 7 days by default, with the lowest priority
• log.retention.minutes: minutes, default is null, priority 2
• log.retention.ms: milliseconds, default null, priority 1
• log.retention.check.interval.ms: The period for detecting the expiration time of data files, which is detected every 5 minutes by default.
• log.cleanup.policy: log cleanup policy setting, default delete

Log clear strategy:

1. delete delete log: delete expired data, log.cleanup.policy = delete.

   • Time-based: On by default, the file's timestamp is taken as the largest timestamp among all records in the segment.
   • Size-based: Disabled by default. If the total size of all logs exceeds the set size, the oldest segment will be deleted. log.retention.bytes, the default is -1, which means infinity.

   If part of the log data in a segment has expired and part of it has not expired (the part with the largest timestamp in the record), then the segment will not be deleted.

2. compact: log compaction, log.cleanup.policy = compact.

   For the same key, only the last version is kept. After compression, there will be offsets that are not continuous. If the offset you are looking for does not exist, then the nearest offset that is larger than it will be returned, and consumption will start from this offset.

14. Kafka consumers

Partition assignment process for consumer groups:

1. The election of the coordinator boss is carried out inside the Kafka cluster, which is used to assist in the initialization of consumer groups and the allocation of partitions. The election method is the hash value of group.id%50, and the coordinator on the broker corresponding to the calculated value is the coordinator boss.
2. All consumers in the consumer group send a request to the coordinator boss to join the current group group. After receiving the request, the coordinator boss organizes these consumers into a group, and elects a leader consumer from these consumers, and then the leader consumer formulates a consumption plan .
3. The leader consumer formulates a consumption plan and sends it to the coordinator boss. The coordinator boss assigns the task to the corresponding consumer, and then each consumer pulls data from the partition assigned to it.
4. Each consumer will maintain a heartbeat with the coordinator boss (3 seconds by default). Once a timeout occurs (session.timeout.ms=45s), the coordinator boss will remove the consumer, and then trigger load balancing to transfer tasks from other partitions Balanced to the remaining consumers. After pulling the data, if the consumer processing time is too long (max.poll.interval.ms=5 minutes), load balancing will also be triggered to balance the tasks of other partitions to the remaining consumers.

The data pull process of the consumer group:

1. The consumer group creates a ConsumerNetworkClient network link to interact with the Kafka cluster. The consumer sends a consumption request and initializes some parameters: batch.min.bytes, the minimum value of data fetched in each batch, and the default is 1k. fatch.max.wait.ms, the timeout for a batch of data not being fetched, the default is 500ms. fatch.max.bytes, the size of data fetched in each batch, the default is 50m. max.poll.records, the maximum number of data to be pulled at one time, the default is 500.
2. After ConsumerNetworkClient pulls the data, it caches it in the corresponding queue.
3. The consumer starts to pull data from the cache queue. By default, 500 pieces of data are pulled at a time.
4. Deserialization starts after pulling the data, because on the producer side, the data is serialized.
5. The data also needs to pass through the interceptor before it is actually processed.

Consumer allocation partition strategy:

Kafka provides four consumer allocation partition strategies (partition.assignment.strategy), which are:

• org.apache.kafka.clients.consumer.RangeAssignor: Assign partitions by topic, so RangeAssignor is for each topic, first sort the partitions according to the serial number, as follows, consumer1 will consume topicA-p1, topicA-p2, consumer2 will consume topicA-p3. The number of partitions/number of consumers is used to determine how many partitions each consumer should consume. If it is inexhaustible, the top consumers will consume a few more partitions. This method is obvious. If there are multiple topics, all the redundant partitions of the topic will be handed over to the front consumers, which will cause the front consumers to be very busy and the latter consumers to be very leisurely. This is data skew. question.

• org.apache.kafka.clients.consumer.RoundRobinAssignor: Assign partitions to consumers in a round-robin fashion. That is, the partitions are allocated to each consumer in a round-robin manner. As follows, consumer1 will consume topicA-p1 and topicA-p3 two partitions, consumer2 will consume topicA-p2.

• org.apache.kafka.clients.consumer.StickyAssignor: Guarantees that the assignment is maximally balanced while preserving as many existing partition assignments as possible. A bit similar to RangeAssignor, but the difference is that the assignment is random, consumer1 may consume topicA-p1, topicA-p3/or topicA-p2, topicA-p3, consumer2 will consume topicA-p2 or topicA-p1.
• org.apache.kafka.clients.consumer.CooperativeStickyAssignor: Follows the same StickyAssignor logic, but allows cooperative rebalancing.
  The default assignor is [RangeAssignor, CooperativeStickyAssignor], RangeAssignor will be used by default, but upgrades to CooperativeStickyAssignor are allowed, implementing the org.apache.kafka.clients.consumer.ConsumerPartitionAssignor interface allows you to plug in custom assignment strategies.

Consumer offset position maintenance:

The offset before kafka0.9 version is stored in zookeeper, and the offset after kafka0.9 version is stored in the kafka system topic (__consumer_offset). Only by setting the parameter exclude.internal.topic=false in consumer.properties can you view the system topic. By default is true. __consumer_offset stores data in the form of key-value. The key is group.id+topic+partition number, and the value is the consumed offset of the partition number of the current topic. After a period of time every day, kafka will compact the content in the topic, so the latest offset is kept in consumer_offset.

# 查看系统主题__consumer_offset中的主题信息
kafka-console-consumer.sh --bootstrap-server node-01:9092,node-02:9092 --topic __consumer_offset --consumer.config /orkasgb/software/kafka-3.1.0/config/consumer.properties --formatter "kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter" --from-beginning

Specify offset location/time consumption:

Kafka provides three offset consumption methods: auto.offset.reset=earliest | latest | none, and the default is latest.

• earliest: Automatically reset the offset to the earliest offset, --from-beginning.
• latest: Automatically reset the offsets to the latest offsets.
• none: throws an exception if no previous offset for the consumer group is found.

By converting the time into the corresponding offset, you can consume according to the time.

Missed consumption and repeated consumption:

• Repeated consumption: caused by the automatic submission of offsets. If the previous offset is automatically submitted successfully, the consumer continues to consume at this time, and the consumption is successful, but it hangs up before the automatic submission of this offset. When the consumer is started again, the consumer It will read the offset that was successfully submitted last time, and this will cause the problem of repeated consumption.
• Missing consumption: caused by manually submitting offsets. If the manual submission of offsets happens to be successful, but the consumer happens to be killed at this time, then restarting will read the last successfully submitted offsets. This offset was manually submitted successfully, but it is not The offset that is actually consumed will cause missed consumption.

To avoid missed consumption and repeated consumption, it is necessary to use the transaction method, and to ensure that all links from the entire consumption to the downstream support transactions. For example, from consuming to storing data in mysql, transactions must be supported in order to achieve accurate one-time consumption.

Consumers increase throughput:

• If Kafka's consumption capacity is insufficient, you can consider increasing the number of topic partitions, and increase the number of consumers in the consumption group at the same time, the number of consumers = the number of topic partitions (both are indispensable).
• If the downstream processing data is not timely: increase the number of data pulled in each batch from the default of 500 to 1000.

15. Production tuning

1. Increase the throughput of producers:

• batch.size: batch size, default 16K.
• linger.ms: Waiting time, it is recommended to change it to 5~10ms.
• compression-type: consider using snappy for data compression before sending.
• RecordAccumulator: The buffer size can be adjusted to 64M appropriately, and the default is 32M.

2. ACKS response level tuning:

• 0: After the producer sends the data, it can respond to the client without waiting for the leader to respond successfully.

**Problem analysis:** After the data is sent, it will return immediately without waiting for the leader’s response. If the leader fails to process the data after the data is sent to the leader, then all the data has been lost at this time.

**Data reliability analysis: **Data loss, but the highest efficiency.

• 1: After the producer sends the data, it needs to wait for the leader to respond successfully before responding to the client.

**Problem analysis:** After the data is sent, the leader returns immediately after the response is successful. If the data is sent to the leader, the leader fails before processing the data. At this time, according to the internal leader selection strategy, a certain follower is called a new follower. leader, the producer will immediately interact with the new leader, but because the previous leader responded successfully, the producer will think that the previous data has been sent and will not send it again, so all the data has been lost at this time.

After the data is sent, all the followers in the leader and ISR queues have processed the data and are preparing to respond with acks, but the leader fails. At this time, according to the internal leader selection strategy, a certain follower is called the new leader, and the producer will Immediately interact with the new leader, but because the previous leader did not respond successfully, the producer will think that the previous data has not been sent, and continue to send again, then the data has been repeatedly processed at this time.

**Data reliability analysis: **Data loss, data duplication.

• -1 (all): After the producer sends the data, it needs to wait for the leader and all followers in the ISR to respond successfully before responding to the client.

**Problem analysis:** After the data is sent, the product waits for the leader and other followers to reply, but one of the followers fails, which prevents the acks response. At this time, the product sends the data but cannot return. In order to solve this problem, Kafka allows the leader to maintain a dynamic set of synchronous information of the leader and follower, in the form: (leader:1,isr:[1,2,3]). If a follower does not communicate with the leader for more than 30s (replica.lag.time.max.ms, default 30s), then the leader considers the follower to have failed, and removes the follower from the ISR queue.

**Data reliability analysis:** If the number of copies is 1, there is still data loss.

Data reliability guarantee:

At least once = (ACKS level = -1) + (number of partition copies >= 2) + (at least two node information must be guaranteed in the ISR queue).

Exactly once = idempotency + at least once.

3. Manually adjust the partition copy storage strategy

1. Create partition copy storage policy plan xxxx.json.

{
    
    
  "partitions" : [
  	{
    
    "topic" : "xxx","partition": 0,"replicas": [1,3]},
    {
    
    "topic" : "xxx","partition": 1,"replicas": [2,3]},
    {
    
    "topic" : "xxx","partition": 2,"replicas": [1,2]}
  ],
  "version" : 1
}

2. Execute the load balancing plan according to the xx.json file.

kafka-reassign-partitions.sh --bootstrap-server node-01:9092 --reassignment-json-file xxxx.json --execute

3. Verify that the load balancing plan is executed.

kafka-reassign-partitions.sh --bootstrap-server node-01:9092 --reassignment-json-file xxxx.json --verify