Kafka Java client data production process analysis, from sending type implementation code to serializer implementation code!

1. Message sending

1. Analysis of Kafka Java client data production process

1. To construct a first ProducerRecordobject that can be declared topic Topic , partition Partition , key Key and value Value , themes and values that must be declared, partitions and keys can not be specified.
2. Call the send() method to send the message.
3. Because the message is to be transmitted on the network, it must be serialized. The role of the serializer is to serialize the key and value objects of the message into a byte array.
4. The next data to the partition is, if among the ProducerRecordobjects specified partition, then the partition will not do anything directly to the specified partition return; if not, then the partition will be selected based on a partition Key, a good choice After partitioning, the producer knows which topic and partition to send records to.
5. Then this record will be added to a record batch, and all messages in this batch will be sent to the same topic and partition. There will be a separate thread to send these batches of records to the corresponding Broker.
6. Broker successfully receives the message, indicating that the transmission is successful, and returns the metadata of the message (including the topic and partition information and the offset recorded in the partition) . Sending failed, you can choose to retry or throw an exception directly.

Dependent packages <kafka.version>2.0.0</kafka.version>

<dependency>
	<groupId>org.apache.kafka</groupId> 
	<artifactId>kafka_${
    
    scala.version}</artifactId> 
	<version>${
    
    kafka.version}</version> 
	<exclusions> 
		<exclusion> 
			<groupId>org.apache.zookeeper</groupId> 
			<artifactId>zookeeper</artifactId> 
		</exclusion> 
		<exclusion> 
			<groupId>org.slf4j</groupId> 
			<artifactId>slf4j-log4j12</artifactId> 
		</exclusion> 
		<exclusion> 
			<groupId>log4j</groupId> 
			<artifactId>log4j</artifactId> 
		</exclusion> 
	</exclusions> 
</dependency>

2. Necessary parameter configuration

See the code base:com.heima.kafka.chapter2.KafkaProducerAnalysis

public static Properties initConfig() {
    
     
	Properties props = new Properties(); 
	// 该属性指定 brokers 的地址清单，格式为 host:port。清单里不需要包含所有的 broker 地址， 
	// 生产者会从给定的 broker 里查找到其它 broker 的信息。——建议至少提供两个 broker 的信息，因为一旦其中一个宕机，生产者仍然能够连接到集群上。 
	props.put("bootstrap.servers", brokerList); 
	// 将 key 转换为字节数组的配置，必须设定为一个实现了 org.apache.kafka.common.serialization.Serializer 接口的类， 
	// 生产者会用这个类把键对象序列化为字节数组。 
	// ——kafka 默认提供了 StringSerializer和 IntegerSerializer、 ByteArraySerializer。当然也可以自定义序列化器。 
	props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer"); 
	// 和 key.serializer 一样，用于 value 的序列化 
	props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
	// 用来设定KafkaProducer对应的客户端ID，默认为空，如果不设置KafkaProducer会自动 生成一个非空字符串。 
	// 内容形式如："producer-1" 
	props.put("client.id", "producer.client.id.demo"); 
	return props; 
}

Properties props = initConfig(); 
	KafkaProducer<String, String> producer = new KafkaProducer<>(props); 
// 	KafkaProducer<String, String> producer = new KafkaProducer<>(props, 
// new StringSerializer(), new StringSerializer()); 
//生成 ProducerRecord 对象，并制定 Topic，key 以及 value 
	ProducerRecord<String, String> record = new ProducerRecord<>(topic, "hello, Kafka!"); 
	try {
    
    
		// 发送消息 
		producer.send(record);
	}

3. Delivery type

Send and forget

producer.send(record)

Send synchronously

//通过send()发送完消息后返回一个Future对象,然后调用Future对象的get()方法等待kafka响应 
//如果kafka正常响应，返回一个RecordMetadata对象，该对象存储消息的偏移量 
// 如果kafka发生错误，无法正常响应，就会抛出异常，我们便可以进行异常处理 
producer.send(record).get();

Asynchronous send

producer.send(record, new Callback() {
    
     
	public void onCompletion(RecordMetadata metadata, Exception exception) {
    
     
	if (exception == null) {
    
     
		System.out.println(metadata.partition() + ":" + metadata.offset()); 
	} 
} 
});

4. Serializer

Messages must be serialized to be transmitted on the network, and the role of the serializer is just that.

Kafka provides the default string serializer ( org.apache.kafka.common.serialization.StringSerializer), as well as integer ( IntegerSerializer) and byte array ( BytesSerializer) serializers. These serializers all implement the interface ( org.apache.kafka.common.serialization.Serializer) and can basically meet the needs of most scenarios.

5. Custom serializer

See the code base:com.heima.kafka.chapter2.CompanySerializer

/**
	* 自定义序列化器
	*/
public class CompanySerializer implements Serializer<Company> {
    
     
	@Override 
	public void configure(Map configs, boolean isKey) {
    
     
	}
	
	@Override 
	public byte[] serialize(String topic, Company data) {
    
     
		if (data == null) {
    
     
			return null; 
		}
		byte[] name, address; 
		try {
    
    
			if (data.getName() != null) {
    
     
				name = data.getName().getBytes("UTF-8"); 
			} else {
    
     
				name = new byte[0]; 
			}
			if (data.getAddress() != null) {
    
     
				address = data.getAddress().getBytes("UTF-8"); 
			} else {
    
     
				address = new byte[0]; 
			}
			ByteBuffer buffer = ByteBuffer. allocate(4 + 4 + name.length + address.length); 
			buffer.putInt(name.length); 
			buffer.put(name); 
			buffer.putInt(address.length); 
			buffer.put(address); 
			return buffer.array(); 
		} catch (UnsupportedEncodingException e) {
    
     
			e.printStackTrace(); 
		}
		return new byte[0]; 
	}
	
	@Override 
	public void close() {
    
     
	} 
}

• Use a custom serializer

See the code base:com.heima.kafka.chapter2.ProducerDefineSerializer

public class ProducerDefineSerializer {
    
     
	public static final String brokerList = "localhost:9092"; 
	public static final String topic = "heima"; 
	
	public static void main(String[] args) throws ExecutionException, InterruptedException {
    
     
		Properties properties = new Properties(); 
		properties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName()); 
		properties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, CompanySerializer.class.getName()); 
// properties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
// ProtostuffSerializer.class.getName()); 
	properties.put("bootstrap.servers", brokerList); 
	
	KafkaProducer<String, Company> producer = new KafkaProducer<>(properties);
	Company company = Company.builder().name("kafka") .address("北京").build(); 
// Company company = Company.builder().name("hiddenkafka") 
// .address("China").telphone("13000000000").build(); 
	ProducerRecord<String, Company> record = new ProducerRecord<>(topic, company); 
	producer.send(record).get(); 
	} 
}

6. Partitioner

Kafka itself has its own partition strategy. If not specified, the default partition strategy will be used:

Kafka allocates partitions according to the key of the message, that is hash(key) % numPartitions. If the Key is the same, it will be assigned to a unified partition.

Source code org.apache.kafka.clients.producer.internals.DefaultPartitioneranalysis

public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) {
    
     
	List<PartitionInfo> partitions = cluster.partitionsForTopic(topic); 
	int numPartitions = partitions.size(); 
	if (keyBytes == null) {
    
     
		int nextValue = this.nextValue(topic); 
		List<PartitionInfo> availablePartitions = cluster.availablePartitionsForTopic(topic); 
		if (availablePartitions.size() > 0) {
    
     
			int part = Utils.toPositive(nextValue) % availablePartitions.size(); 
			return ((PartitionInfo)availablePartitions.get(part)).partition(); 
		} else {
    
     
			return Utils.toPositive(nextValue) % numPartitions; 
		} 
	} else {
    
     
		return Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions; 
	} 
}

• Custom partitioner see code base com.heima.kafka.chapter2.DefinePartitioner

/**
	* 自定义分区器 
	*/ 
public class DefinePartitioner implements Partitioner {
    
     
	private final AtomicInteger counter = new AtomicInteger(0); 
	
	@Override 
	public int partition(String topic, Object key, byte[] keyBytes,Object value, byte[] valueBytes, Cluster cluster) {
    
     
	List<PartitionInfo> partitions = cluster.partitionsForTopic(topic); 
	int numPartitions = partitions.size(); 
	if (null == keyBytes) {
    
     
		return counter.getAndIncrement() % numPartitions; 
		} else return Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions; 
	}
	@Override 
	public void close() {
    
     
	}
	@Override 
	public void configure(Map<String, ?> configs) {
    
     
	} 
}

• Implementing a custom partitioner needs ProducerConfig.PARTITIONER_CLASS_CONFIGto be achieved through configuration parameters

// 自定义分区器的使用 
props.put(ProducerConfig.PARTITIONER_CLASS_CONFIG,DefinePartitioner.class.getNam e());

7. Interceptor

The Producer Interceptor ( interceptor) is a fairly new feature. It and the consumer-side interceptor were introduced in Kafka version 0.10 and are mainly used to implement customized control logic on the client side .

The producer interceptor can be used to do some preparatory work before the message is sent.

scenes to be used

1. Filter out unqualified messages according to a certain rule
2. Modify the content of the message
3. Statistics requirements

See Code Base: Custom Interceptorcom.heima.kafka.chapter2.ProducerInterceptorPrefix

/**
	* 自定义拦截器 
	*/ 
public class ProducerInterceptorPrefix implements ProducerInterceptor<String, String> {
    
     
	private volatile long sendSuccess = 0; 
	private volatile long sendFailure = 0; 
	
	@Override 
	public ProducerRecord<String, String> onSend( ProducerRecord<String, String> record) {
    
     
		String modifiedValue = "prefix1-" + record.value(); 
		return new ProducerRecord<>(record.topic(), record.partition(), record.timestamp(), record.key(), modifiedValue, record.headers()); 
// if (record.value().length() < 5) { 
// throw new RuntimeException();
// } 
// return record; 
	}
	
	@Override 
	public void onAcknowledgement( RecordMetadata recordMetadata, Exception e) {
    
     
		if (e == null) {
    
     
			sendSuccess++; 
		} else {
    
     
			sendFailure++; 
		} 
	}
	
	@Override 
	public void close() {
    
     
		double successRatio = (double) sendSuccess / (sendFailure + sendSuccess); 
		System.out.println("[INFO] 发送成功率=" + String.format("%f", successRatio * 100) + "%"); 
	}
	
	@Override 
	public void configure(Map<String, ?> map) {
    
     
	} 
}

• After implementing a custom interceptor, you need to specify the interceptor in the configuration parameters. The default value of this parameter is empty, as follows:

// 自定义拦截器使用 
props.put(ProducerConfig.INTERCEPTOR_CLASSES_CONFIG,ProducerDefineSerializer.cla ss.getName());

• Demo:

Sender

receiver

---

Students who need the reference article "Kafka Technical Manual" can add assistant VX: C18173184271 Remarks:CSDN Java_Caiyo Get free Java information!