I try to simulate Gwen (Chen) Shapira's kafka-clickstream-enrich kafka-stream project on GitHub (https://github.com/onurtokat/kafka-clickstream-enrich). When I consume a topic using consumer class using Deserializers, I encounter an error. The customized Serde class have serializer and deserializer. But, I try to understand why custom serde is used for deserializer, then consumer API gives error as it is not an instance of org.apache.kafka.common.serialization.Deserializer
The topic can be consumed using KTable with Serdes.Integer() Serializer and new ProfileSerde() Deserializer like below.
KTable<Integer, UserProfile> profiles = builder.table(Constants.USER_PROFILE_TOPIC,
Consumed.with(Serdes.Integer(), new ProfileSerde()),
Materialized.as("profile-store"));
Customized Serde is defined as;
static public final class ProfileSerde extends WrapperSerde<UserProfile> {
public ProfileSerde() {
super(new JsonSerializer<UserProfile>(), new JsonDeserializer<UserProfile>(UserProfile.class));
}
}
And generic Serde is customized liked below;
package com.onurtokat.serde;
import org.apache.kafka.common.serialization.Deserializer;
import org.apache.kafka.common.serialization.Serde;
import org.apache.kafka.common.serialization.Serializer;
import java.util.Map;
public class WrapperSerde<T> implements Serde<T> {
final private Serializer<T> serializer;
final private Deserializer<T> deserializer;
public WrapperSerde(Serializer<T> serializer, Deserializer<T> deserializer) {
this.serializer = serializer;
this.deserializer = deserializer;
}
@Override
public void configure(Map<String, ?> configs, boolean isKey) {
serializer.configure(configs, isKey);
deserializer.configure(configs, isKey);
}
@Override
public void close() {
serializer.close();
deserializer.close();
}
@Override
public Serializer<T> serializer() {
return serializer;
}
@Override
public Deserializer<T> deserializer() {
return deserializer;
}
}
My Consumer is so simple and can be seen below;
package com.onurtokat.consumers;
import com.onurtokat.ClickstreamEnrichment;
import com.onurtokat.Constants;
import com.onurtokat.model.UserProfile;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.serialization.IntegerDeserializer;
import java.time.Duration;
import java.util.Arrays;
import java.util.Properties;
public class ConsumeProfileData {
public static void main(String[] args) {
//prepare config
Properties config = new Properties();
config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
config.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, IntegerDeserializer.class.getName());
config.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, ClickstreamEnrichment.ProfileSerde.class);
KafkaConsumer<Integer, UserProfile> consumerProfileTopic = new KafkaConsumer<>(config);
consumerProfileTopic.subscribe(Arrays.asList(Constants.USER_PROFILE_TOPIC));
while (true) {
ConsumerRecords<Integer, UserProfile> records = consumerProfileTopic.poll(Duration.ofMillis(100));
for (ConsumerRecord<Integer, UserProfile> record : records) {
System.out.println(record.key() + " " + record.value());
}
}
}
}
The error when I try to consume topic with my consumer is;
log4j:WARN No appenders could be found for logger (org.apache.kafka.clients.consumer.ConsumerConfig).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:811)
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:659)
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:639)
at com.onurtokat.consumers.ConsumeProfileData.main(ConsumeProfileData.java:25)
Caused by: org.apache.kafka.common.KafkaException: com.onurtokat.ClickstreamEnrichment$ProfileSerde is not an instance of org.apache.kafka.common.serialization.Deserializer
at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:304)
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:712)
... 3 more
The difference is:
- Serdes are used by Kafka's Streams API (aka Kafka Streams). A Serde is a wrapper for a pair of (1) serializer and (2) deserializer for the same data type. That is, a
Serde<T>
has aSerializer<T>
and aDeserializer<T>
. The first code snippet you posted (with e.g. aKTable
) is a Kafka Streams code snippet, that's why it needs aSerde
. Kafka Streams needs aSerde
because it both produces messages (for which it needs aSerializer
) and reads messages (for which it needs aDeserializer
). - Deserializers are used by Kafka's consumer API (aka consumer client) for reading messages. Your last code snippet (with e.g. a
KafkaConsumer
) is using the consumer client and therefore needs aDeserializer
, not aSerde
. - Serializers are used by Kafka's producer API (aka producer client) for writing messages.
Regarding:
Caused by: org.apache.kafka.common.KafkaException: com.onurtokat.ClickstreamEnrichment$ProfileSerde is not an instance of org.apache.kafka.common.serialization.Deserializer
at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:304)
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:712)
... 3 more
Your Kafka consumer client code was given a Serde
where it was expecting a Deserializer
.