kafka基本知识了解

kafka 的默认配置比较简单，但想把其端口暴露给外网（指定端口），则有一些额外的注意情况
kafka 的版本，0.8和0.10 的配置不同，这里以0.10.0.0 为主,
没有用集群，一台机做测试

zookeeper安装与启动

kafka 的安装包里自带有zookeeper,不过这里还是自行下载.
这里采用3.4.8版本
下载及安装请参考
http://zookeeper.apache.org/doc/r3.4.8/zookeeperStarted.html
启动:
bin/zkServer.sh start

zk默认客户端的连接
bin/zkCli.sh -server 127.0.0.1:2181

kafka的安装与启动

kafka采用0.10.0.0版本，文档请查看
http://kafka.apache.org/quickstart

【2018-7月最新版本是1.1.0，下面的命令都是确认过的，有部分有改动会注明】

官网中是内置了一个zookeeper, 要注意不用去执行了.

关闭kfk服务端(kafka没有关闭的命令，如果想关，可参考使用kill 命令)

ps ax | grep -i 'kafkaServer-gc.log' | grep java | awk '{print $1}' | xargs kill -9

启动
bin/kafka-server-start.sh config/server.properties

配置
conf/service.properties
里面有一个和zookeeper的配置，

zookeeper.connect=localhost:2181

日志
tail -111f logs/server.log

创建1个topic
bin/kafka-topics.sh –create –zookeeper localhost:2181 –replication-factor 1 –partitions 1 –topic test

查看topic list
bin/kafka-topics.sh –list –zookeeper localhost:2181

查看topic 描述
bin/kafka-topics.sh –describe –zookeeper localhost:2181 –topic test

开启生产者（使用内置工具开启一个通过产者,这是一个单独的进程，和kafka服务端无关,下面消费者同理)
bin/kafka-console-producer.sh –broker-list localhost:9092 –topic test2

开启消费者
0.10.0版本：bin/kafka-console-consumer.sh –zookeeper localhost:2181 –topic test –from-beginning
1.1.0版本：bin/kafka-console-consumer.sh –bootstrap-server localhost:9092 –topic test2 –from-beginning

如果在1.1.0 中执行老版本的开启消费者，会有下面这个warning,但也可以使用
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].

创建kafka集群(这一部分内容在官网可直接查看)

准备另两个节点的配置文件

cp config/server.properties config/server-1.properties
cp config/server.properties config/server-2.properties

修改端口等参数:

config/server-1.properties:
    broker.id=1
    listeners=PLAINTEXT://:9093
    log.dir=/tmp/kafka-logs-1

config/server-2.properties:
    broker.id=2
    listeners=PLAINTEXT://:9094
    log.dir=/tmp/kafka-logs-2

开启另两个节点

bin/kafka-server-start.sh config/server-1.properties &
bin/kafka-server-start.sh config/server-2.properties &

使用kafka集群

创建topic

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic

replication-factor为3，说明数据会分三个副本，这是因为我们有三台server了，如果设置成大于3是会抛错的:

Error while executing topic command : Replication factor: 5 larger than available brokers: 3.
[2018-07-05 11:34:58,818] ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 5 larger than available brokers: 3.
 (kafka.admin.TopicCommand$)

查看topic列表

bin/kafka-topics.sh --list --zookeeper localhost:2181

查看topic详情

bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic

开启消费者和生产者参考上文

java客户端

官网里也可以直接找

生产者java客户端

 public static void main(String[] args) throws ExecutionException, InterruptedException {
        Properties props = new Properties();
        props.put("bootstrap.servers", "120.197.3.90:8180");
//        props.put("bootstrap.servers", "10.1.1.29:9092");
        props.put("acks", "all");
        props.put("retries", 0);
        props.put("batch.size", 16384);
        props.put("linger.ms", 1);
        props.put("buffer.memory", 33554432);
        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

        Producer<String, String> producer = new KafkaProducer<>(props);
        for(int i = 0; i < 10000; i++) {
            Future<RecordMetadata> futureresult = producer.send(new ProducerRecord<String, String>("test2", Integer.toString(i), Integer.toString(i)+"gogo"), new Callback() {
                @Override
                public void onCompletion(RecordMetadata recordMetadata, Exception e) {
                    // 发送的返回

                }
            });
//            futureresult.get();//阻塞
        }

        producer.close();
    }

消费者java客户端

    public static void main(String[] args) {
        Properties props = new Properties();
        props.put("bootstrap.servers", "120.xx.xx.xx:8180");
//        props.put("bootstrap.servers", "10.1.1.29:9092");
        props.put("group.id", "tests");
        props.put("enable.auto.commit", "true");
        props.put("auto.commit.interval.ms", "1000");
        props.put("session.timeout.ms", "30000");
        props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
        consumer.subscribe(Arrays.asList("test2"));
        while (true) {
            ConsumerRecords<String, String> records = consumer.poll(100);
            for (ConsumerRecord<String, String> record : records)
                System.out.printf("offset = %d, key = %s, value = %s", record.offset(), record.key(), record.value());
        }
    }

关于spring 结合，参考：http://www.baeldung.com/spring-kafka

问题

为什么java客户端不需要 zookeeper ip和端口?
kafak内置的消费端工具是这样子的
/home/huge/projects/kfk_cluster/kfk/kafka_2.11-0.10.0.0/bin/kafka-console-consumer.sh –zookeeper ZN-YST-Backup03:8181 –topic test2 –from-beginning
而java没有填zookeeper的ip端口，而是kafka的ip端口>>.
java producer 访问不了服务端，没有抛错，如果用回调，可以看到有 timeout 的情况
server.properties 加上 advertised.host.name=10.1.1.29

如果消费者出现
org.apache.kafka.clients.consumer.NoOffsetForPartitionException: Undefined offset with no reset policy for partition

消费者配置加上:auto.offset.reset=latest
这个参数关于官网的说法：
What to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server (e.g. because that data has been deleted):(当没有存在的offset时候，用哪个？)

earliest: automatically reset the offset to the earliest offset
latest: automatically reset the offset to the latest offset
none: throw exception to the consumer if no previous offset is found for the consumer’s group
anything else: throw exception to the consumer.