大数据集群搭建和使用之八——kafka配置和使用

这个系列指南使用真实集群搭建环境，不是伪集群，用了三台腾讯云服务器

或者访问我的个人博客站点，链接

Kafka

配置

kafka依赖zookeeper，所以先确保集群已经安装zookeeper并且能够正常启动。
浪费了一整天的时间debug结果bug很简单（至少现在集群没有崩溃）

建立目录树 /opt/kafka/kafka2.12
在/root/kafka/kafka-logs/logs建立一个用于存放日志的文件
配置环境变量/etc/profile，添加bin目录
修改配置文件kafka/config/server.properties
1. 修改broker.id id和zookeeper的myid一致（应该是这样，这个bug我查了一天），每个主机的id都不一样，每次修改前，需要确认（或者干脆删除）kafka日志文件（/root/kafka/kafka-logs/metaxxxx中的id是否和broker.id一致）
2. 修改zookeeper.connect和zookeeper.connection.timeout.ms

zookeeper.connect=master:2181,slave1:2181,slave2:2181
zookeeper.connection.timeout.ms=6000

3. 修改logdir（记得必须先创建文件，kafka不会自己创建文件夹）
4. 修改两处listener（vim使用/listener查找），手动添加hostname（例如master,slave1,slave2），其实按照文档，只需要修改一处即可。

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# see kafka.server.KafkaConfig for additional details and defaults

############################# Server Basics #############################

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0

############################# Socket Server Settings #############################

# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://master:9092

# Hostname and port the broker will advertise to producers and consumers. If not set,
# it uses the value for "listeners" if configured.  Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
advertised.listeners=PLAINTEXT://master:9092

# Maps listener names to security protocols, the default is for them to be the same. See the config documentation for more details
#listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL

# The number of threads that the server uses for receiving requests from the network and sending responses to the network
num.network.threads=3

# The number of threads that the server uses for processing requests, which may include disk I/O
num.io.threads=8

# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=102400

# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=102400

# The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=104857600


############################# Log Basics #############################

# A comma separated list of directories under which to store log files
log.dirs=/root/kafka/kafka-logs

# The default number of log partitions per topic. More partitions allow greater
# parallelism for consumption, but this will also result in more files across
# the brokers.
num.partitions=1

# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.
# This value is recommended to be increased for installations with data dirs located in RAID array.
num.recovery.threads.per.data.dir=1

############################# Internal Topic Settings  #############################
# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended for to ensure availability such as 3.
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

############################# Log Flush Policy #############################

# Messages are immediately written to the filesystem but by default we only fsync() to sync
# the OS cache lazily. The following configurations control the flush of data to disk.
# There are a few important trade-offs here:
#    1. Durability: Unflushed data may be lost if you are not using replication.
#    2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush.
#    3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to excessive seeks.
# The settings below allow one to configure the flush policy to flush data after a period of time or
# every N messages (or both). This can be done globally and overridden on a per-topic basis.

# The number of messages to accept before forcing a flush of data to disk
#log.flush.interval.messages=10000

# The maximum amount of time a message can sit in a log before we force a flush
#log.flush.interval.ms=1000

############################# Log Retention Policy #############################

# The following configurations control the disposal of log segments. The policy can
# be set to delete segments after a period of time, or after a given size has accumulated.
# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens
# from the end of the log.

# The minimum age of a log file to be eligible for deletion due to age
log.retention.hours=168

# A size-based retention policy for logs. Segments are pruned from the log unless the remaining
# segments drop below log.retention.bytes. Functions independently of log.retention.hours.
#log.retention.bytes=1073741824

# The maximum size of a log segment file. When this size is reached a new log segment will be created.
log.segment.bytes=1073741824

# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
log.retention.check.interval.ms=300000

############################# Zookeeper #############################

# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=master:2181,slave1:2181,slave2:2181/kafka

# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000


############################# Group Coordinator Settings #############################

# The following configuration specifies the time, in milliseconds, that the GroupCoordinator will delay the initial consumer rebalance.
# The rebalance will be further delayed by the value of group.initial.rebalance.delay.ms as new members join the group, up to a maximum of max.poll.interval.ms.
# The default value for this is 3 seconds.
# We override this to 0 here as it makes for a better out-of-the-box experience for development and testing.
# However, in production environments the default value of 3 seconds is more suitable as this will help to avoid unnecessary, and potentially expensive, rebalances during application startup.
group.initial.rebalance.delay.ms=0

delete.topic.enable=true

启动

cd /${KAFKA_HOME}

bin/kafka-server-start.sh config/server.properties -daemon > /root/kafka/kafka-logs/logs &

指定日志的存放地点为/root/kafka/kafka-logs/logs
使用jps命令查看kafka是否配置成功。

记一次服务器被黑客攻击

起因：kafka启动总是异常（kafka进程启动一两分钟后自动退出），日志却没有记录

• [ ] 系统运行情况查看工具top，关于top的介绍点这里
• [ ] 重新配置kafka，检查各个配置项，仍然宕机
• [ ] 重新配置与kafka相关联的zookeeper，仍然宕机
• [ ] 关闭不需要的进程，例如hbase,yarn,storm,hdfs，重启kafka，仍然宕机
• [ ] 重启服务器，重新开启各个进程，顺序为hdfs,yarn,zookeeper,hbase,storm,kafka，仍然宕机
• [ ] 重启服务器，更换kafka版本，仍然宕机
• [ ] 发现kafka总是宕机的服务器cpu占用几乎100%，而master却正常，100%的cpu占用由java进程贡献，具体执行任务未知。
• [ ] google发现，hadoop集群有cpu占用过高的风险，参考这个链接，怀疑datanode导致cpu占用过高，原因是linux内核内存申请优化对hadoop的副作用。
• [ ] 关闭hadoop相关进程（只剩jps）cpu占用仍然是100%，但是master节点cpu正常，100%的cpu占用由java进程贡献，具体执行任务未知。
• [ ] 尝试启动kafka，master正常，slave失败，slave节点cpu占用仍然是100%
• [ ] 重启所有集群服务器，实时检测cpu动态(top命令），先启动zookeeper，正常，再启动kafka，所有节点正常
• [ ] 挨个启动其他服务。hdfs无影响，yarn无影响，zookeeper无影响，hive无影响,hbase启动后cpu飙升100%，导致kafka宕机,具体导致宕机的进程为regionserver。
• [ ] 启动除了hbase其他所有服务，一切正常。原因未知。
• [ ] 第二天一早服务器宕机
• [ ] 百度重新配置了yarn框架参数
• [ ] 重启服务器仍然宕机
• [ ] 参考这里和这里的链接有理由相信云服务器被黑了。
• [ ] 暂时的解决方案
• [ ] 咨询腾讯云客服，修改了安全组配置，关闭了8088端口。
  
• [ ] 所有组件运行正常，之前怀疑是hbase的原因是，病毒文件的执行需要一定的时间，而在这段时间里，我刚好启动了hbase，也有可能是病毒文件需要依赖hbase作为数据库？

使用KAFKA

基本概念

• kafka是一个分布式的消息缓存系统
• kafka集群中的服务器叫做broker
• kafka有两种客户端，producer（消息生产者），consumer（消息消费者），客户端（两种）与kafka服务器之间使用tcp通信
• kafka中不同业务系统的消息可以通过topic进行区分，而且每一个消息topic都会被分区，以分担消息读写的负载
• 每一个分区可以有多个副本，防止数据的丢失
• 如果某个分区中的数据需要更新，必须通过该分区所有副本中的leader来更新
• 消费者可以分组，比如有两个消费者AB，共同消费一个topic:testTopic，AB所消费的消息不会重复，比如testTopic中有100个消息，编号为0-99，如果A消费0-49，那么B就消费50-99。消费者在消费时可以指定消息的起始偏移量

• kafka架构图：
  
  producer是数据源，比如flume架构，consumer是数据的输出，例如storm架构。

• kafka服务器支持消息的分主题、分区。不同的子系统可以使用不同的主题。分区的意义在于负载均衡。
  

kafka shell

• 创建话题

bin/kafka-topics.sh --create --zookeeper master:2181 --replication-factor 3 --partitions 1 --topic mytopics

创建的话题名称是有要求的Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.

• 如果在zookeeper里指定了kafka的目录，例如/kafka，那么在用shell进行topic操作的时候，需要指定被操作的topic所属的zookeeper目录，例如bin/kafka-topics.sh –create –zookeeper master:2181 /kafka –replication-factor 3 –partitions 1 –topic mytopics。（因为kafka的集群化是归zookeeper管的）

• 列出当前话题

bin/ kafka-topics.sh --list --zookeeper master:2181,slave1:2181,slave2:2181

• 删除话题

kafka-topics.sh --delete --zookeeper master:2181 --topic mytopics

注意由控制台的提示：Note: This will have no impact if delete.topic.enable is not set to true.可知，需要修改一下server.properties文件，在最后一行加上delete.topic.enable=true

• 创建一个生产者

kafka-console-producer.sh --broker-list master:9092 --topic t_test

• 创建一个消费者

kafka-console-consumer.sh --zookeeper master:2181 --from-beginning --topic t_test

• 查看话题状态信息

kafka-topics.sh --describe --zookeeper master:2181 --topic t_test

isr表示现在处于同步状态的broker，如果杀掉某一台服务器，例如杀掉leader:0的服务器，也就是0号服务器：master中的kafka进程。执行kill -9 pid

kafka会立即进行容灾处理，同时，生产和消费并不受影响。
再次恢复kafka进程，三台服务器又会立即同步。

kafka java demo

推荐使用maven来构建项目，如果没有使用maven，导入kafka压缩包里的libs中的jar包即可

生产者样例

配置说明

• bootstrap.servers： kafka的地址。
• acks:消息的确认机制，默认值是0。
• acks=0 ：如果设置为0，生产者不会等待kafka的响应。
• acks=1 ：这个配置意味着kafka会把这条消息写到本地日志文件中，但是不会等待集群中其他机器的成功响应。
• acks=all ：这个配置意味着leader会等待所有的follower同步完成。这个确保消息不会丢失，除非kafka集群中所有机器挂掉。这是最强的可用性保证。
• retries：配置为大于0的值的话，客户端会在消息发送失败时重新发送。
• batch.size:当多条消息需要发送到同一个分区时，生产者会尝试合并网络请求。这会提高client和生产者的效率。
• key.serializer: 键序列化，默认org.apache.kafka.common.serialization.StringDeserializer。
• value.deserializer:值序列化，默认org.apache.kafka.common.serialization.StringDeserializer。

添加完配置之后，producer就可以生产数据，使用producer.send()方法。传入的参数为topic,key,value。如果topic在kafka集群中还没有被创建，那么便会自动创建一个新的topic（新建的topic各个属性我不知道）

package cn.colony.cloudhadoop.kafka;

import java.util.Properties;

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;

public class ProducerDemo {

    public static void main(String[] args) throws InterruptedException{
        Properties props = new Properties();//配置项
        props.put("bootstrap.servers", "master:9092,slave1:9092,slave2:9092");//使用新的API指定kafka集群位置
        props.put("acks", "all");
        props.put("retries", 0);
        props.put("batch.size", 16384);
        props.put("key.serializer", StringSerializer.class.getName());
        props.put("value.serializer", StringSerializer.class.getName());
        KafkaProducer<String, String> producer = new KafkaProducer<String, String>(props);

        String messageStr = null;
        for (int i = 1;i<1000;i++){
            Thread.sleep(50);
            messageStr = "hello, this is "+i+"th message";
            producer.send(new ProducerRecord<String, String>("t_topic","Message",messageStr));
        }
        producer.close();
    }
}

消费者样例

配置说明

• bootstrap.servers： kafka的地址。
• group.id：组名 不同组名可以重复消费。例如你先使用了组名A消费了kafka的1000条数据，但是你还想再次进行消费这1000条数据，并且不想重新去产生，那么这里你只需要更改组名就可以重复消费了。
• enable.auto.commit：是否自动提交，默认为true。
• auto.commit.interval.ms: 从poll(拉)的回话处理时长。
• session.timeout.ms:超时时间。
• max.poll.records:一次最大拉取的条数。
• auto.offset.reset：消费规则，默认earliest 。
• earliest: 当各分区下有已提交的offset时，从提交的offset开始消费；无提交的offset时，从头开始消费 。
• latest: 当各分区下有已提交的offset时，从提交的offset开始消费；无提交的offset时，消费新产生的该分区下的数据 。
• none: topic各分区都存在已提交的offset时，从offset后开始消费；只要有一个分区不存在已提交的offset，则抛出异常。
• key.serializer: 键序列化，默认org.apache.kafka.common.serialization.StringDeserializer。
• value.deserializer:值序列化，默认org.apache.kafka.common.serialization.StringDeserializer。

首先订阅一个topic，consumer就可以开始消费数据。

package cn.colony.cloudhadoop.kafka;

import java.util.Arrays;
import java.util.Properties;

import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.errors.InterruptException;
import org.apache.kafka.common.serialization.StringDeserializer;

public class ConsumerDemo implements Runnable{
    private final KafkaConsumer<String, String> consumer;
    private ConsumerRecords<String, String> msgList;
    private final String topic;
    private static final String GROUDID = "groupA";

    public ConsumerDemo(String topicName){
        Properties props = new Properties();
        props.put("bootstrap.servers", "master:9092,slave1:9092,slave2:9092");
        props.put("group.id", GROUDID);
        props.put("enable.auto.commit", "true");
        props.put("auto.commit.interval.ms", "1000");
        props.put("session.timeout.ms", "30000");
        props.put("auto.offset.reset", "earliest");
        props.put("key.deserializer", StringDeserializer.class.getName());
        props.put("value.deserializer", StringDeserializer.class.getName());
        this.consumer = new KafkaConsumer<String, String>(props);
        this.topic = topicName;
        this.consumer.subscribe(Arrays.asList(topic));
    }

    @Override
    public void run(){
        int messageNum = 1;
        try{
            for (;;){
                msgList = consumer.poll(500);
                if (msgList!=null && msgList.count()>0){
                    for (ConsumerRecord<String, String> record : msgList){
                        if (messageNum % 50 ==0){
                            System.out.println(messageNum+"=receive: key = " + record.key() + ", value = " + record.value()+" offset==="+record.offset());
                        }
                        if (messageNum % 1000 == 0)
                            break;
                        messageNum++;
                    }
                }
                else{
                    Thread.sleep(1000);
                }
            }
        }
        catch (InterruptedException e){
            e.printStackTrace();
        }
        finally{
            consumer.close();
        }
    }

    public static void main(String[] args){
        ConsumerDemo demo = new ConsumerDemo("t_topic");
        Thread thread = new Thread(demo);
        thread.start();
    }
}

运行说明

在eclipse中使用两个控制台查看输出，由于先前的配置，可以在本地通过代码来监测云服务器集群中的运行情况。生产者生产出的消息可以被消费者消费。
两个控制台分别对应不同Java程序输出的方法点这里