kafka集群运维

kafka集群维护
一、kafka集群启停
#启动kafka
/home/cluster/kafka211/bin/kafka-server-start.sh -daemon /home/cluster/kafka211/config/server.properties

#关闭kafka
/home/cluster/kafka211/bin/kafka-server-stop.sh

二、kafka集群基本信息实时查看和修改
#列出所有有效主题
/home/cluster/kafka211/bin/kafka-topics.sh --list --zookeeper 134.32.123.101:2181,134.32.123.102:2181,134.32.123.103:2181

#查看特定主题
/home/cluster/kafka211/bin/kafka-topics.sh --describe --zookeeper 134.32.123.101:2181,134.32.123.102:2181,134.32.123.103:2181 --topic REC-CBBO-MSG-TOPIC

#创建主题尽量不要使用下划线"_"或者点好".",可以使用横线"-"
/home/cluster/kafka211/bin/kafka-topics.sh --create --zookeeper 134.32.123.101:2181,134.32.123.102:2181,134.32.123.103:2181 --replication-factor 3 --partitions 17 --topic REC-CBBO-MSG-TOPIC

#修改主题如果主题创建了之后发现分区不够用，可以增加，不可以减少
/home/cluster/kafka211/bin/kafka-topics.sh --zookeeper 134.32.123.101:2181,134.32.123.102:2181,134.32.123.103:2181 --alter --topic REC-CBBO-MSG-TOPIC --partitions 50

#删除主题 delete.topic.enable要设置为true
/home/cluster/kafka211/bin/kafka-topics.sh --delete --zookeeper 134.32.123.101:2181,134.32.123.102:2181,134.32.123.103:2181 --topic REC-CBBO-MSG-TOPIC

三、kafka集群leader平衡机制
#leader手动平衡
/home/cluster/kafka211/bin/kafka-preferred-replica-election.sh --zookeeper 134.32.123.101:2181,134.32.123.102:2181,134.32.123.103:2181
#leader自动平衡参数配置
auto.leader.rebalance.enable=true

四、kafka集群分区日志迁移
1、迁移topic数据到其他broker
1.1写json文件格式
cat topics-to-move.json
{
"topics":[{"topic","foo1"},{"topic","foo2"}],
"version":1
}

1.2使用generate生成迁移计划只是生成迁移计划并未迁移
bin/kafka-reassign-partitions.sh --zookeeper 134.32.123.101:2181,134.32.123.102:2181,134.32.123.103:2181 --topics-to-move-json-file topics-to-move.json --broker-list "5,6" --generate

Current partition replica assignment 指当前配置要保存用来回滚

Proposed partition reassignment configuration 指生成的迁移计划保存到expand-cluster-reassignment.json

1.3使用-execute执行计划执行前最好保存当前的分配情况以防出错回滚
将上一步生成的执行计划json数据保存到expand-cluster-reassignment.json
bin/kafka-reassign-partitions.sh --zookeeper 134.32.123.101:2181,134.32.123.102:2181,134.32.123.103:2181 --reassignment-json-file expand-cluster-reassignment.json --execute

1.4使用-verify验证是否已经迁移完成
bin/kafka-reassign-partitions.sh --zookeeper 134.32.123.101:2181,134.32.123.102:2181,134.32.123.103:2181 --reassignment-json-file expand-cluster-reassignment.json -verify

2、迁移某个topic的某些特定的partition的数据到其他broker 步骤与上面一样但是json文件如下
cat custom-reassignment.json
{
"version":1,"partitions":[{"topic":"foo1","partition":0,"replicas":[5,6]},{"topic":"foo2","partition":1,"replicas":[2,3]}]
}
可以指定到topic的分区编码

注意：迁移会导致leader失衡需要使用平衡命令重新平衡

猜你喜欢