table of Contents
1. The data obtained after Flume docks with Kafka has double quotes
三、Failed to construct kafka consumer
Fourth, a single node fails to create a topic
Five, Slipstream built es table error
1. The data obtained after Flume docks with Kafka has double quotes
【Problem Description】
As shown in the figure below, although Flume, Kafka, and Slipstream are successfully connected and the data is retrieved, the data queried in the Inceptor are all with double quotes.
【problem causes】
The possible reason is that the default transmission mechanism of Flume is to use json string to transmit data, so the transmitted data are all enclosed in double quotes.
【Solution】
Flume began to support the interceptor function after version 1.7. The interceptor can be configured in the Flume configuration file (this is the edited configuration file flume-topic-oracle.conf) to achieve the purpose of filtering double quotes. The configuration method is as follows:
Parameter explanation:
(1) a1.sources.r1.interceptors: define the name of the interceptor;
(2) a1.sources.r1.interceptors.i1.type: define the type of interceptor (here is search and replace);
(3) a1.sources.r1.interceptors.i1.searchPattern: define the characters to be searched;
(4) a1.sources.r1.interceptors.i1.replaceString: defines the character to be replaced.
After the configuration is complete, restart the Flume component, insert a few pieces of data in the Oracle database for testing, and the results obtained in the Inceptor show normal.
二、Failed to create topic
【Problem Description】
When creating a Kafka topic, it shows that the topic creation fails. The specific errors are as follows:
【problem causes】
（1）./kafka-topics.sh --create --topic qwe --zookeeper node1:2181,node2:2181,node3:2181 --partition 3 --replication-factor 3
（2）./kafka-broker-topics.sh --bootstrap-server node3:9092,node2:9092,node1:9092 --create --topic test3 --partitions 3 --replication-factor 3 --consumer.config ../config/consumer.properties
(1) The way to create a topic is to create a topic after obtaining the controller information through zookeeper verification; (2) The way to create a topic is not to pass the zookeeper verification, so the controller information cannot be obtained. At this time, kafka will think The node that connects to the server fastest is the controller. Once it is different from the controller elected in zookeeper, the above error will be reported.
【Solution】
1. It is recommended to use the method in (1) to create a topic.
2. If you need to use the method in (2) to create a topic, you can go to zookeeper to check which node the controller is located at, and then execute the statement that creates the topic. The specific operations are as follows:
First go to the bin directory of kafka and start the zookeeper client:
Then execute the command to get the controller:
After getting Brokerid, execute the following command:
As can be seen from the above figure, the controller at this time is located on node3, so use the following command to create a topic:
./kafka-broker-topics.sh --bootstrap-server node3:9092 --create --topic test3 --partitions 3 --replication-factor 3 --consumer.config ../config/consumer.properties三、Failed to construct kafka consumer
【Problem Description】
After the flow and table are created in the interactive interface, the following error is encountered in the process of triggering the flow:
[Possible cause of the problem]
1. The current user does not have the authority to operate the cluster; 2. There is no authority to assign production data and consumption data to the current topic; 3. The principal parameter when creating the stream is wrong.
【Solution】
Execute the following commands in the bin directory of Kafka to grant authorization (take topic indexes as an example).
--给当前用户赋操作集群的权限
./kafka-acls.sh --authorizer-properties zookeeper.connect=node3:2181,node2:2181,node1:2181 --add --allow-principal User:hive –cluster
--给当前topic赋生产数据的权限
./kafka-acls.sh --authorizer-properties zookeeper.connect=node3:2181,node2:2181,node1:2181 --add --allow-principal User:hive --topic indexs –producer
--给当前topic赋消费数据的权限
./kafka-acls.sh --authorizer-properties zookeeper.connect=node3:2181,node2:2181,node1:2181 --add --allow-principal User:hive --topic indexs --consumer --group hadoop
After empowerment, log in to the 8380 interface: "Permissions" -->KAFKA-->kafka1.
Add the "hive" user to the "GLOBAL" column of "topic authority", "consumer group authority" and "cluster authority".
Go to the /etc/slipstream1/conf directory to find the previously distributed kafka.keytab file, and execute the following command:
klist -ket kafka.keytabCheck the settings of principal parameters as follows:
Check the principal parameter in the flow establishment statement, and rebuild the flow after it is correct, and then the flow is triggered. At this time, the trigger is successful.
Fourth, a single node fails to create a topic
【Problem Description】
When using a node to create a topic, the creation fails, and the error is as follows:
【problem causes】
The Zookeeper mechanism will elect one node as the Leader, and the remaining nodes as the Follower. The failure to create a topic here indicates that node1 is not a Leader, that is, it is not a Controller.
【Solution】
Write all nodes. E.g:
./kafka-broker-topics.sh --bootstrap-server node3:9092,node2:9092,node1:9092 --create --topic yxy --partitions 3 --replication-factor 3 --consumer.config ../config/consumer.propertiesFive, Slipstream built es table error
【Problem Description】
When creating the es table in Slipstream, the error is as follows:
【problem causes】
Slipstream is not set to rely on Search.
【Solution】
Go to the Slipstream component of the 8180 monitoring interface and set Slipstream to rely on Search.