Description:
1, this topic only implements two data synchronization, subsequent optimization will be continuously updated. . . . .
2, self-built cluster CDH5.8, kafka2.1.0; Ali cloud cluster Standard Edition kafka0.10.x
stepped pit:
1, cdh add kafka role instance CMM, should not support SSL connections
2, VPC network access, do not know examples are later aliyun VPC network, this connection is not encrypted SSL
3, kafka0.10.2 not connected to the self mirrormaker cluster
4, SSl aliyun control hint is the access point, the actual authentication mode needs SASL_SSL
. 5, understand java I do not know what this is added to the position KAFKA_OPTS = Export "- Djava.security.auth.login.config = kafka_client_jaas.conf"
6, ssl.truststore.location = kafka.client.truststore.jks
ssl.truststore.password = KafkaOnsClient this certificate is required to specify the path, and the password is fixed, I use another password for
preparation:
1, download kafka_2.12-2.2.1.tgz, Bie Li cloud of highly recommended a small version
2, download kafka. client.truststore.jks, need to be cloud with Ali, or Ali cloud document provided in the download link
3, create kafka_client_jaas.conf files manually, the following will be posted content
deployment:
1, to ensure that the server can access Q self-built cluster of 9092 and Ali cloud clusters 9093
2, upload, extract kafka_2.12-2.2.1.tgz (do not configure zookeepr here, do not start Kafka used to live)
3, New kafka_client_jaas.conf config directory file (kafka decompression directory)
4, a new directory cert, and upload kafka. client.truststore.jks certificate (Kafka decompression directory)
. 5, Vim / ERC / Profile bottommost added export KAFKA_OPTS = "- Djava.security.auth.login.config = xxxxxx / kafka_client_jaas.conf" ( It should be the actual directory)
6, edit kafka_client_jaas.conf, consumer.properties and producer.properties
7, start nohup bin / kafka-mirror-maker.sh --consumer.config config / consumer.properties --producer.config config / producer.properties --whitelist AIS_11_AisMySql, AIS_99_FWP & (background)
8, the target topic to see if there is a message
profile content
1, kafka_client_jaas.conf KafkaClient {<br/>org.apache.kafka.common.security.plain.PlainLoginModule required<br/>username="xxxxxx"<br/>password="xxxxxx";<br/>};
2, consumer.properties
3、#这里的用户和密码从阿里云控制台获取<br/>KafkaClient {<br/>org.apache.kafka.common.security.plain.PlainLoginModule required<br/>username="xxxxxx"<br/>password="xxxxxx";<br/>};