Kafka(4)Multiple Kafka and Scala Client
1. Create Multiple Nodes
>cp server.properties server1.properties
>cp server.properties server2.properties
The content are as follow:
config/sever1.properties
broker.id = 1
port =9093
log.dir=/tmp/kafka-logs-1
config/server2.properties
broker.id = 2
port = 9094
log.dir=/tmp/kafka-logs-2
Start the 2 nodes
>JMX_PORT=9997 bin/kafka-server-start.sh config/server1.properties &
>JMX_PORT=9998 bin/kafka-server-start.sh config/server2.properties &
Create the topic
>bin/kafka-create-topic.sh --zookeeper localhost:2181 --replica 2 --partition 1 --topic my-replicated-topic
List the topics
>bin/kafka-list-topic.sh --zookeeper localhost:2181
topic: my-replicated-topicpartition: 0leader: 1replicas: 1,2isr: 1,2 topic: testpartition: 0leader: 0replicas: 0isr: 0
Here is the command to kill one node
>pkill -9 -f config/server1.properties
2. The Scala Client
The Scala Class will be as follow:
"com.rabbitmq" % "amqp-client" % "3.1.4",
"org.apache.kafka" % "kafka_2.10" % "0.8.0" intransitive(),
"com.yammer.metrics" % "metrics-core" % "2.2.0",
"com.twitter" %% "util-collection" % "6.3.6",
"com.101tec" % "zkclient" % "0.3"
The producer class
package com.sillycat.superduty.jobs.producer
import java.util.Properties
import kafka.javaapi.producer.Producer
import kafka.producer.{ KeyedMessage, ProducerConfig }
object NewTaskKafka extends App {
val props2: Properties = new Properties()
props2.put("zk.connect", "localhost:2181")
props2.put("metadata.broker.list", "localhost:9092");
props2.put("serializer.class", "kafka.serializer.StringEncoder")
props2.put("zk.connectiontimeout.ms", "15000")
val config: ProducerConfig = new ProducerConfig(props2)
val producer: Producer[String, String] = new Producer[String, String](config)
val data = new KeyedMessage[String, String]("test", "test-message, it is ok")
producer.send(data)
producer.close
}
The Consumer class
package com.sillycat.superduty.jobs.consumer
import kafka.api.{ FetchRequestBuilder, FetchRequest }
import kafka.javaapi.consumer.SimpleConsumer
import kafka.javaapi.FetchResponse
import kafka.javaapi.message.ByteBufferMessageSet
import scala.collection.JavaConversions._
import java.nio.ByteBuffer
import com.typesafe.scalalogging.slf4j.Logging
import java.util.Properties
import kafka.consumer.{Consumer, ConsumerConfig}
import scala.collection.JavaConverters._
object WorkerKafka extends App with Logging {
val props = new Properties()
props.put("group.id", "console-consumer-2222222")
props.put("socket.receive.buffer.bytes", (2 * 1024 * 1024).toString)
props.put("socket.timeout.ms", (ConsumerConfig.SocketTimeout).toString)
props.put("fetch.message.max.bytes", (1024 * 1024).toString)
props.put("fetch.min.bytes", (1).toString)
props.put("fetch.wait.max.ms", (100).toString)
props.put("auto.commit.enable", "true")
props.put("auto.commit.interval.ms", (ConsumerConfig.AutoCommitInterval).toString)
props.put("auto.offset.reset", "smallest")
props.put("zookeeper.connect", "localhost:2181")
props.put("consumer.timeout.ms", (-1).toString)
props.put("refresh.leader.backoff.ms", (ConsumerConfig.RefreshMetadataBackoffMs).toString)
val config = new ConsumerConfig(props)
val consumer = Consumer.createJavaConsumerConnector(config)
val topicMap = Map[String, Integer]("test" -> 1)
println("about to get the comsumerMsgStreams")
val consumerMap = consumer.createMessageStreams(topicMap.asJava)
val streamz = consumerMap.get("test")
val stream = streamz.iterator().next()
println("listening... (?) ")
val consumerIter = stream.iterator()
while(consumerIter.hasNext()){
System.out.println("MSG -> " + new String(consumerIter.next().message))
}
}
We can use zkCli.sh to view some of the configuration.
>zkCli.sh -server localhost:2181
zkCli>ls /
zkCli>get /brokers/topics/my-replicated-topic
zkCli>get /brokers/topics/test
And here is the document for the zookeeper data structure
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper
We can also configure the multiple nodes for zookeeper in server.properties, there is a configuration like this.
zookeeper.connect=localhost:2182,localhost:2183
References:
http://kafka.apache.org/08/quickstart.html
https://cwiki.apache.org/confluence/display/KAFKA/Index
Kafka(4)Multiple Kafka and Scala Client
猜你喜欢
转载自sillycat.iteye.com/blog/2015181
今日推荐
周排行