使用KafkaUtils.createRDD（spark和kafka）创建JavaPairRDD

Question

I'm writing a batch job to replay events from Kafka. Kafka v. 0.10.1.0 and spark 1.6.

I'm trying to use the JavaPairRDD javaPairRDD = KafkaUtils.createRDD(...) call:

Properties configProperties = new Properties();
configProperties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "10.4.1.194:9092");
configProperties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
configProperties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
        org.apache.kafka.clients.producer.Producer producer = new KafkaProducer(configProperties);
for (String topic : topicNames) {
            List<PartitionInfo> partitionInfos = producer.partitionsFor(topic);
    for (PartitionInfo partitionInfo : partitionInfos) {
                log.debug("partition leader id: {}", partitionInfo.leader().id());
        JavaSparkContext sparkContext = new JavaSparkContext(sparkConf);
        Map<String, String> kafkaParams = new HashMap();
        kafkaParams.put("metadata.broker.list", "10.4.1.194:9092");
        kafkaParams.put("zookeeper.connect", "10.4.1.194:2181");
        kafkaParams.put("group.id", "kafka-replay");
        OffsetRange[] offsetRanges = new OffsetRange[]{OffsetRange.create(topic, partitionInfo.partition(), 0, Long.MAX_VALUE)};

        JavaPairRDD<String, String> javaPairRDD = KafkaUtils.createRDD(
                        sparkContext,
                        String.class,
                        String.class,
                        StringDecoder.class,
                        StringDecoder.class,
                        kafkaParams,
                        offsetRanges);

        javaPairRDD
                .map(t -> getInstrEvent(t._2))
                .filter(ie -> startTimestamp <= ie.getTimestamp() && ie.getTimestamp() <= endTimestamp)
                .foreach(s -> System.out.println(s));
    }
}

However it fails with the error:

2016-12-14 15:45:44,700 [main] ERROR     com.goldenrat.analytics.KafkaToHdfsReplayMain - error
org.apache.spark.SparkException: Offsets not available on leader:     OffsetRange(topic: 'sfs_create_room', partition: 0, range: [1 -> 100])
    at     org.apache.spark.streaming.kafka.KafkaUtils$.org$apache$spark$streaming$kaf    ka$KafkaUtils$$checkOffsets(KafkaUtils.scala:200)
    at org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createRDD$1.apply(KafkaUtils.scala:253)
    at org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createRDD$1.apply(KafkaUtils.scala:249)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
    at org.apache.spark.SparkContext.withScope(SparkContext.scala:714)
    at org.apache.spark.streaming.kafka.KafkaUtils$.createRDD(KafkaUtils.scala:249)
    at org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createRDD$3.apply(KafkaUtils.scala:338)
    at org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createRDD$3.apply(KafkaUtils.scala:333)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
    at org.apache.spark.SparkContext.withScope(SparkContext.scala:714)
    at org.apache.spark.streaming.kafka.KafkaUtils$.createRDD(KafkaUtils.scala:333)
    at org.apache.spark.streaming.kafka.KafkaUtils.createRDD(KafkaUtils.scala)
    at com.goldenrat.analytics.KafkaToHdfsReplayMain$KafkaToHdfsReplayJob.start(KafkaToHdfsReplayMain.java:172)

I can use other clients to connect to the broker, and fetch messages, so I know it's not the broker. Any help?

Answer 1

Looks like you cannot specify a non-existing offset for your range. I was hoping i could get all offsets by specifying 0 to Long.MAX_VALUE, but it fails if the offset is invalid with that error message. If I specify a valid offset (min/max) for the range, it does work. For anyone else that stumbles upon this, you can get them with something like:

        Properties configProperties = new Properties();
        configProperties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "10.4.1.194:9092");
        configProperties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
        configProperties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
        org.apache.kafka.clients.producer.Producer producer = new KafkaProducer(configProperties);
        for (String topic : topicNames) {
            offsets.get(topic).getMinimum(), offsets.get(topic).getMaximum());
            log.debug("doing topic: {}", topic);
            List<PartitionInfo> partitionInfos = producer.partitionsFor(topic);
            for (PartitionInfo partitionInfo : partitionInfos) {

                TopicAndPartition topicAndPartition = new TopicAndPartition(topic, partitionInfo.partition());
                Map<TopicAndPartition, PartitionOffsetRequestInfo> requestInfo = new HashMap<>();
                SimpleConsumer consumer = new SimpleConsumer("10.4.1.194", 9092, 10000, 64 * 1024, "kafka-replay");

                requestInfo.put(topicAndPartition, new PartitionOffsetRequestInfo(kafka.api.OffsetRequest.EarliestTime(), 1));
                kafka.javaapi.OffsetRequest request = new kafka.javaapi.OffsetRequest(requestInfo, kafka.api.OffsetRequest.CurrentVersion(), "kafka-replay");
                OffsetResponse response = consumer.getOffsetsBefore(request);
                if (response.hasError()) {
                    log.error("error, " + response.errorCode(topic, partitionInfo.partition()));
                }
                long[] earliestOffsetsArray = response.offsets(topic, partitionInfo.partition());


                requestInfo = new HashMap<>();
                requestInfo.put(topicAndPartition, new PartitionOffsetRequestInfo(kafka.api.OffsetRequest.LatestTime(), 1));
                request = new kafka.javaapi.OffsetRequest(requestInfo, kafka.api.OffsetRequest.CurrentVersion(), "kafka-replay");

                response = consumer.getOffsetsBefore(request);
                if (response.hasError()) {
                    log.error("error, " + response.errorCode(topic, partitionInfo.partition()));
                }
                long[] latestOffsetsArray = response.offsets(topic, partitionInfo.partition());

                ...