为Kafka编译自定义生成器

Question

I've downloaded Kafka from https://www.apache.org/dyn/closer.cgi?path=/kafka/0.8.1.1/kafka_2.8.0-0.8.1.1.tgz and set up a kafka cluster on my machine using VMs. The cluster is working fine - it was tested using the console producer and consumer provided with the kafka package.

Now, I've implemented a custom Producer class for Kafka. But I couldn't figure out how to compile this class and what the dependencies are.

Questions

• Could someone explain how I need to go about fetching the dependencies for the Producer, building the class and running it?

• Do I need sbt to build it? I couldn't find any online resources which clearly explained how to go about building a custom kafka producer class.

Following are the packages imported in the Producer class:

org.apache.kafka.clients.producer.Callback;
org.apache.kafka.clients.producer.KafkaProducer;
org.apache.kafka.clients.producer.ProducerConfig;
org.apache.kafka.clients.producer.ProducerRecord;
org.apache.kafka.clients.producer.RecordMetadata;
org.apache.kafka.common.record.Records

Thanks in advance

Answer 1

I developed a custom Kafka producer, as a Maven project, dependency I used was:

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>0.8.2.0</version>
</dependency>

Imports I used:

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.ByteArraySerializer;
import org.apache.kafka.common.serialization.StringSerializer;

A snippet of my producer message sending code:

Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, zkConnection);
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, ByteArraySerializer.class);

byte[] byteData = null;
File myInputFile = new File(...);
try (InputStream inputStream = new FileInputStream(myInputFile)) {
    byteData = IOUtils.toByteArray(inputStream);
}

try (KafkaProducer<String, byte[]> producer = new KafkaProducer<String, byte[]>(props)) {
    producer.send(new ProducerRecord<String, byte[]>(topic, byteData));
}