Sending Data to Kafka Producer

Question

i am trying to read the 100k file and send it to kafka topic. Here is my Kafka Code Which sends data to Kafka-console-consumer. When i am sending data i am receiving the data like this

java.util.stream.ReferencePipeline$Head@e9e54c2

Here is the sample single record data what i am sending:

173|172686|548247079|837113012|0x548247079f|7|173|172686a|0|173|2059 22143|0|173|1|173|172686|||0|||7|0||7|||7|172686|allowAllServices|?20161231:22143|548247079||0|173||172686|5:2266490827:DCCInter;20160905152146;2784

Any suggestion to get the data which i had showned in above...Thanks

Code:

import java.io.BufferedReader;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.Properties;
import java.util.Properties;
import java.util.concurrent.ExecutionException;
import java.util.stream.Stream;

import kafka.javaapi.producer.Producer;
import kafka.producer.KeyedMessage;
import kafka.producer.ProducerConfig;

@SuppressWarnings("unused")
public class HundredKRecords { 
   private static String sCurrentLine;
   public static void main(String args[]) throws InterruptedException, ExecutionException{ 
       String fileName = "/Users/sreeeedupuganti/Downloads/octfwriter.txt";

       //read file into stream, try-with-resources
       try (Stream<String> stream = Files.lines(Paths.get(fileName))) {
           stream.forEach(System.out::println);
           kafka(stream.toString());
       } catch (IOException e) {
           e.printStackTrace();
       }
       }

   public static void kafka(String stream)  {
       Properties props = new Properties();
       props.put("metadata.broker.list", "localhost:9092");
       props.put("serializer.class", "kafka.serializer.StringEncoder");
       props.put("partitioner.class","kafka.producer.DefaultPartitioner");
       props.put("request.required.acks", "1");
       ProducerConfig config = new ProducerConfig(props);
       Producer<String, String> producer = new Producer<String, String>(config);
       producer.send(new KeyedMessage<String, String>("test",stream));
       producer.close();
   }
}

Answer 1

Problem is in line kafka(stream.toString());

Java stream class doesn't override method toString . By default it returns getClass().getName() + '@' + Integer.toHexString(hashCode()) . That's exactly that you recieve.

In order to receive in kafka the whole file, you have manually convert it to one String (array of bytes).

Please, note, that kafka has limit for message size.