通过Kafka和Spark消费大数据

Question

I have a stream data in Json that provides by Websocket that size is changing between 1MB and 60 MB per second.

I got to decode the data then parse it and finally write to mysql.

I thought 2 ideas:

1) To read data from Socket then decode the data and send to Consumer via Avro in Producer, Then to get the data and write to mysql on Spark map, reduce in Consumer

2) To read data from Socket then send the data to Consumer in Producer, Then to get the data in Consumer then decode on Spark and send the parsed data to Spark Job for writing to mysql.

Do you have any idea?

Producer

/*
 * To change this license header, choose License Headers in Project Properties.
 * To change this template file, choose Tools | Templates
 * and open the template in the editor.
 */
package com.tan;


import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.ProducerConfig;

import java.util.Properties;


import java.util.stream.Stream;

import java.io.BufferedReader;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Paths;
/**
 *
 * @author Tan
 */
public class MainKafkaProducer {

    /**
     * @param args the command line arguments
     */
    public static void main(String[] args) throws InterruptedException {
        // TODO code application logic here
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.ByteArraySerializer");
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");

        //props.put("group.id", "mygroup");
        //props.put("max.partition.fetch.bytes", "100000000");
        //props.put("serializer.class", "kafka.serializer.StringEncoder");
        //props.put("partitioner.class","kafka.producer.DefaultPartitioner");
        //props.put("request.required.acks", "1");

         KafkaProducer<String, String> producer = new KafkaProducer<>(props);

         // Read the data from websocket and send it to consumer
         //for (int i = 0; i < 100; i++) {
            String fileName = "/Users/Tan/Desktop/feed.json";
            try{
                BufferedReader file = new BufferedReader(new FileReader(fileName));
                String st = file.readLine();
                for(int i = 0; i < 100; i++)
                {
                    ProducerRecord<String, String> record = new ProducerRecord<>("mytopic", st);
                    producer.send(record);
                }
            }catch(IOException e){
                e.printStackTrace();
            }
        //}

        /*
        for(int i = 0; i < 100; i++)
        {
            ProducerRecord<String, String> record2 = new ProducerRecord<>("mytopic", "Hasan-" + i);
            producer.send(record2);
        }
        */


        producer.close();
    }

}

Consumer

/*
 * To change this license header, choose License Headers in Project Properties.
 * To change this template file, choose Tools | Templates
 * and open the template in the editor.
 */
package com.tan;

import kafka.serializer.DefaultDecoder;
import kafka.serializer.StringDecoder;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.streaming.Duration;
import org.apache.spark.streaming.api.java.JavaPairInputDStream;
import org.apache.spark.streaming.api.java.JavaStreamingContext;
import org.apache.spark.streaming.kafka.KafkaUtils;

import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
import java.util.Set;
/**
 *
 * @author Tan
 */
public class MainKafkaConsumer {
    /**
     * @param args the command line arguments
     */
    public static void main(String[] args) {

        SparkConf conf = new SparkConf()
                .setAppName(MainKafkaConsumer.class.getName())
                .setMaster("local[*]");
        JavaSparkContext sc = new JavaSparkContext(conf);
        JavaStreamingContext ssc = new JavaStreamingContext(sc, new Duration(2000));

        Set<String> topics = Collections.singleton("mytopic");
        Map<String, String> kafkaParams = new HashMap<>();
        kafkaParams.put("metadata.broker.list", "localhost:9092");

        JavaPairInputDStream<String, String> directKafkaStream = KafkaUtils.createDirectStream(ssc, 
                String.class, String.class, 
                StringDecoder.class, StringDecoder.class, 
                kafkaParams, topics);

        directKafkaStream.foreachRDD(rdd -> {

            rdd.foreach(records -> {

                System.out.println(records._2);

            });

        });
        /*
        directKafkaStream.foreachRDD(rdd -> {
            System.out.println("--- New RDD with " + rdd.partitions().size()
                    + " partitions and " + rdd.count() + " records");
            rdd.foreach(record -> {
                System.out.println(record._2);
            });
        });
        */



        ssc.start();
        ssc.awaitTermination();

    }

}

Answer 1

Your process is good, the point is just for the avro conversion. Your data is not that big, 1Mb to 60Mb.

Here I have a similar process, reading from an MQ, process the data, convert to avro, send to kafka, consume from kafka, parse the data and post in other MQ.

The Avro help a lot when our data is huge, like >= 1Gb. But in some cases our data is really small like < 10Mb. In this case Avro make our processing be a little bit slow, there is no gain in network transmission.

What I suggest to you, if your network is good enought to not convert to avro, better do it without avro. To improove performance in Spark Side configure the topic of kafka with a good number of partitions, because if you have just one partition, your spark will not do the parallization correctly. Check this text that can help you.