如何激发提交Spark Streaming应用程序

Question

I am new to Spark and does not have too much idea on it. I am working on an application in which data is traversing on different-2 Kafka topic and Spark Streaming reading the data from this topic. Its a SpringBoot project and i have 3 Spark consumer classes in it. The job of these SparkStreaming classes is to consume the data from a Kafka topic and send it to another topic. Code of SparkStreaming class is below-

    @Service
public class EnrichEventSparkConsumer {

    Collection<String> topics = Arrays.asList("eventTopic");

    public void startEnrichEventConsumer(JavaStreamingContext javaStreamingContext) {

        Map<String, Object> kafkaParams = new HashedMap();
        kafkaParams.put("bootstrap.servers", "localhost:9092");
        kafkaParams.put("key.deserializer", StringDeserializer.class);
        kafkaParams.put("value.deserializer", StringDeserializer.class);
        kafkaParams.put("group.id", "group1");
        kafkaParams.put("auto.offset.reset", "latest");
        kafkaParams.put("enable.auto.commit", true);


        JavaInputDStream<ConsumerRecord<String, String>> enrichEventRDD = KafkaUtils.createDirectStream(javaStreamingContext,
                LocationStrategies.PreferConsistent(),
                ConsumerStrategies.<String, String>Subscribe(topics, kafkaParams));

        JavaDStream<String> enrichEventDStream = enrichEventRDD.map((x) -> x.value());
        JavaDStream<EnrichEventDataModel> enrichDataModelDStream = enrichEventDStream.map(convertIntoEnrichModel);

        enrichDataModelDStream.foreachRDD(rdd1 -> {
            saveDataToElasticSearch(rdd1.collect());
        });

        enrichDataModelDStream.foreachRDD(enrichDataModelRdd -> {
            if(enrichDataModelRdd.count() > 0) {
                if(executor != null) {
                    executor.executePolicy(enrichDataModelRdd.collect());       
                }
            }
        }); 

    }

    static Function convertIntoEnrichModel = new Function<String, EnrichEventDataModel>() {

        @Override
        public EnrichEventDataModel call(String record) throws Exception {
            ObjectMapper mapper = new ObjectMapper();
            EnrichEventDataModel csvDataModel = mapper.readValue(record, EnrichEventDataModel.class);
            return csvDataModel;
        }
    };

    private void saveDataToElasticSearch(List<EnrichEventDataModel> baseDataModelList) {
        for (EnrichEventDataModel baseDataModel : baseDataModelList)
            dataModelServiceImpl.save(baseDataModel);
    }
}

I am calling the method startEnrichEventConsumer() using CommandLineRunner.

public class EnrichEventSparkConsumerRunner implements CommandLineRunner {

    @Autowired
    JavaStreamingContext javaStreamingContext;

    @Autowired
    EnrichEventSparkConsumer enrichEventSparkConsumer;

    @Override
    public void run(String... args) throws Exception {
        //start Raw Event Spark Cosnumer.
        JobContextImpl jobContext = new JobContextImpl(javaStreamingContext);

        //start Enrich Event Spark Consumer.
        enrichEventSparkConsumer.startEnrichEventConsumer(jobContext.streamingctx());
    }

}

Now i want to submit these three Spark Streaming classes on to the cluster. I read somewhere that i have to create a Jar file first then after it i can use Spark-submit command but i have some questions in my mind -

1. Should i create a different project with these 3 Spark Streaming classes?
2. As of now i am using CommandLineRunner to initiate SparkStreaming then when to submit cluster , should i create main() method in these class?

Please tell me how to do it. Thanks in advance.

Answer 1

• No need for a different project.
• You should create entry point/ main which is responsible of the JavaStreamingContext creation.
• Create your jar with dependencies, the dependencies in one single jar file, don't forget to put provided scope for all your spark dependencies since you will use cluster's libraries.

---

Executing assembled Spark application is using spark-submit command-line application as follows:

./bin/spark-submit \
  --class <main-class> \
  --master <master-url> \
  --deploy-mode <deploy-mode> \
  --conf <key>=<value> \
  ... # other options
  <application-jar> \
  [application-arguments]

For local submit

bin/spark-submit \
  --class package.Main \
  --master local[2] \
  path/to/jar argument1 argument2