Separate StreamingContext from Receiver in Spark Streaming

Question

I'd like to generalize the reception in my Main. After setting the SparkConf and the JavaContextStreaming, I'd like to receive an arbitrary object and then pass it to the analyzer.

In the cases below I get an exception: Task not serializable

Main.java
/**
*
**/
  SparkConf conf = new SparkConf().setMaster("local[*]").setAppName("MyApp");
  JavaStreamingContext jsc = new JavaStreamingContext(conf, BATCH_DURATION);
  JavaReceiverInputDStream<String> input = jsc.socketTextStream(HOST, PORT);
  OtherClass.analyze(input);
/*
 */
  jsc.start();
  jsc.awaitTermination();
  jsc.close();

OtherClass.java
/**
*
**/
public void analyze(JavaReceiverInputDStream<String> input){
  JavaPairDStream<String, String> events = input.mapToPair( ...);
// other code
//
// Task not serializable (at the line where I call lines.mapToPair(...))
//
  }

I've tried also to put the receiver insiede the method, and I get the same result at the same line.

Main.java
/**
*
**/
  SparkConf conf = new SparkConf().setMaster("local[*]").setAppName("MyApp");
  JavaStreamingContext jsc = new JavaStreamingContext(conf, BATCH_DURATION);
  OtherClass.analyze(jsc);
/*
 */

OtherClass.java
/**
*
**/
public void analyze(JavaStreamingContext jsc){

  JavaReceiverInputDStream<String> input = jsc.socketTextStream(HOST, PORT);
  JavaPairDStream<String, String> events = input.mapToPair( ...);
// other code
//
// Task not serializable (at the line where I call lines.mapToPair(...))
//
  jsc.start();
  jsc.awaitTermination();
  jsc.close();
}

There is a way to separate the receiver (in this case JavaReceiverInputDStream, but can easily be JavaDStream) from the logic part that analyze the data? Alternately, there is a way to separate the JavaStreamingContext form the receiver and the part that analyze the data?

Answer 1

Solution by OP.

I just needed to implement Serializable (java.io) in OtherClass.java. Apparently it is necessary for every environment, including in a local environment.