如何将jar附加到正在执行作业的Spark集群上？

Question

Spark streaming is really awesome. But when I use it, I encountered an issue.

Scenario: I use the Spark Streaming to consume the message from Kafka. currently there are two topics and I hard code them. But it's not good for extensibility.

For example, if there is a new topic, I need to define a scala class for the parquet schema. then stop the running spark and start the spark again.

What I'm expecting is that spark is still running, I can add the new jar library and notify spark to load the new class in the new jar. thus spark can consume the new topic message and write the related parquet to HDFS.

It's appreciated that you can give me some suggestions about this. I searched for dynamically loading, but the question is how to attach the new jar to the existing running spark without stopping it.

Thank you in advance.

Answer 1

Metadata is an ideal solution for your case. You need to maintain a metadata service, which is consumed by the spark streaming application as a reference for its consumers.

Something like this exposed over a REST API -

{
topicName: {},
schema: {},
outputPath:
}

And add a trigger from custom SparkListener implementation.