来自一个 Kafka 主题源的并发 Spark stream 作业

Question

We have a simple spark stream from a kafka topic (with 8 partitions) created like below and submitted with 2 executors (4 cores each).

dataSet
   .writeStream()
   .trigger(Trigger.ProcessingTime(0))
   .format("kafka");
   .start();

Now consider this scenario:

1. One request comes to the partition #0 of this topic.
2. A spark job will start with 8 tasks and only one of them is running (the others are success).
3. Suppose it takes 1 minute to process this request.
4. During this 1 minute 100 requests comes to this topic (in all 8 partitions).
5. Spark waits for the current job to finish then it creates another job to process new requests.

Our expectation is that Spark process other requests in another job while it's processing first request, but that's not happening. Now suppose that first job takes 1 hour instead on 1 minute while the other requests waiting to be processed while 7 cores are idle. That's our problem.

I already tried to send this jobs multiple times (like 4 times) from 4 different threads but behavior is still the same. Also I tried setting this config spark.streaming.concurrentJobs to more than 1 but no change!

So my question is that is it possible to have multiple jobs for one kafka stream dataset at all? And if yes how?

We are using Spark 2, Kafka 1 and Java 8.

Answer 1

So after days of studying and testing I finally figured out that none of concurrentJob setting or sending jobs in different threads are not solutions.

The only working solution is to create different streams for each (or group of) topic partitions .

The parallelism factor in kafka is partition. And Spark (and kafka) has this ability to read only from specific partition(s). So if our partition has 4 topics I split my Spark job into 4 different jobs, each of them listening (assigning) to one partition but all of them are sinking into same target.

So now if one job is busy with a time consuming process, the other jobs (here 3) can still process data from their assigned partitions and they don't need to wait for finishing process on the other partitions.

The config is like below:

assign: {"topic-name":[0,1,2]}

instead of

subscribe: "topic-name"

Pay attention to config structure, it should be a valid JSON and list of topics should be mentioned in comma separated string (not supporting range or exclude)