Does Spark read data from Kafka partition into executor, for a batch which is queued?

Question

During spark streaming with streaming-kafka-0-8-integration Direct Approach, If the batches are getting queued, will the executors pull the data for queued batches into their memory? If not, what is the harm in having a very long backlog of batches?

Answer 1

是的，Spark将从Kafka Queue中提取数据并在内存上进行处理，并且由于Kafka的批次积压很长，因此危害将是对Kafka资源的压力。