Spark Structured Streaming：处理负载是否会影响输入速率/numInputRecords？

Question

My current Structured Streaming application writes to a huge Delta table. When I (stop the stream) and point it to write to a brand new delta table:

1. It becomes much faster - batch duration drops almost 1/4th
2. The input rate increases almost 3 times

I understand it might become faster since any aggregations/writes it is doing on the older/bigger table is not needed on the new table. But the input rate change is something I am hoping someone can explain?

Source is Azure EventHubs.

Thanks!

Answer 1

Answering my own question:

The logic behind InputRate and Processing Rate seems to be the below:

Input rate =  numInputRows (or batch size )/ Trigger Interval in secs
Processing Rate = numInputRows (or batch size )/ Batch Duration in secs

without trigger interval, they shd be the same because BatchDuration = Trigger Interval.

So with a bigger table with lot of partitions, the writes and aggregates take longer which incerases the Batch Duration and thereby decreases the InputRate (and Processing Rate). So that should explain the opposite case for smaller target tables having faster input/processing rates.