火花流失败的批次

Question

I see some failed batches in my spark streaming application because of memory related issues like

Could not compute split, block input-0-1464774108087 not found

, and I was wondering if there is a way to re process those batches on the side without messing with the current running application, just in general , does not have to be the same exact exception.

Thanks in advance Pradeep

Answer 1

This may happen in cases where your data ingestion rate into spark is higher than memory allocated or can be kept. You can try changing StorageLevel to MEMORY_AND_DISK_SER so that when it is low on memory Spark can spill data to disk. This will prevent your error.

Also, I don't think this error means that any data was lost while processing, but that input block which was added by your block manager just timed out before processing started.

Check similar question on Spark User list .

Edit:

Data is not lost, it was just not present where the task was expecting it to be. As per Spark docs :

You can mark an RDD to be persisted using the persist() or cache() methods on it. The first time it is computed in an action, it will be kept in memory on the nodes. Spark's cache is fault-tolerant – if any partition of an RDD is lost, it will automatically be recomputed using the transformations that originally created it.