Spark Streaming failed executor tasks

Question

When i look at Jobs tab on spark UI, i can see Tasks status like 20/20/ (4 failed).

Does it mean there is data loss on failed tasks? Aren't those failed tasks moved to a diff executor?

Answer 1

While you should be wary of failing tasks (they are frequently an indicator of an underlying memory issue), you need not worry about data loss. The stages have been marked as successfully completed, so the tasks that failed were in fact (eventually) successfully processed.