Spark dataframe.map() 多次处理每一行

Question

Running this code:

(1)

val resultDf = myDataFrame.map(row => { println(s"$row"); return row })

I can see exactly one print out (use "yarn logs -applicationId xxxx" to get the log) for each row. However when the processing code is more complex:

(2)

val resultDf = myDataFrame.map(row => { println(s"$row"); /* complex processing code */})

I find about 2 or 3 times more print out than the actual row count. But in both cases myDataFrame.count == resultDf.count

Question: in case (2) I see more print out, is that because Spark runs dataFrame.map() in more containers for redundancy, and throws away extra results when redundant executions all return successfully? Thanks.

BTW I run spark jobs on aws emr, spark 3.1.2

Answer 1

The result dataframe passed on downstream, at one point it's not cached but referenced 3 times, resulting in multiple executions of myDataFrame.map().