由于评估是懒惰的，如何登录Spark Java API？

Question

As the evaluation in Spark is lazy, the logs tracing the time to execute a statement could go wrong. For ex: if user captures start time before reading a text file and after file has been read(before and after sc.textFile()), it would lead to wrong information as the logging will be done but the file has not yet read due to lazy evaluation. Any solution to this, like lazy logging?

Answer 1

You have to call a spark action on your RDD in order to trigger the effective computation. Transformation actions like: map, filter, read of a file etc will not trigger a spark action. In order to measure effective time between start-end logging put a spark action on the rdd (maybe you want to cache them for future use and call rdd.cache() )