Pyspark并行执行多个作业

Question

I have the following situation with my Pyspark:

In my driver program (driver.py), I call a function from another file (prod.py)

latest_prods = prod.featurize_prods(). 

Driver code:

from Featurize import Featurize
from LatestProd import LatestProd
from Oldprod import Oldprod

sc = SparkContext()

if __name__ == '__main__':
    print 'Into main'

featurize_latest = Featurize('param1', 'param2', sc)

latest_prod = LatestProd(featurize_latest)
latest_prods = latest_prod.featurize_prods()
featurize_old = Featurize('param3', 'param3', sc)

old_prods = Oldprod(featurize_old)
old_prods = oldprod.featurize_oldprods()
total_prods =  sc.union([latest_prods, old_prods])

Then I do some some reduceByKey code here... that generates total_prods_processed .

Finally I call:

total_prods_processed.saveAsTextFile(...)

I would like to generate latest_prods and old_prods in parallel. Both are created in the same SparkContext . Is it possible to do that? If not, how can I achieve that functionality?

Is this something that does Spark automatically? I am not seeing this behavior when I run the code so please let me know if it is a configuration option.

Answer 1

After searching on the internet, I think your problem can be addressed by threads. It is as simple as create two threads for your old_prod and latest_prod work.

Check this post for a simplified example. Since Spark is thread-safe, you gain the parallel efficiency without sacrifice anything.

Answer 2

The short answer is no, you can't schedule operations on two distinct RDDs at the same time in the same spark context. However there are some workarounds, you could process them in two distinct SparkContext on the same cluster and call SaveAsTextFile. Then read both in another job to perform the union. (this is not recommended by the documentation). If you want to try this method, it is discussed here using spark-jobserver since spark doesn't support multiple context by default : https://github.com/spark-jobserver/spark-jobserver/issues/147

However according to the operations you perform there would be no reason to process both at the same time since you need the full results to perform the union, spark will split those operations in 2 different stages that will be executed one after the other.