如何在pyspark代码中创建python线程

Question

I have around 70 hive queries which I am executing in pyspark in sequence. I am looking at ways to improve the runtime be running the hive queries in parallel. I am planning to do this by by creating python threads and running the sqlContext.sql in the threads. Would this create threads in driver and improve performance.

Answer 1

I am assuming, you do not have any dependency on these hive queries and so they can run in parallel. You can accomplish this by threading, but not sure of the benefit in a single user application - because the total number of resources is fixed for your cluster ie the total time to finish the all the queries will be the same - as the spark scheduler will round robing across these individual jobs - when you multi thread it.

https://spark.apache.org/docs/latest/job-scheduling.html explains this 1) SPARK by default uses a FIFO scheduler ( which you are observing) 2) By threading you can use a "fair" scheduler 3) Ensure the method that is being threaded -set this sc.setLocalProperty("spark.scheduler.pool", ) 4) The pool id needs to be different for each thread

Example use case of threading from a code perspective:

# set the spark context to use a fair scheduler mode
conf = SparkConf().setMaster(...).setAppName(...)
conf.set("spark.scheduler.mode", "FAIR")
sc = new SparkContext(conf)


# runs a query taking a spark context, pool_id and query..
def runQuery(sc,<POOL_ID>,query):
    sc.setLocalProperty("spark.scheduler.pool", pool_id)
    .....<your code>
    return df

t1 = threading.thread(target=runQuery,args=(sc,"1",<query1>)
t2 = threading.thread(target=runQuery,args=(sc,"2",<query2>)

# start the threads...
t1.start()
t2.sart()

# wait for the threads to complete and get the returned data frames...
df1 = t1.join()
df2 = t2.join()

Like the spark documentation indicates, you will not observe an improvement in the overall throughput.. it is suited for multi-user sharing of resources. Hope this helps.