如何在Spark中一次应用多种方法？

Question

df is a dataframe contains all car data(| id | time | speed | gps |...|);

trips is a series list contains(id,start,end) which generate from df.

method1 is used to get each id's stats information. method2 is used to get each id's other stats information.

Like this code:

val a = method1(trips,df,sc)
val b = method2(trips,df,sc)
val c = method3(trips,df,sc)
val d = method4(trips,df,sc)
val e = method5(trips,df,sc)
val f = method6(trips,df,sc)

Because each method take a certain time, is there any way to apply the methods for assignments at the same time? The type of a,b...,f is dataframe.

Answer 1

yes you can run multiple jobs at a same time in spark cluster with the help of asynchronous actions like collectAsync(),countAsync() etc.

 yo just set configuration with context .set("spark.scheduler.mode", "FAIR") 

And use asynchronous actions and so all jobs runs asynchronously and its return future so your methods also return future so all methods run at a time.