如何在Apache Spark中缓存可被其他Spark作业使用的数据

Question

I have a simple spark code in which I read a file using SparkContext.textFile() and then doing some operations on that data, and I am using spark-jobserver for getting output. In code I am caching the data but after job ends and I execute that spark-job again then it is not taking that same file which is already there in cache. So, every time file is getting loaded which is taking more time.

Sample Code is as:

val sc=new SparkContext("local","test")
val data=sc.textFile("path/to/file.txt").cache()
val lines=data.count()
println(lines)

Here, if I am reading the same file then when I execute it second time then it should take data from cache but it is not taking that data from cache.

Is there any way using which I can share the cached data among multiple spark jobs?

Answer 1

是的-通过在RDD上调用持久/缓存，您可以在相同的上下文中获得并提交其他作业