如何通过Scala spark-shell将大型RDD写入本地磁盘？

Question

Through a Scala spark-shell, I have access to an Elasticsearch db using the elasticsearch-hadoop-5.5.0 connector.

I generate my RDD by passing the following command in the spark-shell:

val myRdd = sc.esRDD("myIndex/type", myESQuery)

myRDD contains 2.1 million records across 15 partitions. I have been trying to write all the data to a text file(s) on my local disk but when I try to run operations that convert the RDD to an array, like myRdd.collect(), I overload my java heap.

Is there a way to export the data (eg. 100k records at a time) incrementally so that I am never overloading my system memory?

Answer 1

When you use saveAsTextFile you can pass your filepath as "file:///path/to/output" to have it save locally.

Another option is to use rdd.toLocalIterator Which will allow you to iterate over the rdd on the driver. You can then write each line to a file. This method avoids pulling all the records in at once.