如何在 AWS 上使用 saveAsTextFile？

Question

I hope to get something by println, but by using AWS, it may not work, How can I save the content of println as a file on AWS using "saveAsTextFile"? The original content of println is as following:

println("\n[ First output is ]")
output1.foreach(a => println("(" + a +"," + titles(a - 1) + ")"));
println("\n[ Second output us ]")
output2.foreach(a => println("(" + a +"," + titles(a - 1) + ")"));

output1 and output2 are both list made up of numbers. titles is also a list. Thanks.

Answer 1

Well if both are List s, you may convert them into RDD s, using SparkContext 's method parallelize .

val rdd1 = sc.parallelize(List("[ First output is ]") ++ output1.map(a => "(" + a + "," + titles(a - 1) + ")"))
val rdd2 = sc.parallelize(List("[ Second output is ]") ++ output2.map(a => "(" + a + "," + titles(a - 1) + ")"))

After this you can use saveAsTextFile , in your desired s3 path.

rdd1.saveAsTextFile("s3://yourAccessKey:yourSecretKey@/out1.txt")
rdd2.saveAsTextFile("s3://yourAccessKey:yourSecretKey@/out2.txt")

I recommend you to read this blog, it might help you do understand important things about S3 and Apache-Spark Writing s3 data with Apache Spark