Scala文件中的RDD处理

Question

I have loaded the 2 csv files,converted RDD to DF, and I have written some JOIN conditions to perform on them. I have used spark shell for these. Now I want to put/bundle all these commands in a .scala file and run through spark-submit job. Currently I am not using any IDE for this and want to run from terminal. Do I need to have main method for this? If yes kindly suggest how can I proceed with this?

Thanks much for your time and inputs.

Answer 1

You don't need main method to run Scala script in spark shell.

1.Write your all the steps in a file and save as file.scala

2.Run the spark shell like spark-shell -i C:\\spark\\file.scala

Below is the sample code I wrote in file.scala

val rdd=sc.textFile("C:\\Users\\manoj kumar dhakad\\Desktop\\scores.txt")
rdd.collect.foreach(println)

Below is the way in which I submitted it.

spark-shell -i C:\spark\file.scala

Sample output

rdd: org.apache.spark.rdd.RDD[String] = C:\Users\manoj kumar dhakad\Desktop\scores.txt MapPartitionsRDD[1] at textFile at <console>:24
75,89,150,135,200,76,12,100,150,28,178,189,167,200,175,150,87,99,129,149,176,200,87,35,157,189