如何将文本文件作为参数scala传递

Question

I wish to write my wordcount program specifically so that I can pass my input textfile as an argument in main.I am very new to scala so I don't know the specifics as to how to pass it. I tried directly mentioning it in my main function as def main(args:"C:/Users/rsjadsa/Documents/input.txt" )

import org.apache.spark.SparkContext
import org.apache.spark.SparkConf
object WordC {
 def main(args: String, args1 : String){
 val cf = new SparkConf().setAppName("WordCount").setMaster("local")
 val sc = new SparkContext(cf)
 val words = args.flatMap(line => line.split(" "))
 val wordCount = words.map(word => (word, 1)).reduceByKey(_ + _)
 wordCount.foreach(println)

 }
}

I just want to pass my textfile as a argument instead of hardcoding it and apply the same wordcount program on it. I know I am new to this language so sorry for the silly question

Answer 1

It would be the first element in the Array of strings args but it depends on how you run the program as to what you want to do. This is just reading from command line argument and assigning to variable. You also need to make a schema if you want it to be in a DataFrame (which you probably should).

EDIT: Since you want to do the wordcount piece with the RDD I took out the DataFrame stuff because it was confusing. Also, you should collect the RDD to the Driver before you print to screen or it might do crazy stuff as the data is still in the executors.

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext
import org.apache.spark.SparkConf
import sqlContext.implicits._
import org.apache.spark.sql.types.{StructType,StructField,StringType};
import org.apache.spark.sql.Row;


object WordC {
  def main(args: Array[String]): Unit = {

    // retrieve the filename 
    val filename = args(0)

    val cf = new SparkConf().setAppName("WordCount").setMaster("local")
    val sc = new SparkContext(cf)

    val inputRDD = sc.textFile(filename)

    val wordsRDD = inputRDD.flatMap(line => line.split(" "))
    val wordCountRDD = words.map(word => (word, 1)).reduceByKey(_ + _)
    wordCountRDD.collect.foreach(println(_))

  }
}

And then however you are running the program, the command line argument would just be C:/Users/rsjadsa/Documents/input.txt like scala WordC.scala "C:/Users/rsjadsa/Documents/input.txt"