Spark：将CSV转换为RDD [行]

Question

I have a .csv file, which contains 258 columns in following structure.

["label", "index_1", "index_2", ... , "index_257"]

Now I wanna transform this .csv file to a RDD[Row]:

val data_csv = sc.textFile("~/test.csv")

val rowRDD = data_csv.map(_.split(",")).map(p => Row( p(0), p(1).trim, p(2).trim)) 

If I do the transform in this way, I have to write down 258 columns specifically. So I tried:

val rowRDD = data_csv.map(_.split(",")).map(p => Row( _ => p(_).trim)) 

and

val rowRDD = data_csv.map(_.split(",")).map(p => Row( x => p(x).trim))

But these two also not working and report error:

error: missing parameter type for expanded function ((x$2) => p(x$2).trim)

Can anyone tell me how to do this transform? Thanks a lot.

Answer 1

you should use sqlContext instead of sparkContext as

val df = sqlContext.read
  .format("com.databricks.spark.csv")
  .option("header", true)
  .load(("~/test.csv")

this will create dataframe . calling .rdd on df should give you RDD[Row]

val rdd = df.rdd

Answer 2

Rather reading as a textFile read CSV files with the spark-csv

In your case

val df = sqlContext.read
    .format("com.databricks.spark.csv")
    .option("header", "true") // Use first line of all files as header
    .option("inferSchema", "true") // Automatically infer data types
    .option("quote", "\"")  //escape the quotes 
    .option("ignoreLeadingWhiteSpace", true)  // escape space before your data
    .load("cars.csv")

This loads data as a dataframe, now you can easily change it to RDD.

Hope this helps!

Answer 3

Apart from the other answers that are correct, the correct way to do what you're trying to do is to use Row.fromSeq inside the map function.

val rdd = sc.parallelize(Array((1 to 258).toArray, (1 to 258).toArray) )
            .map(Row.fromSeq(_))

This will turn your rdd to type Row :

 Array[org.apache.spark.sql.Row] = Array([1,2,3,4,5,6,7,8,9,10...