如何从 Scala 中的数组创建 DataFrame？

Question

I have a use case where I need to create a DataFrame from an array.

I've created a DataFrame that reads a CSV then I am using a map to process/transform it further.

    var mapTransform = df1.collect.map( 
      line => {
      // line.split(",") logic for fields separation
      //transformation logic here for various fields

      (field1+","+field2+","+field3);  
      }
    )

From this, I am getting an array(Array[String]) which is transformed result.

I want to further convert it DataFrames with separate columns so that later it can be used to write to DB or file, however, I am facing an issue. Is it possible to do it? Any solutions?

Answer 1

This does your job: spark.sparkContext.parallelize(mapTransform.toSeq) But note that you must avoid methods that produce non-rdd, as they load all the contents of the array to the one node and that's ineffective in the general case.

Also, there's a convention turn var s to val s as much as possible.