类型不匹配 Spark Scala

Question

I am trying to create an empty dataframe an using it on a function but I am having the following error all time:

Required: DataFrame
Found: Dataset[DataFrame]

This is how I am doing it:

//Create empty DataFrame
val schema = StructType(
    StructField("g", StringType, true) ::
    StructField("tg", StringType, true) :: Nil)

var df1 = spark.createDataFrame(spark.sparkContext
      .emptyRDD[Row], schema)
//or
var df1 = spark.emptyDataFrame

Then I try to use it calling a functions as you can see here:

  df1 = kvrdd1_toDF.map(x => function1(x, df1))

And this is the function:

  def function1(input: org.apache.spark.sql.Row, df: DataFrame): DataFrame = {
    val v1 = spark.sparkContext.parallelize(Seq("g","tg"))
    var df3 = v1.toDF("g","tg")
    if (df.take(1).isEmpty){
      df3 = Seq((input.get(2), "nn")).toDF("g", "tg")
    } else {
      df3 = df3.union(df)
    }
    df3
  }

What am I doing wrong?

Answer 1

You have a DataFrame which is an alias for Dataset[Row] . You map that Row to a DataFrame so that's how you end up with a Dataset[DataFrame] . I don't know what you are trying to do but it will never work. The functions (and all its dependencies) you use to map the contents of a Dataset are serialized and distributed over your spark cluster. You can't use another DataFrame or a SparkSession or SparkContext in such a function.