将 dataframe 写入现有 csv 文件 scala

Question

I have the following data frame with data

+---------------------------+-------+
|sport                      |value  |
+---------------------------+-------+
|table tennis               |12     |
+---------------------------+-------+

and I want to write this dataframe into an exisiting csv file. and my code is follows

val existingSparkSession = SparkSession.builder().getOrCreate()
    import existingSparkSession.implicits._
    val data = Seq((inputSentence, analysedCategoryLabel))
    val emojiRdd = existingSparkSession.sparkContext.parallelize(data)
    val finalEmojiAnalyzedDataFrame = emojiRdd.toDF("sport", "value")
    finalEmojiAnalyzedDataFrame.write.format("com.springml.spark.sftp").option("delimiter",";").mode(SaveMode.Append).save("./src/main/resources/sportsData.csv") 

But this code isn't working and im getting the following error.

Exception in thread "main" java.lang.ClassNotFoundException: Failed to find data source: com.springml.spark.sftp.

To write into a csv file continuously do I need to use com.springml.spark.sftp? are there any other way of doing it? If this is the only way do I need to add this library import into my build file in scala?

Answer 1

It's not possible to save to a single file across multiple partitions because Spark is meant to be a distributed processing library and write to a shared filesystem

In other words, the output path needs to be a directory

Otherwise, you would need to collect the dataframe to row objects, then use non-Spark methods to write/append to a single, local file