Spark Scala copy column from one dataframe to another

Question

I have a modified version of the original dataframe on which I did clustering, Now I want to bring the predicted column back to the original DF (the index is ok, so it matches). How am I supposed to do this? With this code I get an error.

println("Predicted:")
dfWithOutput.show
println("Original:")
originalDF = originalDF.withColumn("cluster", dfWithOutput.col("prediction")

Exception in thread "main" org.apache.spark.sql.AnalysisException: Resolved attribute(s) prediction#2121 missing from (list of columns in the original df)

Answer 1

您需要加入两个数据框，然后选择您感兴趣的列