如何在 Java 中附加具有不同列的两个 Spark 数据帧

Question

I have One Dataframe on which I am performing one UDF operation and then operation of UDF gives only one column in Dataframe. How i can append it to previous Datafram. Example: Dataframe 1: sr_no , name, salary

Dataframe 2: UDF is giving output as ABS(Salary) - only one column as output from UDF applied on Dataframe1

How i can have output dataframe as Dataframe1 + Dataframe2 in JAVA ie sr_no, name, salary, ABS(Salary) output

Answer 1

Looks like you are searching for .withColumn method:

df1.withColumn("ABS(salary)", yourUdf.apply(col("salary")))

(Snippet requires to import static method col from org.apache.spark.sql.functions )

Answer 2

Got the ans. Just do it like this : df= df.selectExpr("*","ABS(salary)"); This will give you output of udf with your entire dataframe. Else it will give only one column.