在 spark/scala 的另一个数据框中查找多个列值

Question

I am having 2 data frames A & B.

A is having 30 columns- reason1,reason2.......reason30

B having 2 columns- reason, Value

Now I need to look for all the columns starting with reason* into B and fetch the corresponding value in one column in data frame A.

So, the Final data frame would have reason1,reason2.......reason30, value

I was trying to join each column with other data frame but that's not an neat approach.

Please help me to get an optimized and fast solution using spark/scala.

Answer 1

You can get the Array of columns of a dataframe using

df.columns 

Then, you can iterate over the Array and check which column is present and prepare the join condition