如何避免在Apache Spark数据框中的列选择中进行硬编码 斯卡拉

Question

I have the following data frame and I need to run logistic regression using spark ml on it:

uid  a  b  c  label d
1    0  1  3  0     2
2    3  0  0  1     0

While using the the ml package, i came to know that I need to create the data in the format

label  feature
0      [0,1,3,2]
1      [3,0,0,0]

Now i came across VectorAssembler to create the feature column and while doing so I need to do something like

val assembler = new VectorAssembler()
.setInputCols(Array("a", "b", "c", "d"))
.setOutputCol("features")

Is there anyway i can avoid the hardcoding of individual feature column names

Answer 1

Depends on your data. If you know that you will always have a certain set of columns that is not a part for your feature vector (uid and label) AND can assume that all other columns are, you can do like this:

// df is your data frame
val assembler = new VectorAssembler()
.setInputCols(df.columns.diff(Array("uid","label")))
.setOutputCol("features")