在 Spark Scala 中选择 Dataframe 的列

Question

If you want to select the first column of a dataframe this can be done:

df.select(df.columns(0))

df.columns(0) returns a string, so by giving the name of the column, the select is able to get the column correctly.

Now, suppose I want to select the first 3 columns of the dataset, this is what I would intuitively do:

df.select(df.columns.split(0,3):_*)

The _* operator would pass the array of strings as a varag to my understanding, and it would be the same as passing (df.column(1), df.column(2), df.column(3)) to the select statement. However this doesn't work and it is necessary to do this:

import org.apache.spark.sql.functions.col
df.select(sf.columns.split(0,3).map(i => col(i)):_*))

What is going on?

Answer 1

I think in the question you meant slice instead of split .

And as for your question, df.columns.slice(0,3):_* is meant to be passed to functions with *-parameters ( varargs ), ie if you call select(columns:_*) then there must be a function defined with varargs , eg def select(cols: String*) .

But there can only be one such function defined - no overloading here is possible. Example on why it's not possible to define two different functions with same vararg -parameter declaration:

def select(cols: String*): String = "string"
select() // returns "string"
def select(cols: Column*): Int = 3
select() // now returns 3

And in Spark, that one function is defined not for String s but for Column s:

def select(cols: Column*)

For String s, the method is declared like this:

def select(col: String, cols: String*)

I suggest you to stick to Column s, like you do now, but with some syntax sugar:

df.select(df.columns.slice(0,3).map(col):_*))

Or if there's a need to pass column names as String s, then you can use selectExpr :

df.selectExpr(df.columns.slice(0,3):_*)