scala: method that return varargs

Question

From a scala method, I want to return a variable number a Spark columns, like this:

def getColumns() : (Column*) = {...}

This idea is then to use it with spark sql:

myDf.select(getColumns, "anotherColumns"..)

The thing is I have about 30 requests that all have the same select clause, that I want to put in common.

Any idea what to replace with the ...? I tried something like:

($"col1", "$col2")

but it doesn't compile.

Answer 1

Try this:

val df = Seq((1,2,3,4),(5,6,7,8)).toDF("a","b","c","d")

Typecast String to spark columns using map function and append addition column in an array as required.

val lstCols = List("a","b")
df.select(lstCols.map(col) ++ List(col("c"),col("d")): _*).show()   

+---+---+---+---+
|  a|  b|  c|  d|
+---+---+---+---+
|  1|  2|  3|  4|
|  5|  6|  7|  8|
+---+---+---+---+