从Spark数据框中选择几列，并以列列表作为开始

Question

Assuming that I have a list of spark columns and a spark dataframe df, what is the appropriate snippet of code in order to select a subdataframe containing only the columns in the list?

Something similar to maybe:

var needed_column: List[Column]=List[Column](new Column("a"),new Column("b"))

df(needed_columns)

I wanted to get the columns names then select them using the following line of code.

Unfortunately, the column name seems to be in write mode only.

df.select(needed_columns.head.as(String),needed_columns.tail: _*)

Answer 1

Your needed_columns is of type List[Column] , hence you can simply use needed_columns: _* as the arguments for select :

val df = Seq((1, "x", 10.0), (2, "y", 20.0)).toDF("a", "b", "c")

import org.apache.spark.sql.Column

val needed_columns: List[Column] = List(new Column("a"), new Column("b"))

df.select(needed_columns: _*)
// +---+---+
// |  a|  b|
// +---+---+
// |  1|  x|
// |  2|  y|
// +---+---+

Note that select takes two types of arguments:

def select(cols: Column*): DataFrame

def select(col: String, cols: String*): DataFrame

If you have a list of column names of String type, you can use the latter select :

val needed_col_names: List[String] = List("a", "b")

df.select(needed_col_names.head, needed_col_names.tail: _*)

Or, you can map the list of String s to Column s to use the former select

df.select(needed_col_names.map(col): _*)

Answer 2

I understand that you want to select only those columns from a list(A)other than the dataframe columns. I have a below example, where I select the firstname and lastname using a separate list. check this out

scala> val df = Seq((101,"Jack", "wright" , 27, "01976", "US")).toDF("id","fname","lname","age","zip","country")
df: org.apache.spark.sql.DataFrame = [id: int, fname: string ... 4 more fields]

scala> df.columns
res20: Array[String] = Array(id, fname, lname, age, zip, country)

scala> val needed =Seq("fname","lname")
needed: Seq[String] = List(fname, lname)

scala> val needed_df = needed.map( x=> col(x) )
needed_df: Seq[org.apache.spark.sql.Column] = List(fname, lname)

scala> df.select(needed_df:_*).show(false)
+-----+------+
|fname|lname |
+-----+------+
|Jack |wright|
+-----+------+


scala>