火花 Scala 数据帧 select

Question

I am trying to convert a pyspark code to spark Scala and i am facing the below error:

pyspark code

import pyspark.sql.functions as fn
valid_data = bcd_df.filter(fn.lower(bdb_df.table_name)==tbl_nme)
.select("valid_data").rdd
.map(lambda x: x[0])
.collect()[0]

From bcd_df dataframe I am getting a column with table_name and matching the value of table_name with the argument tbl_name that i am passing and then selecting the valid_data column data.

Here is the code in spark scala.

val valid_data =bcd_df..filter(col(table_name)===tbl_nme).select(col("valid_data")).rdd.map(x=> x(0)).collect()(0)

Error as below:

Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve '`abcd`' given input 
columns:
Not sure why it is taking abcd as column.

Any help is appreciated.

Version scala2.11.8 spark2.3

Answer 1

Enclose table_name column with quotes(") in col

val valid_data =bcd_df.filter(col("table_name")===tbl_nme).select(col("valid_data")).rdd.map(x=> x(0)).collect()(0)