Scala + Spark + Dataframe异常当我尝试动态转换列并分配排序顺序时

Question

I want to sort my dataframe using selected columns by casting them from stringtype to prederred type and prederred order. But even a simple casting of a column doesn't work and giving this exception. I am providing the sample code here.

    val conf = new SparkConf().setAppName("Sparkify").setMaster("local[*]")
    val sparkContext =new SparkContext(conf)
    val sqlContext = new SQLContext(sparkContext)
    var df =  sqlContext.read
      .format("com.databricks.spark.csv")
      .option("header", "true")
      .load("example-data.csv")
//    val colsToSort= List("age")
    df = df.sort( df.col("age").cast(IntegerType).desc)
    df.show()
    sparkContext.stop()

The simple csv looks like this

+-----+---+---+
| name|sex|age|
+-----+---+---+
|Alice|  f| 34|
|  Bob|  m| 63|
|Alice|  f| 14|
|  Bob|  m|  6|
+-----+---+---+

The detailed exception stack.

Exception in thread "main" org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to dataType on unresolved object, tree: unresolvedalias(cast(cast(age#2 as decimal(20,0)) as int))
    at org.apache.spark.sql.catalyst.analysis.UnresolvedAlias.dataType(unresolved.scala:295)
    at org.apache.spark.sql.catalyst.expressions.SortOrder.dataType(SortOrder.scala:49)
    at org.apache.spark.sql.catalyst.expressions.SortOrder.checkInputDataTypes(SortOrder.scala:42)
    at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:62)
    at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:57)
    at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:319)
    at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:319)
    at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:53)
    at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:318)
    at org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpressionUp$1(QueryPlan.scala:107)
    at org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$2(QueryPlan.scala:117)
    at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$2$1.apply(QueryPlan.scala:121)

What am i doing wrong? Or what is the best way to dynamically declare a sorting expression based on multiple columns, with type casting and ordering.

Can anyone help me with this Please?

Answer 1

You can't create the new column within the sort function -- you have to do it before the sort . Try something like this instead:

df.withColumn("age", $"age".cast(IntegerType)).sort($"age".desc)

If you want to use a variable for the column name, try this:

val colName = "age"
df.withColumn(colName, col(colName).cast(IntegerType)).sort(col(colName).desc)

Note that, unless you are joining two tables with the same-named column, you can generally leave off the df. in front of col(...) . You could even create the sort column before you create the df and it works fine:

val sortCol = col("age").desc
val df = ...
df.sort(sortCol)

In that way you can easily apply the same sort to multiple DataFrames

If you want to cast more than one column, you could just do:

df.withColumn(...).withColumn(...).withColumn(...)

Where each of the withColumn is a cast expression. Or you could do:

df.select($"age".cast(IntegerType), $"otherCol".cast(DoubleType), ...)