spark sql select 双列但说 FloatWritable 不能转换为 DoubleWritable

Question

My code is just to select a double column, but I get floatWritable cannot be cast to DoubleWritable error. Is that because spark has different procedure when reading hive double column?

val testBase = spark.sql(
      s"""select
         | cast(clk_rate_7_day as double)
         |from %s
         |where ds between date_sub('%s', 0) and '%s'
         |union all
         |select
         | cast(clk_rate_7_day as double)
         |from %s
         |where ds between date_sub('%s', 0) and '%s'
         |""".stripMargin.format(trainDataInput, jobDate, jobDate, trainDataYuncunInput, jobDate, jobDate))
testBase.show()

but I get this error,

testBase: org.apache.spark.sql.DataFrame = [clk_rate_7_day: double]
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID 11, hadoop4445.jd.163.org, executor 5): java.lang.ClassCastException: org.apache.hadoop.io.FloatWritable cannot be cast to org.apache.hadoop.io.DoubleWritable
    at org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableDoubleObjectInspector.get(WritableDoubleObjectInspector.java:36)
    at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$14$$anonfun$apply$8.apply(TableReader.scala:423)
    at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$14$$anonfun$apply$8.apply(TableReader.scala:423)
    at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$2.apply(TableReader.scala:460)
    at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$2.apply(TableReader.scala:451)
    at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
    at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:255)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:247)
 

Answer 1

Problem solved!

It turns out that I used spark to write this column as a FloatType, but its hive configuration is a Double, and then ended up with this cast error.

Btw, why Float cannot be cast into Double in spark?