我们如何使用 jdbc 驱动程序 Z572D4E421E5E06B9BC111D82EZ 服务器从 mysql 服务器读取 spark scala 中的无效日期列

Question

I am getting error while reading this column from mysql server

id	date
1	0000-00-00
2	0000-00-01

in the above data set we can handle 0000-00-00 by using mysql server Additional parameter zeroDateTimeBehavior=convertToNull

but i don't know how to handle this type of date 0000-00-01

help me error message i got

Exception in User Class: org.apache.spark.SparkException : Job aborted due to stage failure: Task 0 in stage 6.0 failed 4 times, most recent failure: Lost task 0.3 in stage 6.0 (TID 11) (10.100.4.111 executor 1): java.sql.SQLException: YEAR

i am using this

val a = "jdbc:mysql://<host_name>:3306/<database_name>?zeroDateTimeBehavior=convertToNull"

val mysqlServerDF = sparkSession.read.format("jdbc")
                .option("url", a)
                .option("query", sql)
                .option("user",jdbcUserName)
                .option("password", jdbcPassword)
                .load()

sql is a sql query example "select * from table"

Answer 1

If fixing such dates in your database is not an option, I think your best bet would be to handle it directly in your sql query. Eg we can compare for valid date range, which is '1000-01-01' to '9999-12-31' according to docs :

val sql = """
  select 
    id, 
    case 
      when 
        not cast(date as char(10)) 
        between '1000-01-02' and '9999-12-30' 
      then 
        null 
      else 
        date 
    end 
  from table1"""

val mysqlServerDF = sparkSession.read.format("jdbc")
                .option("url", a)
                .option("query", sql)
                .option("user",jdbcUserName)
                .option("password", jdbcPassword)
                .load()