ISO 日期字段在从 MONGO 读取 PySpark 时自动转换

Question

I am trying to read a date field from MongoDB collection to PySpark df. My date field has an ISO format when seen in mongo DB but it gets converted into a different type after reading on Spark.

 In Mongo the date looks like below

 ISODate("2012-07-14T01:00:00+01:00")

 df =  (sqlContext.read.format("com.xyz.datasource.mongodb").options(host="mongo:XXX",database="foo", collection="bar").load())
 df.show()

My date column gets converted like below:

{ "$date": 1.62345674343 }

I understand this got converted to epoch and I have an UDF which converts to human readable timestamp but on why this happens? Is there a fix which avoids or ignores my UDF (I would like to not apply UDF on columns)?

I have multiple createdAt fields which I like to change.

Answer 1

spark only supports epoch seconds and not milliseconds, sometimes data coming out of mongo has epoch milliseconds. In that case you would divide the epoch milliseconds integer by a thousand.

from pyspark.sql.types import *

df = sqlContext.read.format(
        "com.xyz.datasource.mongodb"
    ).options(
        host="mongo:XXX",
        database="foo", 
        collection="bar"
    ).load().withColumn(
        'date', 
         col('exposure.knownEmployeesExposed.latestIncidentReport.createdAt').cast('timestamp')
    )