[英]Spark rdd correct date format in scala?
這是將RDD轉換為Dataframe時要使用的日期值。
Sun Jul 31 10:21:53 PDT 2016
此架構“ DataTypes.DateType”引發錯誤。
java.util.Date is not a valid external type for schema of date
因此,我想提前准備RDD,以使上述架構可以正常工作。 如何更正日期格式以便轉換為數據框?
//Schema for data frame
val schema =
StructType(
StructField("lotStartDate", DateType, false) ::
StructField("pm", StringType, false) ::
StructField("wc", LongType, false) ::
StructField("ri", StringType, false) :: Nil)
// rowrdd : [Sun Jul 31 10:21:53 PDT 2016,"PM",11,"ABC"]
val df = spark.createDataFrame(rddRow,schema)
Spark的DateType
可以從java.sql.Date
進行編碼,因此您應該將輸入的RDD轉換為使用該類型,例如:
val inputRdd: RDD[(Int, java.util.Date)] = ??? // however it's created
// convert java.util.Date to java.sql.Date:
val fixedRdd = inputRdd.map {
case (id, date) => (id, new java.sql.Date(date.getTime))
}
// now you can convert to DataFrame given your schema:
val schema = StructType(
StructField("id", IntegerType) ::
StructField("date", DateType) ::
Nil
)
val df = spark.createDataFrame(
fixedRdd.map(record => Row.fromSeq(record.productIterator.toSeq)),
schema
)
// or, even easier - let Spark figure out the schema:
val df2 = fixedRdd.toDF("id", "date")
// both will evaluate to the same schema, in this case
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.