[英]cast the string column to date for the particular format
How to cast the string column to date column and maintain the same format in spark data frame?如何将字符串列转换为日期列并在火花数据框中保持相同的格式?
I want to cast the string column to date by specifying the format, but the after cast date always comes in the default format which is yyyy-MM-dd.我想通过指定格式将字符串列转换为日期,但转换后的日期始终采用默认格式,即 yyyy-MM-dd。
But I want the Date type with the format which is in the string value(I want the data type as Date only not as String)但我想要日期类型的格式为字符串值(我想要数据类型为日期而不是字符串)
For example:例如:
val spark = SparkSession.builder().master("local").appName("appName").getOrCreate()
import spark.implicits._
//here the format is MMddyyyy(For Col2 which is of String type here)
val df = List(("1","01132019"),("2","01142019")).toDF("Col1","Col2")
import org.apache.spark.sql.functions._
//Here i need the Col3 in Date type and with the format MMddyyyy But it is converting into yyyy-MM-dd
val df1 = df.withColumn("Col3",to_date($"Col2","MMddyyyy"))
//I tried this but this will give me Col3 in String data type which i need in Date
val df1 = df.withColumn("Col3",date_format(to_date($"Col2","MMddyyyy"),"MMddyyyy"))
That's not possible, Spark accepts date type yyyy-MM-dd
format only.这是不可能的,Spark 只接受日期类型
yyyy-MM-dd
格式。
If you need to have MMddyyyy
this format date field then store as String
type(if we cast to date type results null) , While processing change the format and cast as date
type.如果您需要
MMddyyyy
这个格式日期字段然后存储为String
类型(如果我们转换为日期类型结果为 null),在处理时更改格式并转换为date
类型。
Ex:前任:
df.withColumn("Col3",$"col2".cast("date")) //casting col2 as date datatype Results null
.withColumn("col4",to_date($"col2","MMddyyyy").cast("date")) //changing format and casting as date type
.show(false)
Result:结果:
+----+--------+----+----------+
|Col1| Col2|Col3| col4|
+----+--------+----+----------+
| 1|01132019|null|2019-01-13|
| 2|01142019|null|2019-01-14|
+----+--------+----+----------+
Schema:
df.withColumn("Col3",$"col2".cast("date"))
.withColumn("col4",to_date($"col2","MMddyyyy").cast("date"))
.printSchema
Result:结果:
root
|-- Col1: string (nullable = true)
|-- Col2: string (nullable = true)
|-- Col3: date (nullable = true)
|-- col4: date (nullable = true)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.