[英]Validate Date strict to format - more than 4 character for year - pySpark
I am trying to validate date received in file against configured date format(using to_timestamp /to_date).我正在尝试根据配置的日期格式(使用 to_timestamp /to_date)验证文件中收到的日期。
schema = StructType([ \
StructField("date",StringType(),True), \
StructField("active", StringType(), True)
])
input_data = [
("27/10/00",'Y'),
("27/10/01",'Y'),
("27/10/1",'Y'),
("27/10/16",'Y'),
("27/10",'Y'),
("27/10/99",'Y'),
("27/10/100",'Y'),
("27/10/992",'Y'),
("27/10/2000",'Y'),
("27/10/9999",'Y'),
("27/10/20015",'Y'), ]
date_format = "dd/MM/yyyy"
temp_df = spark.createDataFrame(data=input_data,schema=schema)
df = temp_df.select('*',
f.when(f.date_format(f.to_timestamp(f.col('date'), date_format), date_format).isNotNull(), True).otherwise(False).alias('Date_validation'),
f.date_format(f.to_timestamp(f.col('date'), date_format), date_format).alias('converted_date'),
)
df.show(truncate=False)
my expectation is the code should return False since YYYY is a four character date but the code returns True and prints the converted year as valid one.我的期望是代码应该返回 False,因为 YYYY 是一个四字符日期,但代码返回 True 并将转换后的年份打印为有效年份。
is there anyway I could to a strict validation on date?无论如何我可以对日期进行严格的验证吗?
note: formats are configured by user which will be dynamic.注意:格式由用户配置,这将是动态的。
tried to_timestamp and to_date both are having same output尝试 to_timestamp 和 to_date 都具有相同的 output
This is a bug since 2.4 and had been fixed from 3.0 forward.这是自 2.4 以来的错误,并已从 3.0 向前修复。 More information here .
更多信息在这里。 In short, there is a configuration called
spark.sql.legacy.timeParserPolicy
with 3 possible values:简而言之,有一个名为
spark.sql.legacy.timeParserPolicy
的配置,有 3 个可能的值:
LEGACY
: Perform like 2.4 where you see the bug LEGACY
:在你看到错误的地方像 2.4 一样执行EXCEPTION
: Raise exceptions when has an invalid format EXCEPTION
:当格式无效时引发异常CORRECTED
: Fix and return your expected result CORRECTED
:修复并返回您的预期结果This is the result with CORRECTED
:这是
CORRECTED
的结果:
+-----------+------+---------------+--------------+
|date |active|Date_validation|converted_date|
+-----------+------+---------------+--------------+
|27/10/00 |Y |false |null |
|27/10/01 |Y |false |null |
|27/10/1 |Y |false |null |
|27/10/16 |Y |false |null |
|27/10 |Y |false |null |
|27/10/99 |Y |false |null |
|27/10/100 |Y |false |null |
|27/10/992 |Y |false |null |
|27/10/2000 |Y |true |27/10/2000 |
|27/10/9999 |Y |true |27/10/9999 |
|27/10/20015|Y |false |null |
+-----------+------+---------------+--------------+
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.