How to convert string to date in pyspark

Question

I have a file name from which I'm extracting date: some_file_name_20201103114823.csv using substring :

substring(input_file_name(),16,8)

I extracted date part, which is now string: 20201103

How can I convert this string to date in format: MM-dd-yyyy ?

Answer 1

This is what i did, formatted string and then cast it to date - not pretty but does the work:

to_date(concat(substring(input_file_name(),16,4),lit("-"),substring(input_file_name(),20,2),lit("-"),substring(input_file_name(),22,2)),"yyyy-MM-dd")

Reference: https://sparkbyexamples.com/pyspark/pyspark-substring-from-a-column/#:~:text=In%20PySpark%2C%20the%20substring(),using%20substring()%20from%20pyspark .