How to convert "/Date(epoch time)/" string in PySpark

Question

I have a json file where all the dates in the json are in /Date(1602949450000)/

the json is also nested. is there a generic way to parse all of /Date()/ into the timestamp?

I tried regexp_replace but I cannot convert the capture group into a timestamp.

regexp_replace("value", "\\/Date\\((\\-?\\d*?)([\\+\\-]\\d*)?\\)\\/","$1")

Answer 1

Your regex is correct, but the issue seems to be "$1" . When I ran the code with a 1 instead worked for me:

dates_df = dates_df.withColumn(
    'date_extracted',
    F.regexp_extract("date", "\\/Date\\((\\-?\\d*?)([\\+\\-]\\d*)?\\)\\/", 1)
)

A more simple regex which does the same thing would be:

dates_df = dates_df.withColumn(
    'date_extracted',
    F.regexp_extract("date", "^.+(\d{13}).+$", 1)
)