如何将pyspark数据框转换为json列表，没有数字的双引号和额外的//日期列

Question

I am trying to to convert pyspark data frame to json list which i need to pass the json values to api, when am trying to convert all json values populating with "" like valuue =12 but when converting its coming like value ="12" and date with 3\/7\/2022 how can i avoid these extra \ and double quotes.

am using below code:

loan = loan_detail_df.toPandas()
results = loan.to_json(orient='records')

My output is :

[{"Lead_Number":"","Number":"123456","user1_DOB":"3\/20\/1943""ExpectedRate":"0"}]

My desired output :

{"Lead_Number":'',"Number":123456,"user1_DOB":"3/20/1943""ExpectedRate":0}

Answer 1

If you use the following syntax:

df.write.json(<storage path>)

The output will look like this if the date column is in DateType() format:

{"number":1,"date":"2021-03-22"}

If the date column is in StringType() format with / as the separator between date components, the output will be:

{"number":1,"date":"2022/03/12"}

To get the json string values to write out to api
you can use the pyspark.sql.functions.to_json() transformation.

import pyspark.sql.functions as f
df = spark.createDataFrame([
  (1, '2022/03/12')  
], ['number', 'date'])

df = (
    df
    .withColumn('json_value', f.to_json(f.struct(f.col('number'), f.col('date'))))
)

print(df.collect()[0]['json_value'])

And the output is:

{"number":1,"date":"2022/03/12"}