PySpark 读取具有相同架构但在列中具有多种类型的多拼花地板

Question

My problem is pretty simple but I cannot find a great solution. I have a lot of order parquet. I read all of them with this :

df = spark.read.option(
    'mergeSchema',
    True).parquet(*list_order).select(
        'at',
        'order_id',
        'items')

This work great if the schemas are the same. But in my new data one of the column "quantity" we change the type String to Float. This mean that this read code does not work anymore. It create an error like that :

Caused by: org.apache.spark.SparkException: Failed to merge fields 'quantity' and 'quantity'. Failed to merge incompatible data types string and double

Do you know how I can merge this multi type column ? I'll pref to not regenerate my 4 year history of parquet (will take to much time on prod)

Thank You all.

Answer 1

You can try by providing the schema manually using the .schema(your_schema) property

old_df = spark.read.parquet('old_files')
old_schema = old_df.schema

# Make changes in old_schema and declare it as new_schema

df = spark.read.schema(new_schema).parquet(your_files)

The idea is that you are reading your data as SCHEMA_ON_READ instead of the conventional SCHEMA_ON_WRITE approach in Databases. So if you tell spark to read data with a given schema, it will stick to it.

Or you can always read your data as 2 data frames. Cast the columns and union them.