使用带有 jdbc write 的 PySpark Dataframe 在 PostgreSQL 上编写 enun 的问题

Question

So I am moving data from a MySQL (5.7) database to a PostgreSQL (12.7) one using PySpark (Spark 3.0.1, Scala 2.12). A table from the destiny model has an column that is an Enum.

CREATE TYPE ORDER_STATUS AS ENUM (
'SHIPPED','PAID','REFUNDED','PARTIALLY_REFUNDED','PROCESSING');

When inserting:

df_orders.select(df_orders.columns).write.format('jdbc').options(**postgres_write_opts_table).mode('append').save()

I am getting the next exception

Caused by: org.postgresql.util.PSQLException: ERROR: column "status" is of type order_status but expression is of type character varying
Hint: You will need to rewrite or cast the expression.

Basically I need to cast the column status to ORDER_STATUS . I have tried to use a UserDefinedType (PySpark does no have SQLUserDefinedType ) but no really knowing what I am doing because the documentation is not very clear.

class StatusUDT(UserDefinedType):
@classmethod
def sqlType(self):
    return NullType()

@classmethod
def module(cls):
    return cls.__module__

def serialize(self, obj):
    return f"{obj.value}::order_status_type"

def deserialize(self, datum):
    return {x.value: x for x in Some}[datum]

And then I try the casting

df_orders = df_orders.withColumn("status", col("status").cast(StatusUDT()))

Then I am getting the next error then:

AnalysisException: cannot resolve 'CAST(`status` AS NULL)' due to data type mismatch: cannot cast string to null;;

Is there any way to cast this Enum?

Answer 1

So I finally was able to overcome this issue. I temporally removed the Enum so I could keep doing more tests and the I had a similar issue with a JSON type. Searching about it I found this post: How to save String as JSONB type in postgres when using AWS Glue . And I fixed it setting the property:

'stringtype':"unspecified"

as the post answer suggests.

Then I put back the Enum into the table and this property also worked. I was able to run the insertions with no further issues.