如何编写两个Spark DataFrames以原子方式进行Redshift？

Question

I am using Databricks spark-redshift to write DataFrames to Redshift. I have two DataFrames that get appended to two separate tables, but I need this to happen atomically, ie if the second DataFrame fails to write to its table, I'll need the first one to be undone as well. Is there any way to do that?

Answer 1

The solution is to have a staging table for each target table. To write Spark results to the database:

1. Clean staging tables ( DELETE FROM staging_table )
2. Write the data frames to staging tables using spark-redshift (not atomic)
3. Atomically copy from staging tables to target tables in a transaction (for Python use redshift-sqlalchemy package).

Only one instance of the Spark application can be running at a time, ie you can't have two jobs writing to staging tables at the same time, otherwise the resulting data won't be valid.