将 PySpark Dataframe 写入雪花表

Question

I creating jobs using AWS Glue to write spark data frames to a snowflake table. The results are inconsistent. Basically, if I clone an existing successful glue job, then change the inputs so the job will write to a new table, it will succeed and I will have a new table in snowflake. However, if I try to run the same job again (because we are in development) it will fail with this message:

Error occurred while loading files to Snowflake: net.snowflake.client.jdbc.SnowflakeSQLException: SQL compilation error: syntax error line 1 at position 44 unexpected ')'.

This is the line of code that is causing the issue:

spark._jvm.net.snowflake.spark.snowflake.SnowflakeConnectorUtils.enablePushdownSession(spark._jvm.org.apache.spark.sql.SparkSession.builder().getOrCreate())
sfOptions = {
"sfURL" : args['URL'],
"sfUser" : args['USERNAME'],
"sfPassword" : args['PASSWORD'],
"sfDatabase" : args['DB'],
"sfSchema" : args['SCHEMA'],
"sfWarehouse" : args['WAREHOUSE'],
"truncate_table" : "off"
}

df = select_columns.toDF()
df = df.select([col(c).cast("string") for c in df.columns])

df.write.format(SNOWFLAKE_SOURCE_NAME).options(**sfOptions).option("dbtable", snowflake_table).mode("overwrite").save()

snowflake_table is a variable.

As you can see, I am trying to write in "overwrite" mode. Which should just drop the existing table and replace it with the spark data frame. There is some weird config thing going on between GLUE, spark, and snowflake but it doesn't make any sense, because like I said, I can get this to run if I start fresh from a new glue job, it is when I run it again that this job fails.

Answer 1

It appears that you have to either reset the job bookmark if it is enabled. Or you should disable the bookmark altogether to run the job under these conditions.

If someone smarter could explain why you have to do this, that would be great.