雪花 - 每次合并时会自动创建哪种类型的舞台表？

Question

I want to write a Spark DataFrame into a Snowflake table. I'm using a Snowflake connector for Spark and will pass a "query" option with the MERGE into statement like this:

merge_query =  "merge into target_table using stage_table 
        on target_table.id = stage_table.id
        when matched then 
        update set target_table.description = stage_table.description"

df.write
    .format(SNOWFLAKE_SOURCE_NAME)
    .options(sfOptions)
    .option("query", "merge_query")
    .mode(SaveMode.Overwrite)
    .save()

I don't have any external source table and want to find a way to overwrite stage data every time and then merge it to the target table. But I don't quite understand which type of stage should I use in this case. The documentation on Snowflake table stages doesn't specify what type is suitable when using MERGE INTO. I need something like a temporary stage table created automatically by Snowflake.

For the above example, will the stage_table be created automatically? And how should it be named if I don't want to create any stage table explicitly?

Answer 1

There is now way for automatically creating any table. What you can do is in your program, create a temporary table, then load the data into that table and do the merge. Temporary table are only available to the session and will be automatically deleted as soon as the session ends(which means as soon as your job ends). Session tables are not visible to any other sessions. you can create temporary table as below

CREATE TEMPORARY TABLE <STG_TABLE_NAME>(FILED 1 VARCHAR, ...)

Hope this work for you.