[英]Delta Lake MERGE / UPDATE rewriting data even when condition is not met
[英]Snowflake merge into is adding data even when condition is met and even if fields from target and source tables are already exists
我有以下 Snowflake 语句,它将检查来自阶段文件的散列字段是否已存在于目标表中,然后在不匹配时执行插入:
MERGE INTO LINK_DIMENSION_FIELD AS D
USING (SELECT md5(concat(T.$2, T.$4)) DIM_FIELD, T.$2 DIMENSION_NAME, T.$4 FIELD_NAME
FROM @ingest_stage_temp/dimension_field.csv (FILE_FORMAT=>"GENERIC_CSV_FORMAT") T) ST
ON md5(concat(D.DIMENSION_NAME_HASH_KEY, D.FIELD_NAME_HASH_KEY)) = ST.DIM_FIELD
WHEN NOT MATCHED THEN
INSERT (DIMENSION_NAME_FIELD_NAME_HASH_KEY, LOAD_DT, RECORD_SRC, DIMENSION_NAME_HASH_KEY, FIELD_NAME_HASH_KEY)
VALUES(MD5(CONCAT(ST.DIMENSION_NAME, ST.FIELD_NAME)), current_timestamp(), 'TEST', md5(ST.DIMENSION_NAME), md5(ST.FIELD_NAME));
问题是即使在md5(concat(D.DIMENSION_NAME_HASH_KEY, D.FIELD_NAME_HASH_KEY)) = ST.DIM_FIELD
时合并也始终有效。
如果你能看到,这是运行 select 查询后的暂存文件:
SELECT md5(concat(T.$2, T.$4)) DIM_FIELD, T.$2 DIMENSION_NAME, T.$4 FIELD_NAME
FROM @ingest_stage_temp/dimension_field.csv (FILE_FORMAT=>"GENERIC_CSV_FORMAT") T
结果是:
DIM_FIELD DIMENSION_NAME FIELD_NAME
87d7dae13cf0326fd03a348ca6c518b5 cg_child_6mo_receiv_ind_iycf cg_child_6mo_receiv_ind_iycf/nbr_1st_cons_6mc_iycfc
2b75306f968f11b45f066efb9871babb cg_child_6mo_receiv_ind_iycf cg_child_6mo_receiv_ind_iycf/nbr_followup_2nd_time_6mc_iycfc
53273e7133d7a0b513af8c9bcc934437 preg_women_rec_ind_counselling preg_women_rec_ind_counselling/nbr_1st_cons_pregw_iycfc
当我对现有数据运行 select 查询时:
select * from LINK_DIMENSION_FIELD;
您可以清楚地看到DIM_FIELD
中的所有值都已在此表中,因此不应执行插入查询:
在您的 ON-Clause 中,您正在比较:
md5(concat(D.DIMENSION_NAME_HASH_KEY, D.FIELD_NAME_HASH_KEY)) = ST.DIM_FIELD
我认为将 ST.DIM_FIELD 与 DIMENSION_NAME_FIELD_NAME_HASH_KEY(目标表中最终计算的列)进行比较可以解决问题。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.