迭代 dataframe df_a 中的行并根据 Pyspark 中的 df_a 值更新 dataframe df_b

Question

I have a dataframe df_b which has to be updated based on dataframe df_a values

df_a

+-----+-----+------------+---------+
| id_1| id_2| header_oper| head_seq|
+-----+-----+------------+---------+
|  boy|    3|      insert|        1|
|  bat|    4|      delete|        3|
|  cat|    2|      insert|        1|
|  bat|    4|      update|        2|
|  bat|    5|   beforeimg|        1|
+-----+-----+------------+---------+

df_b (before)

+-----+-----+
| id_1| id_2|
+-----+-----+
|  boy|    4|
|  bat|    5|
|  cat|    1|
+-----+-----+

The method I came up with:

1. Sort df_a on 'head_seq'.
2. Iterate df_a
3. if 'header_oper'.isin('insert','update') then append that row to df_b
4. if 'header_oper'.isin('delete','beforeimg') then subtract that row from df_b

Expected df_b (after):

+-----+-----+
| id_1| id_2|
+-----+-----+
|  boy|    4|
|  boy|    3|
|  cat|    2|
|  cat|    1|
+-----+-----+

Need help on how to iterate df_a and perform operations on df_b based on df_a values.

Answer 1

ds= spark.createDataFrame([('boy',4),('bat',5),('cat',1)],['id_1','id_2'])
df_op=spark.createDataFrame([('boy',3,'insert',1),('bat',4,'delete',3),('cat',2,'insert',1),('bat',4,'update',2),('bat',5,'beforeimg',1)], ['id_1','id_2','eff_op','seq'])

effective_op=df_op.groupBy('id_1').agg(max('seq').alias('seq')).join(df_op,['id_1','seq'])

ds_insert=ds.union(effective_op.select('id_1','id_2').filter("eff_op in ('insert')").orderBy(asc('id_1')))

ds_delete=ds_insert.join(effective_op.filter("eff_op in ('delete')").select("*"),['id_1'],'left').select(ds_insert.id_1, ds_insert.id_2).filter("eff_op is null")

display(ds_delete)

Answer 2

Ok I figured this out. As there's beforeimg for every update, the order of the operations didn't matter.

I just had to add all the Inserts and Updates and then delete the Deletes and BeforeImgs

Partitioning the operations and deselecting the header columns

ins=df_a.where(df_a['header_oper']=='insert')
ins=ins.select(id_1,id_2)

upd=df_a.where(df_a['header_oper']=='update')
upd=upd.select(id_1,id_2)

dele=df_a.where(df_a['header_oper']=='delete')
dele=dele.select(id_1,id_2)

bimg=df_a.where(df_a['header_oper']=='delete')
bimg=bimg.select(id_1,id_2)

Appending the Inserts and Updates to df_b

df_b=df_b.union(ins)
df_b=df_b.union(upd)

Removing the Deletes and BeforeImgs from df_b

df_b=df_b.subtract(dele)
df_b=df_b.subtract(bimg)

Answer 3

I have a dataframe df_b which has to be updated based on dataframe df_a values

df_a

+-----+-----+------------+---------+
| id_1| id_2| header_oper| head_seq|
+-----+-----+------------+---------+
|  boy|    3|      insert|        1|
|  bat|    4|      delete|        3|
|  cat|    2|      insert|        1|
|  bat|    4|      update|        2|
|  bat|    5|   beforeimg|        1|
+-----+-----+------------+---------+

df_b (before)

+-----+-----+
| id_1| id_2|
+-----+-----+
|  boy|    4|
|  bat|    5|
|  cat|    1|
+-----+-----+

The method I came up with:

1. Sort df_a on 'head_seq'.
2. Iterate df_a
3. if 'header_oper'.isin('insert','update') then append that row to df_b
4. if 'header_oper'.isin('delete','beforeimg') then subtract that row from df_b

Expected df_b (after):

+-----+-----+
| id_1| id_2|
+-----+-----+
|  boy|    4|
|  boy|    3|
|  cat|    2|
|  cat|    1|
+-----+-----+

Need help on how to iterate df_a and perform operations on df_b based on df_a values.