Compare and remove duplicates from both dataframe

Question

I have 2 dataframes that needs to be compared and remove duplicates (if any)

Daily = DataFrame({'col1':[1,2,3], 'col2':[2,3,4]})
Accumulated = DataFrame({'col1':[4,2,5], 'col2':[6,3,5]})

Out[4]:
   col1  col2
0     1     2
1     2     3
2     3     4
   col1  col2
0     4     6
1     2     3
2     5     5
3     6     6

What I am trying to achieve is to remove duplicates if there are any, from both DF and get the count of remaining entries from daily DF

Expected output:

   col1  col2
0     1     2
2     3     4
   col1  col2
0     4     6
2     5     5
3     6     6

Count = 2

How can i do it? Both or either DFs can be empty, and daily can have more entries than Montlhy and vice versa

Answer 1

Why don't just concat both into one df and drop the duplicates completely?

s = (pd.concat([Daily.assign(source="Daily"),
               Accumulated.assign(source="Accumlated")])
       .drop_duplicates(["col1","col2"], keep=False))

print (s[s["source"].eq("Daily")])

   col1  col2 source
0     1     2  Daily
2     3     4  Daily

print (s[s["source"].eq("Accumlated")])

   col1  col2      source
0     4     6  Accumlated
2     5     5  Accumlated
3     6     6  Accumlated

Answer 2

You can try the below code

 ## For 1st Dataframe   
for i in range(len(df1)):
        for j in range(len(df2)):
            if df1.iloc[i].to_list()==df2.iloc[j].to_list():
                df1=df1.drop(index=i)

Similarly you can do for the second datframe

Answer 3

I would do it following way:

import pandas as pd
daily = pd.DataFrame({'col1':[1,2,3], 'col2':[2,3,4]})
accumulated = pd.DataFrame({'col1':[4,2,5], 'col2':[6,3,5]})
daily['isdaily'] = True
accumulated['isdaily'] = False
together = pd.concat([daily, accumulated])
without_dupes = together.drop_duplicates(['col1','col2'],keep=False)
daily_count = sum(without_dupes['isdaily'])

I added isdaily column to dataframes as True s and False s so they could be easily sum med at end.

Answer 4

If I understood correctly, you need to have both tables separated.

You can concatenate them, keeping the table from where they come from and then recreate them:

Daily = pd.DataFrame({'col1':[1,2,3], 'col2':[2,3,4]})
Daily["Table"] = "Daily"
Accumulated = pd.DataFrame({'col1':[4,2,5], 'col2':[6,3,5]})
Accumulated["Table"] = "Accum"

df = pd.concat([Daily, Accumulated]).reset_index()

not_dup = df[["col1", "col2"]].drop_duplicates()
not_dup = df.loc[not_dup.index,:]

Daily = not_dup[not_dup["Table"] == "Daily"][["col1","col2"]]
Accumulated = not_dup[not_dup["Table"] == "Accum"][["col1","col2"]]

print(Daily)
print(Accumulated)

Answer 5

following those steps:

1. Concatenate the 2 data-frames
2. Drop all duplication
3. For each data-frame find the intersection with the concat data-frame
4. Find count with len

Daily = pd.DataFrame({'col1':[1,2,3], 'col2':[2,3,4]})
Accumulated = pd.DataFrame({'col1':[4,2,5], 'col2':[6,3,5]})

df = pd.concat([Daily, Accumulated]) # step 1
df = df.drop_duplicates(keep=False) # step 2

Daily = pd.merge(df, Daily, how='inner', on=['col1','col2']) #step 3
Accumulated = pd.merge(df, Accumulated, how='inner', on=['col1','col2']) #step 3

count = len(Daily) #step 4