I need help concatenating 1 csv file and 1 pandas dataframe together without duplicates

Question

My code currently looks like this:

df1 = pd.DataFrame(statsTableList)
df2 = pd.read_csv('StatTracker.csv')
result = pd.concat([df1,df2]).drop_duplicates().reset_index(drop=True)

I get an error and I'm not sure why.

The goal of my program is to pull data from an API, and then write it all to a file for analyzing. df1 is the lets say the first 100 games written to the csv file as the first version. df2 is me reading back those first 100 games the second time around and comparing it to that of df1 (new data, next 100 games) to check for duplicates and delete them.

The part that is not working is the drop duplicates part. It gives me an error of unhashable list, I would assume that's because its two dataframes that are lists of dictionaries. The goal is to pull 100 games of data, and then pull the next 50, but if I pull number 100 again, to drop that one, and just add 101-150 and then add it all to my csv file. Then if I run it again, to pull 150-200, but drop 150 if its a duplicate, etc etc..

Answer 1

Based from your explanation, you can use this one liner to find unique values in df1:

df_diff = df1[~df1.apply(tuple,1)\
            .isin(df2.apply(tuple,1))]

This code checks if the rows is exists in another dataframe. To do the comparision it converts each row to tuple (apply tuple conversion along 1 (row) axis).

This solution is indeed slow because its compares each row inside df1 to all rows in df2. So it has time complexity n^2.

If you want more optimised version, try to use pandas built in compare method

df1.compare(df2)