如何在不添加重复项的情况下使用 pandas 更新 CSV 文件

Question

I'm trying to get some data off the web and it's taking a while. In case anything happens I've been periodically saving the data in a csv file.

However, it just appends a new copy of the dataframe to the CSV file. This means that there's loads of duplicates in the file.

df.to_csv('data.csv', mode='a', header=False)

is the command i'm using to save my progress.

Thanks for reading.

Answer 1

IIUC, you have a single dataframe to which you append to over time and which you want to back up periodically.

There are multiple approaches you could try:

1. If writing the file is fast, instead of appending, just write the complete dataframe every time (writing the header potentially could be useful in this case):

df.to_csv('data.csv', header=False)  # or header=True

2. Keep track of which lines you have already written and only append new lines:

# (i) First time write the complete dataframe
df.to_csv('data.csv', header=False)  # or header=True

# (ii) store the length of the dataframe at that point
lines_written = len(df.index)

# More data is being added to the dataframe from the web

# (iii) append new lines to CSV file
df.iloc[lines_written:].to_csv('data.csv', mode='a', header=False)

# (iv) update the line counter
lines_written = len(df.index)

# repeat steps (iii) and (iv)