防止熊猫将每个块的格式化标头重写为csv

Question

I have a dirty csv with an ugly header that I have formatted and stored in a list.

I want to read this csv chunk by chunk, perform some regex on the data, and then write to a new csv.

I'm using this function to do so

def format_data(data_location, formatted_header):
    df = pd.read_csv(data_location, sep=',', skiprows=1,
                     header=0, names=formatted_header, chunksize=10000)

    for chunk in df:
        chunk = chunk.replace('(?!(([^"]*"){2})*[^"]*$),', '', regex=True)
        chunk.to_csv('formatted_data.csv', mode='a', index=False)

As I understand what I am doing here:

pd.read_csv(data_location, sep=',', skiprows=1,
            header=0, names=formatted_header, chunksize=10000)

I am reading the csv from it's location, skipping the first ugly header row and replacing with my formatted_header.

My issue is that for each new chunk that is written to the new CSV, I am seeing the formatted header row repeated after every 10,000 rows. How can I prevent this from happening?

Answer 1

Since you only want to write the header once, use a boolean to see if you're on the first chunk.

For example:

write_header = True
for chunk in df:
    chunk = chunk.replace('(?!(([^"]*"){2})*[^"]*$),', '', regex=True)
    chunk.to_csv('formatted_data.csv', mode='a', index=False, header=write_header)
    write_header = False