Appending a row of data to line 2 in a large CSV File

Question

I'm sure this is a really easy question but I can't seem to find any information on it.

I have a very large CSV file which I need to insert a row directly after the header which helps with another code that reads the csv and joins it to a parcel shapefile.

I have the code to append the row of data that I want, but it will only go to the last line. I cannot figure out how to get the code to insert my row immediately after the header row. Here is my code:

import os
import csv

insert_row = '"AAAAAAAAAAAAAAAAAAA","**********","**********","**********","**********","**********","**********","**","**********","**********","****","**********",999999,9999,00'

os.chdir(r"D:\PROPERTY\PINELLAS\Data_20201001_t")


with open("owner_mail.csv", 'r') as csv_file, open("owner_mail.csv", 'a', newline = "") as new_file:
    csv_reader = csv.reader(csv_file)
    csv_writer = csv.writer(new_file)

    csv_writer.writerow(insert_row)

So that's it. I just need the insert_row line of data to be in row position number 2 instead of at the end of the file. Thank you.

Answer 1

You can't just insert a row in the middle of a file unless replacing data of exactly the same length. You have to read the entire file, edit it, and re-write it.

Something like this should work:

import csv

# This must be an iterable not a string
insert_row = "AAAAAAAAAAAAAAAAAAA","**********","**********","**********","**********","**********","**********","**","**********","**********","****","**********",999999,9999,00

with open("owner_mail.csv", 'r') as csv_file, open("owner_mail_updated.csv", 'w', newline = "") as new_file:
    csv_reader = csv.reader(csv_file)
    csv_writer = csv.writer(new_file)

    header = next(csv_reader)
    csv_writer.writerow(header)

    csv_writer.writerow(insert_row)

    for line in csv_reader:
        csv_writer.writerow(line)

If the CSV file is not too large to fit entirely in memory than you can read all the lines at once, edit them, and write them back out to the same file. It's riskier if there is a problem. Safer to write to a new file, then delete original and rename if no errors:

import csv

# This must be an iterable not a string
insert_row = "AAAAAAAAAAAAAAAAAAA","**********","**********","**********","**********","**********","**********","**","**********","**********","****","**********",999999,9999,00

with open("owner_mail.csv", 'r') as csv_file:
    rows = list(csv.reader(csv_file))

rows.insert(1,insert_row)  # insert after header row

with open("owner_mail.csv", 'w') as csv_file:
    w = csv.writer(csv_file)
    w.writerows(rows)

Answer 2

Please try this:

import os
import csv

insert_row = '"AAAAAAAAAAAAAAAAAA","**********","**********","**********","**********","**********","**********","**","**********","**********","****","**********",999999,9999,00'

with open("owner_mail.csv", 'r') as csv_file, open("owner_mail.csv", 'w') as new_file:
  csv_reader = csv.reader(csv_file)
  reader = list(csv_reader)
  reader.insert(1,insert_row)
  csv_writer = csv.writer(new_file)
  csv_writer.writerows(reader)