python csv header ignore while keep appending data to csv file

Question

I am trying to add some random data into text file and I am successful in that but I am facing problem with the header line. I want to add header line once and then every time I run my script , it should add just data into file and ignore the header line if exists. I tried something like this but I fail. I try to look at this example code in SO python csv, writing headers only once , but couldn't implement properly. If somebody help me to correct my code. I will be thankful.

import random
import csv
import os.path
from time import gmtime, strftime

filename = '/home/robdata/collection1.dat'

file_exists = os.path.isfile(filename)

v = random.randint(0, 100)

with open(filename, "a") as csvfile:
    headers = ['DATE', 'value']
    writer = csv.DictWriter(csvfile, delimiter=',', lineterminator='\n',fieldnames=headers)
    if not file_exists:
        writer.writeheader()  # file doesn't exist yet, write a header

    writer.writerow({'DATE': strftime("%Y-%m-%d %H:%M:%S", gmtime()), 'value': v})

it insert data but without adding any header line. I want to include headers on the first run of script, and next time when I run script several times , it should only add data and not header line. thank a lot for any idea or help.

Answer 1

A slightly simpler alternative to Mr Evans approach would be to use the following test in place of the test for existence:

fileEmpty = os.stat('collection1.dat').st_size == 0

This obviates the need to do a seek, etc.

EDIT: Complete code:

import random
import csv
import os.path
from time import gmtime, strftime

filename = '/home/robdata/collection1.dat'

fileEmpty = os.stat(filename).st_size == 0

v = random.randint(0, 100)

with open(filename, "a") as csvfile:
    headers = ['DATE', 'value']
    writer = csv.DictWriter(csvfile, delimiter=',', lineterminator='\n',fieldnames=headers)
    if fileEmpty:
        writer.writeheader()  # file doesn't exist yet, write a header

    writer.writerow({'DATE': strftime("%Y-%m-%d %H:%M:%S", gmtime()), 'value': v})

Answer 2

Rather than testing if the file exists, you could instead check to see if has zero length by seeking to the end and asking for the file location (which is of course still needed to append).

This approach will then also write the header for the case when the file is empty but still exists. It also avoids needing to import os to make a separate os.stat() to determine the file size:

import random
import csv
from time import gmtime, strftime
    
headers = ['DATE', 'value']
v = random.randint(0, 100)
    
with open('collection1.dat', 'a', newline='') as f_output:
    csv_output = csv.DictWriter(f_output, fieldnames=headers)
    f_output.seek(0, 2)
    
    if f_output.tell() == 0:
        csv_output.writeheader()

    csv_output.writerow({'DATE': strftime("%Y-%m-%d %H:%M:%S", gmtime()), 'value': v})        

writeheader() is then used to write the header only when the file is empty.

---

Note: If Python 2.x is still in use, you should always open the file in binary mode eg ab as follows:

with open('collection1.dat', 'ab') as f_output:

Answer 3

From the documentation of Python CSV module, writeheader can be used only with DictWriter. [ https://docs.python.org/2/library/csv.html#csv.DictWriter.writeheader][Documentation of writeheader]

Hence, the code should not be using writeheader . You can try this.

if not file_exists:
    writer.writerow(headers)
else:
    writer.writerow([strftime("%Y-%m-%d %H:%M:%S", gmtime()), v, w, z])

Answer 4

Looks like you edited your question while we were posting answer. Please add any changes as edit or mention what you have changed. Anyway, while dict fields are different, concept is same.

On top of dictwriter , you need to pass row data as values to the dict keys for each column.

I added a loop for 5 writes delayed by 5 sec as sample.
See below.

Working Code:

import random
import csv
import os.path
from time import gmtime, strftime, sleep

filename = 'collection1.csv'

file_exists = os.path.isfile(filename)

v = random.randint(0, 100)
w = random.randint(0, 100)
z = random.randint(0, 100)
with open(filename, "a") as csvfile:
    for i in range(5):        
        headers = ['timestamp','val1', 'val2', 'val3']
        writer = csv.DictWriter(csvfile, delimiter=',', lineterminator='\n',fieldnames=headers)
        if not file_exists:
            writer.writeheader()
        else:
            mytime = strftime("%Y-%m-%d %H:%M:%S", gmtime())
            writer.writerow({'timestamp':mytime, 'val1': v,'val2': w, 'val3':z})

        sleep(5)

collection1.csv

timestamp,val1,val2,val3
2017-03-23 14:07:20,83,60,11
2017-03-23 14:07:25,83,60,11
2017-03-23 14:07:30,83,60,11
2017-03-23 14:07:47,55,44,88
2017-03-23 14:07:52,55,44,88
2017-03-23 14:07:57,55,44,88
2017-03-23 14:08:02,55,44,88
2017-03-23 14:08:07,55,44,88

Answer 5

In case that you are appending pandas DataFrame to a csv file, and want to keep the header only at the first time you write out, here is what worked for me, much simpler than solutions before:

start = 100
with open('path_to_your_csv', "r",encoding = 'UTF-8') as myfile:
    reader = csv.DictReader(myfile)
    for i,line in itertools.islice(enumerate(reader) , start, 105): # iterate through row 100:105
    df # this is the data frame you want to export
    if i == start:
        df.iloc[0:0].to_csv(recent,sep = ",") # keep only header if this is the first you export
    df.to_csv(total,sep = ",",header =False)                                  
    del df