Reading CSV file with python

Question

filename = 'NTS.csv'
mycsv = open(filename, 'r')
mycsv.seek(0, os.SEEK_END)

while 1:
   time.sleep(1)
   where = mycsv.tell()
   line = mycsv.readline()

if not line:
    mycsv.seek(where)
else:
    arr_line = line.split(',')
    var3 = arr_line[3]

    print (var3)

I have this Paython code which is reading the values from a csv file every time there is a new line printed in the csv from external program. My problem is that the csv file is periodically completely rewriten and then python stops reading the new lines. My guess is that python is stuck on some line number and the new update can put maybe 50 more or less lines. So for example python is now waiting a new line at line 70 and the new line has come at line 95. I think the solution is to let mycsv.seek(0, os.SEEK_END) been updated but not sure how to do that.

Answer 1

You can do a simpler way of reading lines in your program. Instead of trying to use seek in order to get what you need, try using readlines on the file object mycsv .

You can do the following:

mycsv = open('NTS.csv', 'r')
csv_lines = mycsv.readlines()

for line in csv_lines:
    arr_line = line.split(',')
    var3 = arr_line[3]
    print(var3)

Answer 2

What you want to do is difficult to accomplish without rewinding the file every time to make sure that you are truly on the last line. If you know approximately how many characters there are on each line, then there is a shortcut you could take using mycsv.seek(-end_buf, os.SEEK_END), as outlined in this answer . So your code could work somehow like this:

avg_len = 50  # use an appropriate number here
end_buf = 3 * avg_len / 2

filename = 'NTS.csv'
mycsv = open(filename, 'r')
mycsv.seek(-end_buf, os.SEEK_END)
last = mycsv.readlines()[-1]

while 1:

    time.sleep(1)
    mycsv.seek(-end_buf, os.SEEK_END)
    line = mycsv.readlines()[-1]

    if not line == last:

        arr_line = line.split(',')
        var3 = arr_line[3]

        print (var3)

Here, in each iteration of the while loop, you seek to a position close to the end of the file, just far back enough that you know for sure the last line will be contained in what remains. Then you read in all the remaining lines (this will probably include a partial amount of the second or third to last lines) and check if the last line of these is different to what you had before.