Writing a CSV file in Python with root.after gives an Error

Question

Im programming a datalogger (programmed in Python 3.5.3) on a Raspberry Pi, where i write data coming in from an ADC into a CSV file. I also have a GUI (made with tkinter) where i want to control the beginning and the end of a measurement. I start measuring by calling the following Function with a Pushbutton:

def do_start():
  spi.open(0,0)
  ch0 = [0x0c,0x00,0x00]
  a = adc(ch0)
  b = a[1]
  b = b&0b00001111      
  c = a[2]

  value = c+256*b
  voltage = value/4096*5
  zeit = millis()
  voltage = round(voltage,2)
  data=[voltage,zeit]


  with open(csvfilesave,"a") as output:
                 writer = csv.writer(output, delimiter=",",lineterminator = '\n')
                 writer.writerow(data)
  root.after(100, do_start)     

This works fine for the first 2048 repetitions, but after that i get the following error:

Exception in Tkinter callback
Traceback (most recent call last):
File "/usr/lib/python3.5/tkinter/__init__.py", line 1562, in __call__
File "/usr/lib/python3.5/tkinter/__init__.py", line 608, in callit
File "/home/pi/Documents/loggerprojekt/gui.py", line 34, in do_start
OSError: [Errno 24] Too many open files

I dont get this Error when i write the data to the CSV file in an endless loop, but then i can't access the GUI to stop it. Is there a way i can avoid this Error using tkinter?

Answer 1

It's impossible to reproduce your code, but my guess is that the "file" in the error message is not the csv file, but the file-like object that is being opened in the first line of do_start . You do spi.open(0,0) , but you never call close on spi .

Answer 2

I believe what happens is that even though after with open segment output.close() method is automatically called, it is the operating system which buffers the I/O calls and fails to close files as fast as your loop is opening them. This means that even after output.close(), OS still holds on the file a bit longer. There is similar issue discussed here .

However this is difficult to test and verify. One method would be to call a sleep -function to let OS have time to do it's cleanup, but this would slow down the program quite a bit. Also disabling buffering open(csvfilesave,"a", buffering=0) might help. Slow I/O is more likely to cause problems on RPi and quite platform dependend.

Anyhow, an easy fix for your problem would be to only open file once. Below is one example (I also removed recursion, because recursion because it will cause problems with big iterations):

import csv

def do_start():
    with open(r"c:\temp\foobar.txt", "a") as output:
        for i in range(3000):  # replace this with custom looping logic
            append_to_file(output)

def append_to_file(open_file):
    writer = csv.writer(open_file, delimiter=",", lineterminator='\n')
    data = "My hovercraft", "full of eels"  # replace data with your voltage, zeit
    writer.writerow(data)

do_start()