How to read/write to tempfile from multiple threads

Question

Context:

Operating system: Windows 8.1

python --version: Python 2.7.8

I am trying to to read/write from/to a stream, and use a thread to write to that stream and the another to read from it new bits of data that was written. I am using the tempfile module, writing binary data to it using one method, and another to read from it.

In below code, t1 runs the writing, and t2 runs the reading thread method. t3 & t4 exits makes the t1 and t2 loops exit.

My expected output from the read is:

READ @0: 0|1|2|3|.........|N    --- L Bytes read
READ @L: N|N+1|N+2|.......|M    --- J Bytes read
READ @L+J: M|M+1|M+3|.....|P    --- K Bytes read
READ @L+J+K:                    --- 0 Bytes read (nothing was written by write thread)

and so on such that whenever there is more data in the tempfile it is read and output, but at the same time the writing must go on by another thread so that data that is received from the stream is written.

Problem:

When i run it the output i get varies One of the output received:

> python.exe tmp.py
Exception in thread Thread-1:
Traceback (most recent call last):
  [truncated output]
  File "C:/tmp.py", line 11, in write_temp
READ @ 0 :  264261|975|263976|263977|...[truncated output]...|263972|263
    self.myf.write(str(self.current_count)+"|")
IOError: [Errno 0] Error

Another output:

> python.exe tmp.py
READ @ 0 :  0|289721|289722|...[truncated output]...289718|28971
Exception in thread Thread-1:
Traceback (most recent call last):
  [truncated output]
    self.myf.write(str(self.current_count)+"|")
IOError: [Errno 0] Error

Other output are more or less variations of the above output.

I think the issue is due to file descriptor pointer being changed by the read, but i thought append always writes to end of file.

Source Code

The following is an abstraction of the actual code for the stream, the stream is a binary stream of data read from a subprocess stdio, and written to another subprocess stdio.

import threading, tempfile
class MultipleThreadTIO:
    def __init__(self):
        self.myf = tempfile.TemporaryFile(mode='a+b')
        self.current_count = 0
        self.do_write = True
        self.do_read = True

    def write_temp(self):
        while self.do_write:
            self.myf.write(str(self.current_count)+"|")
            self.current_count += 1

    def read_temp(self):
        read_at = 0L
        while self.do_read:
            self.myf.seek(read_at)
            d = self.myf.read()
            if len(d.strip()) > 0:
                print "READ @",read_at,": ", self.myf.read()
            read_at = self.myf.tell()

    def stop_write(self):
        self.do_write = False

    def stop_read(self):
        self.do_read = False

    def __del__(self):
        #self.myf.seek(0)
        #print ":::DATA CONTENT:::\n"
        #print self.myf.read()
        #print ":::END DATA CONTENT:::"
        self.myf.close()

mtio = MultipleThreadTIO()

t1 = threading.Timer(0.1, mtio.write_temp)
t2 = threading.Timer(0.5, mtio.read_temp)

t3 = threading.Timer(5, mtio.stop_write)
t4 = threading.Timer(3, mtio.stop_read)

t1.start()
t2.start()
t3.start()
t4.start()

Questions:

Question 1: any fix to the above issue?

Question 2: should i use queues/os.pipe(/other?) instead of the tempfile for this?

Question 3: any other better approaches to this situation?

Important : solution must be cross platform.

Answer 1

this is an abstraction using a Queue , I think this is better if you're using a queue/pipe behaviour. I'm not sure what you're actually writing, so I just added the running counter to the queue. Writing to the queue is stopped using an Event and reading from the queue is stopped by sending a stop message (but you can add a signal if you want finer control):

import threading
import Queue

class MultipleThreadTIO:
    def __init__(self):
        self.queue = Queue.Queue()
        self.current_count = 0
        self.stop_write = threading.Event()

    def write_temp(self):
        while not self.stop_write.isSet():
            self.queue.put(str(self.current_count)+"|")
            self.current_count += 1
        self.stop()

    def read_temp(self):
        while True:
            msg = self.queue.get()
            if msg == 'close':
                break
            else:
                print "READ @: " + msg

    def stop(self):
        self.queue.put('close')

mtio = MultipleThreadTIO()

t1 = threading.Timer(0.1, mtio.write_temp)
t2 = threading.Timer(0.5, mtio.read_temp)


t1.start()
t2.start()

t3 = threading.Timer(5, mtio.stop_write.set)

t3.start()

I can't test on windows but I think it should work since this is pretty standard. It runs on Ubuntu 14.04 x86_64