python process communications via pipes: Race condition

Question

So I have two Python3.2 processes that need to communicate with each other. Most of the information that needs to be communicated are standard dictionaries. Named pipes seemed like the way to go so I made a pipe class that can be instantiated in both processes. this class implements a very basic protocol for getting information around.

My problem is that sometimes it works, sometimes it doesn't. There seems to be no pattern to this behavior except the place where the code fails.

Here are the bits of the Pipe class that matter. Shout if you want more code:

class Pipe:
    """
    there are a bunch of constants set up here. I dont think it would be useful to include them. Just think like this: Pipe.WHATEVER = 'WHATEVER'
    """
    def __init__(self,sPath):
        """
        create the fifo. if it already exists just associate with it
        """
        self.sPath = sPath
        if not os.path.exists(sPath):
            os.mkfifo(sPath)
        self.iFH = os.open(sPath,os.O_RDWR | os.O_NONBLOCK)
        self.iFHBlocking = os.open(sPath,os.O_RDWR)

    def write(self,dMessage):
        """
        write the dict to the fifo
        if dMessage is not a dictionary then there will be an exception here.  There never is 
        """
        self.writeln(Pipe.MESSAGE_START)
        for k in dMessage:
            self.writeln(Pipe.KEY)
            self.writeln(k)
            self.writeln(Pipe.VALUE)
            self.writeln(dMessage[k])
        self.writeln(Pipe.MESSAGE_END)

    def writeln(self,s):
        os.write(self.iFH,bytes('{0} : {1}\n'.format(Pipe.LINE_START,len(s)+1),'utf-8'))
        os.write(self.iFH,bytes('{0}\n'.format(s), 'utf-8'))
        os.write(self.iFH,bytes(Pipe.LINE_END+'\n','utf-8'))

    def readln(self):
        """
        look for LINE_START, get line size
        read until LINE_END
        clean up
        return string
        """
        iLineStartBaseLength = len(self.LINE_START)+3  #'{0} : '
        try:
            s = os.read(self.iFH,iLineStartBaseLength).decode('utf-8')
        except:
            return Pipe.READLINE_FAIL

        if Pipe.LINE_START in s:
            #get the length of the line
            sLineLen = ''
            while True:
                try:
                    sCurrent = os.read(self.iFH,1).decode('utf-8')
                except:
                    return Pipe.READLINE_FAIL
                if sCurrent == '\n':
                    break
                sLineLen += sCurrent
            try:
                iLineLen = int(sLineLen.strip(string.punctuation+string.whitespace))
            except:
                raise Exception('Not a valid line length: "{0}"'.format(sLineLen))
            #read the line
            sLine = os.read(self.iFHBlocking,iLineLen).decode('utf-8')

            #read the line terminator
            sTerm = os.read(self.iFH,len(Pipe.LINE_END+'\n')).decode('utf-8')
            if sTerm == Pipe.LINE_END+'\n':
                return sLine
            return Pipe.READLINE_FAIL

        else:
            return Pipe.READLINE_FAIL

    def read(self):
        """
        read from the fifo, make a dict
        """
        dRet        = {}
        sKey        = ''
        sValue      = ''
        sCurrent    = None

        def value_flush():
            nonlocal dRet, sKey, sValue, sCurrent
            if sKey:
                dRet[sKey.strip()] = sValue.strip()
            sKey = ''
            sValue = ''
            sCurrent = ''

        if self.message_start():
            while True:
                sLine = self.readln()
                if Pipe.MESSAGE_END in sLine:
                    value_flush()
                    return dRet
                elif Pipe.KEY in sLine:
                    value_flush()
                    sCurrent = Pipe.KEY
                elif Pipe.VALUE in sLine:
                    sCurrent = Pipe.VALUE
                else:
                    if sCurrent == Pipe.VALUE:
                        sValue += sLine
                    elif sCurrent == Pipe.KEY:
                        sKey += sLine
        else:
            return Pipe.NO_MESSAGE

It sometimes fails here (in readln):

        try:
            iLineLen = int(sLineLen.strip(string.punctuation+string.whitespace))
        except:
            raise Exception('Not a valid line length: "{0}"'.format(sLineLen))

It doesn't fail anywhere else.

An example error is:

Not a valid line length: "KE 17"

The fact that it's intermittent says to me that it's due to some kind of race condition, I'm just struggling to figure out what it might be. Any ideas?

EDIT added stuff about calling processes

How the Pipe is used is it is instantiated in processA and ProcessB by calling the constructor with the same path. Process A will then intermittently write to the Pipe and processB will try to read from it. At no point do I ever try to get the thing acting as a two way.

Here is a more long winded explanation of the situation. I've been trying to keep the question short but I think it's about time I give up on that. Anyhoo, I have a daemon and a Pyramid process that need to play nice. There are two Pipe instances in use: One that only Pyramid writes to, and one that only the daemon writes to. The stuff Pyramid writes is really short, I have experienced no errors on this pipe. The stuff that the daemon writes is much longer, this is the pipe that's giving me grief. Both pipes are implemented in the same way. Both processes only write dictionaries to their respective Pipes (if this were not the case then there would be an exception in Pipe.write).

The basic algorithm is: Pyramid spawns the daemon, the daemon loads craze object hierarchy of doom and vast ram consumption. Pyramid sends POST requests to the daemon which then does a whole bunch of calculations and sends data to Pyramid so that a human-friendly page can be rendered. the human can then respond to what's in the hierarchy by filling in HTML forms and suchlike thus causing pyramid to send another dictionary to the daemon, and the daemon sending back a dictionary response.

So: only one pipe has exhibited any problems, the problem pipe has a lot more traffic than the other one, and it is a guarentee that only dictionaries are written to either

EDIT as response to question and comment

Before you tell me to take out the try...except stuff read on. The fact that the exception gets raised at all is what is bothering me. iLineLengh = int(stuff) looks to me like it should always be passed a string that looks like an integer. This is the case only most of the time, not all of it. So if you feel the urge to comment about how it's probably not an integer please please don't.

To paraphrase my question: Spot the race condition and you will be my hero.

EDIT a little example:

process_1.py:

oP = Pipe(some_path)
while 1:
    oP.write({'a':'foo','b':'bar','c':'erm...','d':'plop!','e':'etc'})

process_2.py:

oP = Pipe(same_path_as_before)
while 1:
    print(oP.read())

Answer 1

After playing around with the code, I suspect the problem is coming from how you are reading the file.

Specifically, lines like this:

os.read(self.iFH, iLineStartBaseLength)

That call doesn't necessarily return iLineStartBaseLength bytes - it might consume "LI" , then return READLINE_FAIL and retry. On the second attempt, it will get the remainder of the line, and somehow end up giving the non-numeric string to the int() call

The unpredictability likely comes from how the fifo is being flushed - if it happens to flush when the complete line is written, all is fine. If it flushes when the line is half-written, weirdness.

At least in the hacked-up version of the script I ended up with, the oP.read() call in process_2.py often got a different dict to the one sent (where the KEY might bleed into the previous VALUE and other strangeness).

I might be mistaken, as I had to make a bunch of changes to get the code running on OS X, and further while experimenting. My modified code here

Not sure exactly how to fix it, but.. with the json module or similar, the protocol/parsing can be greatly simplified - newline separated JSON data is much easier to parse:

import os
import time
import json
import errno


def retry_write(*args, **kwargs):
    """Like os.write, but retries until EAGAIN stops appearing
    """

    while True:
        try:
            return os.write(*args, **kwargs)
        except OSError as e:
            if e.errno == errno.EAGAIN:
                time.sleep(0.5)
            else:
                raise


class Pipe(object):
    """FIFO based IPC based on newline-separated JSON
    """

    ENCODING = 'utf-8'

    def __init__(self,sPath):
        self.sPath = sPath
        if not os.path.exists(sPath):
            os.mkfifo(sPath)

        self.fd = os.open(sPath,os.O_RDWR | os.O_NONBLOCK)
        self.file_blocking = open(sPath, "r", encoding=self.ENCODING)

    def write(self, dmsg):
        serialised = json.dumps(dmsg) + "\n"
        dat = bytes(serialised.encode(self.ENCODING))

        # This blocks until data can be read by other process.
        # Can just use os.write and ignore EAGAIN if you want
        # to drop the data
        retry_write(self.fd, dat)

    def read(self):
        serialised = self.file_blocking.readline()
        return json.loads(serialised)

Answer 2

Try getting rid of the try: , except: blocks and seeing what exception is actually being thrown.

So replace your sample with just:

iLineLen = int(sLineLen.strip(string.punctuation+string.whitespace))

I bet it'll now throw a ValueError , and it's because you're trying to cast "KE 17" to an int .

You'll need to strip more than string.whitespace and string.punctuation if you're going to cast the string to an int .