Python: Do I need to catch EINTR in a pipe-reading loop

Question

tl;dr

Should I handle EINTR "system call interrupted" errors when reading a pipe in Python, and if so, how do I test such code?

Description

In the traceback below, self._dataq is a multiprocessing.Queue (technically, I'm using the billiard library, but I think they're basically the same code). A Python subprocess occasionally writes to the other end of the queue. I think what happened is that a system call was reading the pipe that feeds the queue, and a signal arrived -- possibly a SIGINT from a second Ctrl+C event (the first SIGINT occurred where you see the user's ^C on the second line of the logging output, and my signal handler caught that SIGINT, as you can see in the WARNING message in the log).

[INFO     2014-03-05 14:16:06,000] Doing some work, la-dee-da
^C[WARNING 2014-03-05 14:16:07,344] Commencing shutdown. (Signal SIGINT, process 2469.). Press Ctrl+C again to exit immediately.
[DEBUG    2014-03-05 14:16:07,347] Terminating subprocess
Traceback (most recent call last):
[... a bunch of stuff omitted]
  File "mycode.py", line 97, in __next__
    result = self._dataq.get(timeout=0.1)
  File "/usr/local/lib/python2.7/site-packages/billiard/queues.py", line 103, in get
    if timeout < 0 or not self._poll(timeout):
IOError: [Errno 4] Interrupted system call

The statement result = self._dataq.get(timeout=0.1) in the traceback above is in the middle of a loop that looks like the following. The main purpose of the loop is so that I can give up trying to read from self._dataq when self.timedout() starts returning True .

import queue
while True:
    try:
        result = self._dataq.get(timeout=0.1)
    except queue.Empty:
        if self.timedout():
            self.close()
            raise MyTimedoutError()
    else:
        break

Question

If my theory about why the IOError occurred is correct, then the try ... except block above should catch and ignore IOError s when they're caused by interrupted system calls. If it is a signal that caused the EINTR error, than the mere act of coming back into Python to run the except IOError: statement would allow Python-level signal handlers to run.

Is that correct? If so, is it possible to test this change in my code? It's not obvious to me how I would write a unit test that wouldn't contain a severe race condition.

Answer 1

Python 3.5 solves this problem by putting the responsibility for dealing with EINTR on the Python runtime instead of on application code. See PEP 475 and the Python 3.5 Changelog .

Answer 2

I'm going to call this a bug in Python. I cannot find any documentation that multiprocessing.Queue is allowed to raise IOError on EINTR (though it would make sense to raise on other I/O issues, which is why you should check the errno attribute of the raised exception before ignoring it; see errno ), and it doesn't directly map to any low-level C function which would provide such behavior. There was some discussion of handling all EINTR s at the C level, but that didn't make it into 3.4 (and I doubt it will make it into 2.x at all), so this is probably still fair game to report.