How is Python blocking signals while os.system(“sleep…”)?

Question

When I run this Python script with os.system on Ubuntu 12.04:

import os, signal
signal.signal(signal.SIGABRT, lambda *args: os.write(2, 'HANDLER\n'))
print 'status=%r' % os.system('sleep 5')

, and then I send SIGABRT to the script process many times within 5 seconds, I get the following output:

status=0
HANDLER

This indicates that the signal delivery was blocked until sleep 5 exited, and then only a single signal was delivered.

However, with subprocess.call :

import os, signal, subprocess
signal.signal(signal.SIGABRT, lambda *args: os.write(2, 'HANDLER\n'))
print 'cstatus=%r' % subprocess.call('sleep 5', shell=True)

, all individual signals are delivered early:

HANDLER
HANDLER
HANDLER
cstatus=0

To distinguish the magic in glibc from the magic in Python, I rewrote the Python script in C, so os.system became system(3) :

#include <errno.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
static void handler(int signum) { (void)signum; write(2, "HANDLER\n", 8); }
int main(int argc, char **argv) {
  int got;
  struct sigaction sa;
  (void)argc; (void)argv;
  memset(&sa, 0, sizeof sa);
  sa.sa_handler = handler;
  if (0 != sigaction(SIGABRT, &sa, NULL)) return 3;
  got = system("sleep 5");
  return !printf("system=0x%x\n", got);
}

Signals got delivered early:

HANDLER
HANDLER
HANDLER
system=0x0

So I inferred that the magic is in Python 2.7, not in eglibc. But where is the magic? Based on the strace output and looking at the posix_system function in Modules/posixmodule.c , I couldn't figure out how Python blocks the signal until os.system returns.

Relevant code from Modules/posixmodule.c :

static PyObject *posix_system(PyObject *self, PyObject *args) {
  char *command;
  long sts;
  if (!PyArg_ParseTuple(args, "s:system", &command)) return NULL;
  Py_BEGIN_ALLOW_THREADS
  sts = system(command);
  Py_END_ALLOW_THREADS  
  return PyInt_FromLong(sts);
}

Maybe the magic is in Py_BEGIN_ALLOW_THREADS ?

Do I understand correctly that it's impossible for my Python signal handler (set up by signal.signal ) to execute until os.system returns?

Is it because signal handlers are blocked (on the Python level, not on the OS level) until Py_END_ALLOW_THREADS returns?

Here is the strace output of the Python code with os.system : http://pastebin.com/Wjn9KBye

Answer 1

Maybe the magic is in PY_BEGIN_ALLOW_THREADS?

The magic is mostly in system itself. system cannot return EINTR, so the libc implementation goes to pains to resume its wait 'ing on the child process. That means in your use of os.system , control never returns to python until the underlying system completes, and thus the python signal handling mechanics aren't invoked timely.

subprocess.call , however, is essentially doing this:

# Compare subprocess.py:Popen/_eintr_retry_call(os.waitpid, self.pid, 0)
while True:
  try:
    return os.waitpid(the_child_pid, 0)
  except OSError, e:
    if e.errno == errno.EINTR:  # signal.signal() handler already invoked
      continue
    raise

Here control does return to python when the underlying wait is interrupted. The OSError/EINTR prompts python to see if any signals were tripped and, if so, to invoke the user-supplied code block associated with that signal. (And that's how the interpreter adapts the system's signal semantics: set a flag, and check it between "atomic" python operations, invoking the user's code if appropriate.)