FreeBSD Jail fails to start when launched from a python thread

Question

The following works:

ezjail-admin create -f default test 10.0.0.1
ezjail-admin start test

Or in Python:

import subprocess, shlex
command = 'ezjail-admin create -f default test 10.0.0.1'
subprocess.Popen(shlex.split(command)).wait()
command = 'ezjail-admin start test'
subprocess.Popen(shlex.split(command)).wait()

But when executed in a Thread (under CherryPy), the jail installs fine but it is never properly started:

root    15231  0.0  0.0  8296  2056   1  I+    4:52PM   0:00.01 /bin/sh /usr/local/etc/rc.d/ezjail.sh start content
root    15240  0.0  0.0  8296  2120   1  I+    4:52PM   0:00.06 sh /etc/rc.d/jail onestart content
root    15407  0.0  0.0  8296  2016   1  I+J   4:52PM   0:00.01 /bin/sh /etc/rc
root    15467  0.0  0.0  8296  2060   1  I+J   4:52PM   0:00.00 /bin/sh /etc/rc
root    15474  0.0  0.0  6920  1224   1  I+J   4:52PM   0:00.00 /usr/sbin/syslogd -ss

The jail starts, but /etc/rc seems to freeze after starting syslogd.

Somehow, when executed in a thread, the jail rc fails. I suspect some environment setting is missing (maybe a pty?) for jexec to properly run.

Trying to attach a console also fails (will actually launch a separate copy of the jail) with either jexec or ezjail-admin console . /var/log of the jail has no content (aside from empty log files being created at syslogd startup) and neither does the host log files.

Would anybody know why executing working commands in a non-threaded app works, but as soon as it is run under a thread, something goes terribly wrong?

TLDR: When trying to start a jail from a python thread, rc hangs after syslogd has been launched. Same commands will successfully start the jail in a non-threaded app.

Edit: A fork works... It has to do with threads.

The following does not work for ezjail-admin start . The ezjail-admin create works perfectly fine.

class TestThread(threading.Thread):
  def run(self):
    command = 'ezjail-admin create -f content content 10.0.254.33'
    os.system(command)
    command = 'ezjail-admin start content'
    os.system(command)    

tt = TestThread()
tt.start()

Actual code can be seen at https://github.com/masom/Puck/tree/master/client Current implementation use an early fork, but threads would be much cleaner.

Answer 1

os.system does not wait for the command to finish. So your create and start run in parallel resulting the start failing due to race condition. Either put a sleep after call to create or put a subprocess.Popen call with .wait()

What I'm saying is the wait does not work because ezjail-admin is a shell script which does not wait for results either.

To get proof whether I'm right or wrong please try running same commands with 1second sleep in-between.

Jail creation does not need pty. In fact you can run a virtual jail out of memory only statically-linked process with no filesystem at all (though this fu is difficult to find on the network nowadays)...

Sleep is clunky but checking for running jails through jls is a way (ezjail registers there as well as it is unified).

Answer 2

Do you really want to use popen there? Are you expecting to read stdout or stderr, or write to stdin? From the .wait() it doesn't look like it. If so, why use popen?

I don't know about Python specifically, so I am going to assume that subprocess.Popen is implemented in terms of popen(3).

Internally, popen must fork/exec, so I doubt you really have a threading issue. Because you are using popen, it is more likely that something is blocking attempting to use one of the pipes to the parent process. Adding threading will change how Python handles I/O, which might expose the problem.

So, to test this, you can try the same command, but with stdin, stdout and stderr redirected to /dev/null. Use this command:

sh -c 'ezjail-admin start test < /dev/null > /dev/null 2> /dev/null'

The ezjail documentation might tell you about whether it does anything with standard I/O handles. If not, the underlying jail(2) call will use inherited handles, and given your code I am pretty certain you want to close them, or redirect them somewhere useful.

Update:

If the command worked with the standard handles redirected to /dev/null, then the problem has to do with how you deal with standard I/O handles, not threads. You need to decide where you want those handles to go, and set them up correctly when you start the jail. If you are going to read and/or write to pipes connected to the jail child, you need to be aware of the big red warnings on the documentation page you mentioned in comments to avoid deadlocks. That is almost certainly the issue you are seeing.

You talk about ssh and server processes; unclear what you are really getting at. You are creating a jail and then executing /etc/rc in the jail. That process inherits the standard I/O handles from its parent, and sets up a system environment. The processes that make up the system environment will also inherit those handles and use them. The system will behave differently, depending on where they point. The different methods of invoking /etc/rc in a jail vary those handles. To make it work with "threads" just set up the handles correctly.