How can I inspect a linux process to determine how/when a process dies/terminates?

Question

I have a python irc bot which I start up as root by doing /etc/init.d/phenny start . Sometimes it dies, though and it seems to happen overnight.

What can I do to inspect it and see the status of the process in a text file?

Answer 1

If you're interested in really low level process activity, you can run the python interpreter under strace with standard error redirected to a file.

If you're only interested in inspecting the python code when your bot crashes, and you have the location in the source where the crash happens, you can wrap that location with try / except and break into the debugger in the except clause:

import pdb; pdb.set_trace()

You'll probably need to run your bot in non-daemon mode for that to work, though.

Answer 2

If you know it's still running, you can pstack it to see it's walkback. I'm not sure how useful that will be because you will see the call stack of the interpreter. You could also try strace or ltrace as someone else mentioned.

I would also make sure that in whatever environment the script runs in, you have set ulimit -c unlimited so that a core is generated in case python it is outright crashing.

Another thing I might try is to have this job executed by a parent that does not wait it's child. This should cause the proc table entry to stick around as a zombie even when the underlying job has exited.

Answer 3

你可能想尝试Python psutils ，这是我使用和工作的东西。

Answer 4

A cheap way to get some extra clues about the problem would be to start phenny with

/etc/init.d/phenny start 2>/tmp/phenny.out 1>&2

When it crashes, check the tail of /tmp/phenny.out for the Python traceback.

Answer 5

If you only need to verify that the process is running you could just run a script that checks the output of command

ps ax | grep [p]henny

every few seconds. If it's empty, then obviously the process is dead.