Restart Crashed Python Script in Ubuntu?

Question

I've been using AWS' free web service recently to run python scripts in the cloud with EC2. I have it set to start running at a specific time every day using Crontab then run all day until it restarts.

The problem I have is there's a good chance the program will crash. I have a bunch of error catching blocks on the program, but they don't always stop a crash. In cases where it crashes, what I really want is for it to restart and come back online again so I miss as little as possible. But I don't know how to do that.

I've thought about including a bunch of Crontab instances to run it throughout the day, but won't this just create a bunch of instances of my script? Is there some way to configure this to only work if the script isn't currently running?

Answer 1

If it's meant to run all the time, and you have sudo access, then systemd can do this for you. It adds some complexity to your setup, but this is an ideal use-case for it.

Here's an example my_python.service file:

[Unit]
Description=My Python Service

[Service]
Type=simple
WorkingDirectory=/where/it/runs/from
ExecStart=/usr/bin/python /path/to/script.py
User=notroot
Group=notroot
Restart=always

[Install]
WantedBy=multi-user.target

Not all of these options are needed, but this will do what you want and it will provide logging through journalctl -u my_python .

Going through them one by one:

• Description= - Solely for your sanity and making it more self-documenting
• [Service] - A block that describes how systemd should run and control the process
• Type= - is this forking, a socket, or what kind of execution behavior should systemd expect. 9/10 this is just Simple
WorkingDirectory= - (optional) just specifies where the program should be run from. If you rely on file inputs or generate file outputs then this is very helpful
• StartExec= - the actual command to run. It works just like running it from a terminal, except it requires absolute paths, hence the full /usr/bin/python. You may want to run which python to be sure you have the path correct
• User=/Group= - this keeps it from being run as root and is good practice
• Restart=always - if my program stops running, when should I restart it? This is the magic part you want that will have systemd monitor the process for you and automatically try to restart it if it fails. If it can't start it again, it will continue to try, but will log the attempts in systemctl status my_python

The last two lines are there to make it run on the last stage of boot and you don't really need them, but it's a nice to have. It helps you when you want stuff to startup during OS boot.

Once you have this file, put it in systemd 's unit directory ( /usr/lib/systemd/system for yum systems) and run the following commands:

• systemctl daemon-reload - looks for new files and file changes in the unit directory
• systemctl enable my_python - or whatever you named your service, this makes the program run on startup
• systemctl start my_python - this actually runs your program and monitors it for you.