如何自动重启因OOM而被杀死的systemd服务

Question

How do I automatically restart a systemd service that is killed due to OOM.

I have added a restart but I am not sure if this would work on OOMs I cannot reproduce the OOM on my local dev box so knowing this works would be helpful.

[Service]
Restart=on-failure
RestartSec=1s

Error:

Main process exited, code=killed, status=9/KILL

Reading the docs https://www.freedesktop.org/software/systemd/man/systemd.service.html looks like the restart happens on unclean exit code and I think status 9 would come under it, but please can someone validate my thinking.

Answer 1

When a process terminates, the nature of its termination is made available to its parent process of record. For services started by systemd, the parent is systemd.

The available alternatives that can be reported are termination because of a signal (and which) or normal termination (and the accompanying exit status). By "normal" I mean the complement of "killed by a signal", not necessarily "clean" or "successful".

The system interfaces for process management do not provide any other options, but systemd also itself provides for applying a timeout to or using a watchdog timer with services it manages, which can lead to service termination on account of one of those expiring (as systemd accounts it).

The systemd documentation of the behavior of the various Restart settings provides pretty good detail on which termination circumstances lead to restart with which Restart settings. Termination because of a SIGKILL is what the message presented in the question shows, and this would fall into the "unclean signal" category, as systemd defines that. Thus, following the docs, configuring a service's Restart property to be any of always , on-failure , on-abnormal , or on-abort would result in systemd automatically restarting that service if it terminates because of a SIGKILL .

Most of those options will also produce automatic restarts under other circumstances as well, but on-abort will yield automatic restarts only in the event of termination because of unclean signal. (Note that although systemd considers SIGKILL unclean, it considers SIGTERM clean.)