Restart ffmpeg process using monit if TOTAL CPU is less than 1%

Question

I have used kind of similar solution like this Restarting ffmpeg process using monit to restart my ffmpeg stream in case it fails for some reason. Remember its not duplicate problem/question, because I have other issues unlike the example question/solution Restarting ffmpeg process using monit , which I'm gonna explain below. So here is my monit configuration:

    check process FFMPEGStream with pidfile PATH-to-file/streampid.pid
    start program = "PATH-to-file/streambash.sh restart"
    stop program = "PATH-to-file/streambash.sh stop"
    if TOTAL CPU is less than 1% for 10 cycles then restart

Here is my streambash.sh file:

    #!/bin/bash
    pid_file="PATH-to-file/streampid.pid"

    case "$1" in
     restart)
        PATH-to-file/streambash.sh stop
        PATH-to-file/streambash.sh start
           ;;

     start)
        rm $pid_file
        /usr/bin/ffmpeg -i "INPUT-PATH" -c:v libx264 -b:v 900k -preset ultrafast -aspect 16:9 -s 640x376 -strict experimental -c:a aac -b:a 96k -f flv "RTMP-PATH" &> /dev/null &
        ch_pid=$! 
        echo "Start Stream1: ffmpeg = $ch_pid";
        echo $ch_pid > $pid_file
           ;;

     stop)
        echo "Stop ffmpeg Stream1";
        kill `cat $pid_file` &> /dev/null
           ;;

     *)
        echo "Usage: PATH-to-file/streambash.sh {start|stop|restart}"
        exit 1
           ;;

     esac
    exit 0
    echo $pid_file

Monit can start the bash file successfully, but when this condition "if TOTAL CPU is less than 1% for 10 cycles then restart" is matched in monit configuration, it tries to restart, it gives error that process is not running. But in actual the ffmpeg process still runs in the background and I can see that the stream is live on my website. Here is monit logs:

    [CET Jan 10 12:55:02] error    : 'FFMPEGStream' total cpu usage of 0.4% matches resource limit [cpu usage>1.0%]
    [CET Jan 10 12:55:07] error    : 'FFMPEGStream' total cpu usage of 0.0% matches resource limit [cpu usage>1.0%]
    [CET Jan 10 12:55:12] error    : 'FFMPEGStream' total cpu usage of 0.0% matches resource limit [cpu usage>1.0%]
    [CET Jan 10 12:55:17] error    : 'FFMPEGStream' total cpu usage of 0.4% matches resource limit [cpu usage>1.0%]
    [CET Jan 10 12:55:22] error    : 'FFMPEGStream' total cpu usage of 0.9% matches resource limit [cpu usage>1.0%]
    [CET Jan 10 12:55:27] error    : 'FFMPEGStream' total cpu usage of 0.9% matches resource limit [cpu usage>1.0%]
    [CET Jan 10 12:55:32] error    : 'FFMPEGStream' total cpu usage of 0.0% matches resource limit [cpu usage>1.0%]
    [CET Jan 10 12:55:37] error    : 'FFMPEGStream' total cpu usage of 0.0% matches resource limit [cpu usage>1.0%]
    [CET Jan 10 12:55:42] error    : 'FFMPEGStream' total cpu usage of 0.0% matches resource limit [cpu usage>1.0%]
    [CET Jan 10 12:55:47] error    : 'FFMPEGStream' total cpu usage of 0.4% matches resource limit [cpu usage>1.0%]
    [CET Jan 10 12:55:50] info     : 'FFMPEGStream' trying to restart
    [CET Jan 10 12:55:50] info     : 'FFMPEGStream' stop: PATH-to-file/streambash.sh
    [CET Jan 10 12:55:51] info     : 'FFMPEGStream' start: PATH-to-file/streambash.sh
    [CET Jan 10 12:55:56] error    : 'FFMPEGStream' process is not running
    [CET Jan 10 12:55:58] info     : 'FFMPEGStream' trying to restart
    [CET Jan 10 12:55:58] info     : 'FFMPEGStream' start: PATH-to-file/streambash.sh
    [CET Jan 10 12:56:04] error    : 'FFMPEGStream' process is not running
    [CET Jan 10 12:56:04] info     : 'FFMPEGStream' trying to restart
    [CET Jan 10 12:56:04] info     : 'FFMPEGStream' start: PATH-to-file/streambash.sh
    [CET Jan 10 12:56:09] error    : 'FFMPEGStream' process is not running
    [CET Jan 10 12:56:09] info     : 'FFMPEGStream' trying to restart
    [CET Jan 10 12:56:09] info     : 'FFMPEGStream' start: PATH-to-file/streambash.sh

Monit keeps trying to restart the process and on each retry, it dumps a new pid to the PATH-to-file/streampid.pid, but as I said it seems, it somehow can stop the actual ffmpeg stream/pid, which keep running in the background.

Answer 1

Your poll cycle / daemon check interval is very low, 5 seconds?

FFMpeg isn't starting within 5 seconds so monit tries to start it again, constantly in a loop.

If you want to have such a low check interval, you need to set a timeout on the start command, along the lines of:

start program = "PATH-to-file/streambash.sh restart" with timeout 30 seconds

This really helps me to understand monit's way of thinking, watch the live log in a terminal window while monit is doing things:

 tail -f /var/log/monit.log