Ubuntu: reboot if / mounted readonly due to errors=remount-ro

Question

Problem: We generally face a problem where ubuntu os gets mounted readonly. Reason is clear as mentioned in fstab on errors=remount-ro.

Question: Is there any mechanism to reboot the appliance if it comes to readonly mounted state.

Tried: I tried to write a script as below which will get monitored by watchdog. This works but it continuously reboot if script return exit 1 due to any mount point is still readonly. What i expect is to check if uptime if less then a day then it should not reboot even though any mount point is readonly?

root@ubuntu1404:/home/ubuntu# cat /rofscheck.sh

#!/bin/bash

now=`date`
echo "------------"
echo "start : ${now}"
up_time=`awk '{print int($1)}' /proc/uptime`

#if uptime is less than 1 day then skip test
if [ "$up_time" -lt 86400 ]; then
    echo "uptime is less ${now}, exit due to uptime"
    exit 0
fi
grep -q ' ro' /proc/mounts > /dev/null 2>&1 || exit 0

# alert watchdog that fs is readonly.
exit 1

Now in /etc/watchdog.conf below config is done.

test-binary = /rofscheck.sh

To reproduce the problem to mount readonly all mounted fs, ran this:

$ echo u > /proc/sysrq-trigger

which does emergency remount readonly.

Answer 1

this script worked for me, even if it is much similar to yours. I am running it on Ubuntu 14.04 64bit with latest updates.

#!/bin/bash

now=$(date)
up_time=$(awk '{print int($1)}' /proc/uptime)
min_time=7200

#if uptime is less than 2 hours then skip test
if [ ${up_time} -lt ${min_time} ]; then
    echo "uptime is ${up_time} secs, less than ${min_time} secs (now is ${now}): exit 0 due to uptime"
    exit 0
fi

exit=$(grep ' ro,' /proc/mounts | wc -l)
if [ ${exit} -gt 0 ]; then
        exit 1
else
        exit 0
fi

Please note that I have setup the minimum time as a variable, adjust it at your convenience.

A small important notice: I am using this solution on a very cheap couple of cloud servers I own that I use for testing purposes only. I have also setup filesystem check at every reboot with the following command:

tune2fs -c 1 /dev/sda1

I would never use this kind of watchdog usage in a production environment.

Hope this helps. Best regards.