Tux + Linux Items

Help promote Linux and FOSS at the
Sample T-Shirt from the ERACC Cafe Press Store
ERACC Cafe Press Store

Linux: Monitor a Service with a Watchdog Script

Old Unix hands already know this, but new Unix (Linux) users may be asking, ‘What is a “watchdog script”?’ Basically it is a bash or other script that is run via cron periodically to check on a persistent service. The watchdog script takes actions based on the state of the service it monitors.

There are other examples of watchdog scripts on the internet. Just search for them using your favorite search engine to see them. Following is a watchdog script we created recently for a client to monitor an e-mail to pager system my company wrote for the client. Here is the script (with sensitive bits changed to protect the innocent):

#!/bin/bash
#
# watchdog
#
# Run as a cron job to keep an eye on what_to_monitor which should always
# be running. Restart what_to_monitor and send notification as needed.
#
# This needs to be run as root or a user that can start system services.
#
# Revisions: 0.1 (20100506), 0.2 (20100507)

NAME=what_to_monitor
START=/full/path/to/$NAME
NOTIFY=person1email
NOTIFYCC=person2email
GREP=/bin/grep
PS=/bin/ps
NOP=/bin/true
DATE=/bin/date
MAIL=/bin/mail
RM=/bin/rm

$PS -ef|$GREP -v grep|$GREP $NAME >/dev/null 2>&1
case "$?" in
   0)
   # It is running in this case so we do nothing.
   $NOP
   ;;
   1)
   echo "$NAME is NOT RUNNING. Starting $NAME and sending notices."
   $START 2>&1 >/dev/null &
   NOTICE=/tmp/watchdog.txt
   echo "$NAME was not running and was started on `$DATE`" > $NOTICE
   $MAIL -n -s "watchdog notice" -c $NOTIFYCC $NOTIFY < $NOTICE
   $RM -f $NOTICE
   ;;
esac

exit

In case you are a new Linux administrator and are virgin to all things Unix-ish we will explain what this script does.

First of all, if you want to run a script unattended in cron then the first line with “#!”, called a “shebang”, tells whatever is calling the script what to use for processing the script. In this case we want to use bash so the line is “#!/bin/bash” for that. If this were a Perl script then the shebang line may look like “#!/usr/bin/perl”, depending on where the Perl executable resides on your system.

Following the shebang line are several lines of comments, which should be self explanatory. Then the variables used in our script are assigned. These too should be self explanatory. If not please post a comment to ask about them.

The “NAME=what_to_monitor” line is quite important for our purposes. This is the actual name of the program or script that would show up in a process list. We use that in the script to check if that shows up in the process list in the line:

$PS -ef|$GREP -v grep|$GREP $NAME >/dev/null 2>&1

Yes, we could actually try to find a process ID number (PID) for the application we want to monitor. However, as long as the application has a unique name in the process list the method used here will work just fine. There is more we could do to see if the application is hung even though it shows up in the process list. In the case of this particular process we are monitoring it will not hang but may die for some reason or another. If it dies then it will immediately, or nearly immediately, disappear from the process list.

The “$START 2>&1 >/dev/null &” line in our watchdog actually starts our process using the original process script itself from the service’s home directory. This could instead call the “/etc/rc.d/init.d/startupscript” for the script or program that is run as a service. The NAME variable, START variable and line to start the service would then look something like:

NAME=startupscript
START=/etc/rc.d/init.d/$NAME

$START start 2>&1 >/dev/null &

Presuming the startupscript uses the word “start” to start the service.

Once we have our script written we want to use it in cron. We use root’s cron for this but one could use any user that has the ability to (re)start system services. We save the watchdog script under /root/bin/watchdog, set it to be executable with “chmod 700 /root/bin/watchdog” and call it from cron using the following crontab line:

* * * * * /root/bin/watchdog

This causes the watchdog to run every minute so it checks the service as often as possible. One can modify the crontab line to run the watchdog whenever one needs it to run. But for persistent services that need to be running we always use a once per minute cron job for our watchdog scripts.

In this script we redirect the majority of our output to /dev/null because we do not want to inundate root’s, or the calling user’s, e-mail with cron job messages every minute. The default in cron is to mail the output from cron jobs to the calling user’s e-mail account. We do want to notify someone when a problem occurs causing our watchdog to trigger. So the NOTIFY and NOTIFYCC variables are set to the local or remote e-mail addresses of the people who need to be notified. Then these lines handle the notification message:

NOTICE=/tmp/watchdog.txt
echo “$NAME was not running and was started on `$DATE`” > $NOTICE
$MAIL -n -s “watchdog notice” -c $NOTIFYCC $NOTIFY < $NOTICE

Please feel free to post comments with pointers to other watchdog scripts or to “fine tune” what is shown here. Questions are also welcome.

Notice: All comments here are approved by a moderator before they will show up. Depending on the time of day this can take several hours. Please be patient and only post comments once. Thank you.

Share

13 comments to Linux: Monitor a Service with a Watchdog Script

  • One other thing. If the NAME=startupscript is not the name that will show up in the process list with ‘ps’ then the lines that set these variables need to be:

    NAME=processlistname
    START=/etc/rc.d/init.d/startupscript

  • Hi All,

    Some administrative critics are critiquing this script on reddit (apparently afraid I won’t accept criticism here). Since the majority of you reading here probably do not go on reddit I’ll address the critiques here.

    First the suggestion is to just “use monit, or nagios, or daemontools”. Sure, I could use those, so could you. However, it is important that those of us who do know shell scripting pass along what we know for the future users who will want to learn this for themselves. Even if what we do is not “perfect” in the mind’s eye of some intelligentsia administrators it still advances the knowledge base available to new Unix and Linux administrators. This is important. Sometimes one needs to “reinvent the wheel” to learn how to make better wheels. Not just use someone else’s wheel.

    Another critique is that the script “needlessly creates a temp file that is vulnerable to a symlink attack”. Okay, so randomize the name of the temporary file, or suggest a better way to craft and e-mail the warning message. But unless one uses the “symlink attack” to cause the script to overwrite some other file with the contents of its message I do not see what else could be done. I think if someone with malicious intent is deep enough in your system to know about your one-off, custom scripts and craft some sort of exploit for them then you are already in trouble my friends.

    Finally, one critique states, “There is no excuse for a daemon that dies.” Granted. In this case the service ran flawlessly for several years. It even restarted properly following some power failures. However, in one instance this Spring of 2010 following some severe weather that caused extended outages the service did not start when the system automatically restarted when power was restored. The problem was not noticed for several days because the client had other problems with which to contend. As this small business client does not have a system administrator on-site monitoring the systems and wanted something quickly to “keep this from happening again” we wrote the script. The failure may never happen again, but if it does this script will catch it and start the service.

  • Everyone,

    Again a suggestion on reddit is to use ‘pgrep’ in place of “$PS -ef|$GREP -v grep|$GREP $NAME”. While pgrep is available in default installs on Linux and FreeBSD it is not available by default in other Unix systems. As a system administrator relying on an application that may not be available when one changes employers is okay only as long as one knows alternate ways of getting the job done. All Unix systems I know about have some form of ‘ps’ and ‘grep’ in a default install. Not all of them have pgrep in a default install. Don’t kid yourself that you can always convince The Powers That Be to allow you to install some tools that are not part of the base install. Always learn more than one way to get your scripts to work.

  • Zartan

    Good points. I think you can avoid the temp file simply by piping the echo into mail. (AFAIK, /bin/mail reads stdin, instead of bypassing redirection by using /dev/tty.)

    echo “$NAME was not running and was started on `$DATE`” | $MAIL -n -s “watchdog notice” -c $NOTIFYCC $NOTIFY

    Also, you can cut one process from the detection pipeline by using sed, which is so ancient that it should be present on all *nix variants. You’ll have to check against the stdout result instead of using exit status.

    result=` $PS -ef | $SED -n -e ‘\|$NAME|’ `
    if [ "X$result" = "X" ]
    then
    # It’s dead, Jim.
    else
    # It’s alllliiiiivvveeeee!
    fi

  • anonymous

    you can get away with the ps magic if you use pidof.
    using logger in place of echo enables you to use the system
    loggin facilies, what can be used with syslog to call for help or what ever

    the basic idea looks like that here:

    pidof || { logger -t checker ” restarted” ; prg || logger -t checker ” restarted failed” ; }

  • Zartan

    Bah. I messed up the PS results by not having sed delete its own line. Try:

    result=` $PS -ef | $SED -n -e ‘/sed -n -e /d’ -e ‘\|$NAME|’ `

  • [...] ?????Linux: Monitor a Service with a Watchdog Script??watchdog????shell???????? ????????ps, grep?????????Supervise??????????????? [...]

  • Ahmed Shabana

    you should replace all the system commands like PS=/path/to/system/exec
    with PS=`which ps`
    they will make it more portable

  • For the record, this script started sending notices several times a day recently. As a result, it was quickly discovered that a RAM module in the server was bad and causing some services to crash. As the web and e-mail were working “okay”, no one would have noticed this problem for a few days otherwise.

  • KillerDAN

    Exit Code is always 0. Script does nothing.

    • Well, it should do nothing if your process is running. Otherwise you should see your “dead” process get started and output mailed to your e-mail addresses you placed in the script. This script as shown should always exit 0. Otherwise you have an error in this script. If you want a different exit code then add that with “exit” in the case/esac section.

  • Lars

    Hi,

    I thought I could put this to well use on my Raspberry pi… everything works well up to the point of intergrating the * * * * * /etc/bin/watchdog into the crontab. It is not being executed… script itself works fine.

    Thoughts?

Leave a Reply

  

  

  

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Follow the directions below to post a comment if you are human. After 3 failed tries reload the page to start with new images.