ArsDigita Keepalive

by Ben Adida and Philip Greenspun (web interface by Steven Pulito), part of ArsDigita Free Tools

ArsDigita Keepalive is a system that monitors your web services at regular, short intervals, and takes action to resolve problems found. If Keepalive fails to reach a page, depending on how many consecutive previous failures it has seen and the configuration parameters, it will take one of the following actions:

nothing except decrement a counter
send email to a previously defined group of addresses
execute a shell command, presumably one that restarts the stuck service

Keepalive is built using AOLserver (free) and takes advantage of AOLserver's built-in scheduler (like Unix cron but lighter weight) and Tcl API (includes a call to HTTP GET a page from another server). However, unlike most of our AOLserver products, you don't need to install an RDBMS in order to use Keepalive. Web servers generally get stuck because of problems with the RDBMS, so a monitor that depended on an RDBMS would be self-defeating.

Although we generally use Keepalive to monitor AOLserver-based Web services, it will work fine to monitor any HTTP service on a Unix machine.

Installation

Download keepalive.tar.gz (last updated June 12, 2000)
cd /web
gunzip < keepalive.tar.gz | tar xvf - (creates /web/keepalive)
Create an AOLserver whose page root is /web/keepalive/www and whose private Tcl directory is /web/keepalive/tcl
AOLserver 2.3.3
- cp /web/keepalive/modules/nsperm/*.dat /[ns_home]/servers/keepalive/modules/nsperm/
- edit /[ns_home]/servers/keepalive/modules/nsperm/users.dat and change the nsadmin password
AOLserver 3.0
- cd /web/keepalive/modules/nsperm/
- cp passwd perms /[ns_home]/servers/keepalive/modules/nsperm
- edit the passwd file and change nsadmin's password
Make sure that SSL is configured and working properly
Edit /tcl/defs.tcl to set the main Keepalive parameters.
- keepalive_hostname_or_ip: the host name (www.foobar.com) or ip address of the keepalive server
- keepalive_ssl_port: the port number that SSL is listening on
- keepalive_config_file: the full path of the configuration file
- keepalive_email: This must be a valid address otherwise the mail delivery for automatic notifications will fail
- keepalive_timeout: HTTP GET timeout in seconds
- keepalive_debug: 1 or 0, debug mode is more verbose in the error log
- uncomment SSL filter once SSL is up and running
You're done! Start your server and go to the main page to add a web service to monitor

Security

In order to run a secure Keepalive installation it is very important that you run a SSL enabled server. In order to do this you will need to install one of the available nsssl modules and configure it as well as get your signed certificate/key files in place.

WARNING: If an intruder gains unauthorized access to your keepalive server they will be able to execute arbitrary commands by setting the failure action. To restrict the Keepalive pages to SSL make sure that the following line in defs.tcl is uncommented:

ns_register_filter preauth GET /keepalive/* ad_restrict_to_https

Operation

In the past in order to maintain Keepalive it was necessary to edit the configuration file by hand. This is no longer the case as there is now a web interface for editing the configuration file.

Direct your web browser to the host name specified in your AOLserver configuration. You will be redirected to the main Keepalive page. Initially there won't be any servers listed. This can be rectified by clicking on the add a service link. When adding a new service you will have to specify the following:

Service name:
This is important because it will act as a 'primary key'. It must be unique and will be used to identify which service you want to modify/deactivate later on.
Test URL:
This is the URL that Keepalive will request to check the status of the web server it is monitoring. This URL should query the database in order to make sure the database connection is working and return a value based on whether or not the database is available. At ArsDigita we place dbtest.tcl in the /www/SYSTEM directory where /www/ is the web page root.
Expected result:
Keepalive will compare the value that the test URL returns with the expected result to determine if the web server is functioning properly.
Failure action:
This is a shell command that will presumably restart a hung web server. Read the 'Which Shell Command?' section for details. At ArsDigita we use the restart-aolserver script. Note that to get restart-aolserver working on linux the command ps -ef should be replaced with ps auxww
Admin emails:
This is a list of email addresses to notify when there are problems with the web service being monitored. Just enter in as many email addresses as necessary, seperating them with spaces.
Pager emails:
Same as admin emails only the message to be sent is smaller.
Number of retries:
How many times may a web service fail the Keepalive test before Keepalive runs the failure action script, presumably restarted the web server.
Notification threshold:
How many times may a web service fail before Keepalive starts sending out email.
URL of logging script:
Before restarting a web server Keepalive will attempt to retreive this page. The page specified should be a Tcl script that will write to the log file the current status of the broken web server. If the failed web server is not accepting any connections this won't do any good. However in the case that the database has failed this is often a useful debugging tool. At ArsDigita we place log-monitor.tcl in the /www/SYSTEM directory where /www is the web page root.
Active?
It sometimes happens that it is necessary to temporarily stop monitoring a web service, perhaps during scheduled downtime. In this case you could remove the web service from the configuration file but then it would be necessary to re-enter all the information later. Therefore it is possible to 'deactivate' a service. It will still be in the system's config file but it will not be actively monitored.

Also please note that many of the error pages are quite rudimentary, sometimes only displaying what went wrong. In almost all cases simply press the back button in your browser and correct the problem.

Which Shell Command?

You might well ask yourself which shell command will restart a Web server. It depends. In the case of AOLserver, we run the server by inserting a line in /etc/inittab:

nsjw:34:respawn:/home/nsadmin/bin/nsd -i -c /home/nsadmin/nsd.ini

which tells Unix to restart nsd if it should die for any reason. Thus keepalive just needs to kill the existing nsd process. The problem is that Web servers must be owned by root if they are to grab Port 80 and Keepalive can't kill a Web server unless it runs as root (a security risk). The solution at ArsDigita is to build a setuid Perl script that Keepalive can call: restart-aolserver

#!/usr/local/bin/perl

## Restarts an AOLserver. Takes as its only argument the name of the server to kill.

## This is a perl script because it needs to run setuid root, 
## and perl has fewer security gotchas than most shells.


$ENV{'PATH'} = '/sbin:/bin';

# uncomment this stuff if you're at an installation where a server 
# takes a long time to restart or keeps important state

# if (scalar(@ARGV) == 0) {
#     die "Don't run this without any arguments!";
# }

$server = shift;

$< = $>; # set realuid to effective uid (root)

sub getpids {
    ## get the PIDs of all jobdirect servers
    my $ps_output = `/usr/bin/ps -ef`;
    my @pids;
    foreach (split(/\n/, $ps_output)) {
        next unless /^\s*\S+\s+(\d+).*nsd.*$server.ini/;
        push(@pids, $1);
    }
    @pids;
}

@pids = &getpids;
print "Killing ", join(" ", @pids), "\n";
kill 'KILL', @pids;

Upgrading

If you are replacing an older version of Keepalive you avoid having to re-enter the configuration information by following the following steps:

save your existing /[ns_home]/tcl/init.tcl file someplace safe
install the new Keepalive but do not start the server yet
open up a file in your favorite text editor

for every entry in the keepalive_init proc (in init.tcl) create a line in the new config file

Old way (in init.tcl):

lappend keepalive_monitor_list [new_monitor "arsdigita" "http://www.arsdigita.com/SYSTEM/dbtest.tcl" "success" "/usr/local/bin/restart-aolserver arsdigita" [list email_1@arsdigita.com email_2@arsdigita.com] [list your_pager@arsdigita.com] 2 2] http://www.arsdigita.com/SYSTEM/log-monitor.tcl

New way (one line per entry):

arsdigita http://www.arsdigita.com/SYSTEM/dbtest.tcl success {/usr/local/bin/restart-aolserver arsdigita} {email_1@arsdigita.com email_2@arsdigita.com} your_pager@arsdigita.com 2 2 http://www.arsdigita.com/SYSTEM/log-monitor.tcl 1

This new format is simply a tcl list of all the configuration switches
To leave a field blank replace it with empty curly braces {}
Don't forget to add a 1 as the last switch (active_p)
Save your configuration file (same path and name as defined in defs.tcl)
Start your Keepalive AOLserver instance!

Testing

Please perform the following tests to make sure you have a complete working installation of Keepalive:

Browse to the /keepalive/ pages (tests SSL)
Add two new services
Modify the name of one service
Modify anything else of the other
Check for the existence of a backup config file
Make sure debug mode is on
Check error-log for status reports
Deactivate one service
Check the error-log, make sure the deactivated service isn't being monitored
Attempt to remove one of the services from the config file
Break the test URL on the remaining service
Do you get notification emails?
Was the logging script called?
Was the server restarted?
Did the main Keepalive page properly show the status of the failing server?

License

Support and Customization

If you want a extended version of Keepalive or support, you can hire the programmer of your choice to install, maintain, and customize keepalive. ArsDigita offers support as well, but probably not at a price that you'd be happy to pay.

stevenp@arsdigita.com