Apache is the most popular web server with over 50% of servers using it to serve their web sites. We currently use it to serve a web site we manage for a client. This website was having issues where an http get request was being generated every 6 seconds and causing the server to spawn child process like crazy until it eventually become unusable. The issue was causing some problems for the end users and was becoming a problem that needed fixed.
We were unable to find a quick solution to the problem so we had to create a short term fix that would allow us to diagnose the problem while trying to keep as much uptime as possible. As the server was spawning child threads until it would stop responding, so we created a script that would measure the amount of processes owned by Apache and then restart the server if a certain limit was reached. This would allow the server to keep working with minimal downtime while we looked into what was causing all the requests.
The first thing we wanted to do with this script was to monitor the Apache processes. We did this by using a crontab entry to run a script once every 5 minutes. Then in the script we get a list of all processes that are created by the server and get a count. If the count is larger than the set limit we would stop the server, kill all server processes and then start the server again.
We also wanted to add an email to notify us of the event and pass along some data from the logs. The script below is what we created to monitor the server and email the logs when the clients reached max limit.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 |
MAILDIR=/tmp/apache-monitor EMAIL=test@example.com mkdir -p $MAILDIR if [[( $(ps aux | grep apache | wc -l) -ge 200)]]; then # the server has high amount of requests touch ~/apache-highload # check file(if apase-ok then this is first down msg) if [[ -f ~/apase-ok ]] ; then # create email date >> $MAILDIR/mail echo >> $MAILDIR/mail echo "Logs..." >> $MAILDIR/mail echo "access_log" >> $MAILDIR/mail tail -n 30 /var/log/httpd/access_log >> $MAILDIR/mail echo >> $MAILDIR/mail echo "site_access.log" >> $MAILDIR/mail tail -n 30 /var/log/httpd/site_access.log >> $MAILDIR/mail echo >> $MAILDIR/mail echo "error_log" >> $MAILDIR/mail tail -n 30 /var/log/httpd/error_log >> $MAILDIR/mail echo >> $MAILDIR/mail echo "site_error.log" >> $MAILDIR/mail tail -n 30 /var/log/httpd/site_error.log >> $MAILDIR/mail echo >> $MAILDIR/mail # kill server echo "Restarting server...." >> $MAILDIR/mail echo >> $MAILDIR/mail /etc/init.d/httpd stop >> $MAILDIR/mail 2>&1 killall -9 apache >> $MAILDIR/mail 2>&1 /etc/init.d/httpd start >> $MAILDIR/mail 2>&1 # sendmail mail -s "Server Monitor: Exceeded 200 connections" $EMAIL < $MAILDIR/mail # remove first error flag file rm ~/apache-ok fi else # the server is under normal load touch ~/apase-ok fi |