Monitoring is great. Especially if it works and you don't have to take care of restarting the services yourself. Supervisord can make your life a little bit easier by doing this for you. It can monitor the processes you want on your server and (re)start them for you if they are not started. That is, if you play by its rules.
I was completely new to Supervisor (although I knew what it does) and my goal was to monitor the apache2 process on my server. In doing that I encountered some problems. This blogpost shows the summary of my journey and concludes with a working configuration.
This post is not limited to Apache and holds for webserver processes that fork.
Supervisord is trying to start / restart Apache. However, Apache normally starts a process and forks itself. While the parent process is usually started as root under Unix in order to bind to port 80, the child processes are launched by Apache as a less-privileged user. The User and Group directives are used to set the privileges of the Apache child processes. The child processes must be able to read all the content that will be served, but should have as few privileges beyond that as possible.
Supervisord can't handle this. You'll see this ending up in the supervisor logs:
2014-09-15 19:22:40,597 DEBG fd 7 closed, stopped monitoring <POutputDispatcher at 45711232 for <Subprocess at 45013688 with name webserver in state STARTING> (stdout)> 2014-09-15 19:22:40,597 INFO exited: webserver (exit status 0; not expected) 2014-09-15 19:22:40,597 DEBG received SIGCLD indicating a child quit 2014-09-15 19:22:41,599 INFO gave up: webserver entered FATAL state, too many start retries too quickly
Exit status is 0, which means that the command was executed successfully / the process as stopped successfully (in a proper way). However, as the parent process is running, not another process can be started (as there still is a pid file) and supervisor stops after a number of times resulting in a FATAL state. Note that the FATAL state means that supervisor will NOT be monitoring your processes. Fatal states need to be recovered by hand!!
The solution is to start Apache in the foreground so Supervisor can monitor the process. However, it's not 100% straightforward as there are various commands to start apache.
I've tried to have commands like:
- service apache start
- /etc/init.d/apache2 start
(also with -DFOREGROUND)
These did not work.
I ended up with the command line:
A full working example for a conf file (I called it apache2.conf) could be:
[program:apache2] command=apachectl -DFOREGROUND autostart=true autorestart=true startretries=1 startsecs=1 stderr_logfile=/var/log/apache2/supervisor.error.log stdout_logfile=/var/log/apache2/supervisor.access.log user=root
When using autostart=true, be sure to (re)move /etc/init.d/apache2. This way, when the server (re)starts, apache will be started by supervisord (by /etc/init.d/supervisor) so it has apache under its command. If you don't, there will be no monitoring on your apache processes as /etc/init.d/apache2 doesn't result in apache being started in the foreground.
With all things you do the first time: It's most certainly not first time right, so for sure there are plenty of things that can be improved either in the configuration and the way of thinking. I just didn't want to keep it to myself. Any feedback and/or suggestions are welcome in the comments!