Monitor and keep services + proccesses alive with monit

Monit is a simple and lightweight, and really useful tool. I discovered this program some time ago, with the question: "How to monitor a process, and keep it alive always, in the best possible way?".

The answer was Monit. It is a ~500Kb program for UNIX/Linux, which checks up on defined services and processes every 2 minute.

I had setup an ElasticSearch/Logstash/Kibana central server for at customer, with Redis as a buffer, receiving data from the Logstash-agents. The Redis server had a few accidents, with too much memory load, where it unfortunately went down.

I opened the monit-configuration file (monit.conf), and added the following ($ vi /etc/monit.conf), which will create checks for both Redis, MySQL and NGINX:

# Check Redis
check process redis-server with pidfile /var/run/redis/redis.pid
    start program = "/etc/init.d/redis start"
    stop program = "/etc/init.d/redis stop"

# Check MySQL
check process redis-server with pidfile /var/run/mysql/mysql.pid
    start program = "/etc/init.d/mysql start"
    stop program = "/etc/init.d/mysql stop"

# Check NGINX
check process nginx with pidfile /var/run/nginx/nginx.pid
    start program = "/etc/init.d/nginx start"
    stop program = "/etc/init.d/nginx stop"

This is the most simple form of checks you can create in Monit - there is a lot of options, and you can - just to notice - also use it for other monitoring than services and processes.

When your Monit changes are done, you need to reload Monit. This is done with the following line:

$ monit reload

You can eventually test the syntax of the monit configuration first:

$ monit -t

I stopped my Redis server from another SSH instance, and tailed the monit-log:

$ tail -f /var/log/monit

CEST Jul 31 14:07:09] error    : 'redis-server' process is not running
[CEST Jul 31 14:07:09] info     : 'redis-server' trying to restart
[CEST Jul 31 14:07:09] info     : 'redis-server' start: /etc/init.d/redis

It worked as expected!

You can check the monit overall status with:

$ monit status
The Monit daemon 5.1.1 uptime: 9h 43m 

Process 'redis-server'
  status                            running
  monitoring status                 monitored
  pid                               17009
  parent pid                        1
  uptime                            16h 48m 
  children                          1
  memory kilobytes                  936
  memory kilobytes total            936
  memory percent                    0.0%
  memory percent total              0.0%
  cpu percent                       0.1%
  cpu percent total                 0.1%
  data collected                    Fri Aug  1 08:02:27 2014

System 'hostname.host.net'
  status                            running
  monitoring status                 monitored
  load average                      [0.13] [0.08] [0.07]
  cpu                               4.7%us 1.7%sy 3.1%wa
  memory usage                      714184 kB [70.3%]
  data collected                    Fri Aug  1 08:02:27 2014

The web interface

If the monit status, and the other CLI tools are not enough for you, then Monit comes with a simple user interface.

If you want to enable this, uncomment the following lines in monit.conf:

set httpd port 2812 and
use address localhost  # only accept connection from localhost
allow localhost        # allow localhost to connect to the server and
allow admin:monit      # require user 'admin' with password 'monit'
allow @monit           # allow users of group 'monit' to connect (rw)
allow @users readonly  # allow users of group 'users' to connect readonly

To activate, reload monit:

$ monit reload

Now monit will have http running on port 2812. I proxy it to port 80 with NGINX. In my default NGINX configuration i just added the following lines to make it work (on /monit/):

$ cat /etc/nginx/conf.d/default.conf

location /monit {
    rewrite ^/monit/(.*) /$1 break;
    proxy_pass http://127.0.0.1:2812;
}

Restart NGINX:

$ /etc/init.d/nginx restart

You must use admin/monit as basic http authentication, if you did not edit the lines we just uncommented in the monit.conf.

Have fun :-)

See more on the Monit website:
http://mmonit.com/monit/

Menu

Monitor and keep services + proccesses alive with monit - lightweight, and great.

The web interface