We switched back to Monit after experimenting some with God (oh, what a nightmare). This post is just meant to be a collection of tips for working with Monit. I’ll update it as I discover/remember more.
1. Monit does not honor limits.conf
We have a Solr process with 3 cores. The number of open file descriptors can get pretty big between optimizes, exceeding the default limit of 1024 in Ubuntu. No problem, I’ll just edit /etc/security/limits.conf and reboot! No dice. Looking at /proc/<solr_pid>/limits shows that the changes didn’t take.
At least the solution is easy. Change the Monit start command to execute a bash script instead of starting the Solr process directly. In the bash script, just call ulimit -n 2048 before starting the Solr process.
2. Monit kills your environment
This is actually well known and documented. Again the solution is to have Monit call a bash script and just set the environment there before starting your process.
But if it’s just one or two environments variable you need to set, you can easily do it in the Monit start command using /usr/bin/env .
check process some_process start = "/usr/bin/env RAILS_ENV=production DEBUG=false /path/to/some_executable --arg1 foo -arg2 bar ...