My Rails production setup

I’ve recently moved one of the products I’m maintaining to a new server, because it wasn’t performing as well as it was supposed to. In the process I’ve spent some time tweaking the server and simplifying the setup, and the following is an overview over some of the tools I’ve found most useful.

The application

I thought it might be useful to give you a basic idea of what the application does: The application TidTilMere is a highly specialized administration app for Danish folk high-schools (højskoler). It is build around the idea of having a single place to store information on everyone the school has had contact with, and it therefore spans many different areas:

• Enrollment: …for courses (such as “fall ’09″) and subjects (ex. “painting”)
• Accommodation: Room booking and availability
• Orders: Ordering of brochures and newsletter signup
• Search: Find people based on complex criteria
• Mass-mailing: Send template-based emails to people in a search result
• Mail-merge: Documents can be produced from people/enrollments in a search result, based on uploaded templates in Word and RTF format.
• CRM: A single place to store everyone the school has been in contact with, their contact information, and their relation to the school (including all of the above)

The application is used on a daily basis by the schools staff (1-4 people), and has to be very responsive. It is not likely to win any awards for front-end design, but it gets the job done. If you want to know more (and happen to speak danish), you can visit TidTilMere.net.

With that out of the way, lets get to the setup.

The server

I’ve been hosting the application with HEXONET for the past two years, and have been very happy with them. The reason I went with them, was that they were hosting out of Germany, which meant lower round-trip-times than a server farm in the US (~20ms vs. ~150ms), and we wanted the application to be very responsive.

HEXONET only sell to companies (not individuals), but have a wide range of virtual server offerings. The few times I’ve needed to contact support, they’ve been quick in getting back to me. The old server was a “vServer Platinum Pro”, whereas the new server is a “dServer VNS Premium”, which translates into 28€ for a Debian-etch with 1500MHz CPU, 45GB HD, 1000GB traffic and 1024MB RAM (1536 MB burstable). You can see more options on their feature/price list.

If you need to service customers in the EU, I think HEXONET is a good option.

Phusion Passenger (mod_rails)

A couple of weeks ago i switched all my applications from Apache 2 + Mongrel to Apache 2 + Passenger, and I’m glad I did, because it is so much easier to install and maintain. I haven’t noticed any performance gains/hits, but the application is much easier to deploy and maintain now than it was before, and installation was simpler than getting Mongrel to work with Apache 2 through mod_proxy_balancer.

Passenger will automatically launch additional workers according to load, and will also relieve them of duty when they are no longer needed. A sideeffect of this, is that if no one is accessing your site for a period of 5 minutes (default), all the workers will be killed, and Passenger will have to instantiate a new worker + the rails environment when the next visitor comes by. This can take some time, and proved to be a little annoying for my users, because they were often doing other tasks for 5 minute periods. Fortunately, it is very easy to set up a cron job (or something similar) to poll your application every once in a while, to make sure a least one worker is alway ready:

# m h  dom mon dow   command
*/3 *  * * * wget -O /dev/null http://localhost:80/admin/login --no-check-certificate 2>/dev/null

I highly recommend giving Passenger a try.

Ruby Enterprise Edition (REE)

I use Ruby Enterprise Edition together with Passenger, to improve performance and decrease memory consumption.

Recently though, comparisons here and here indicate that the tides may be changing, and that Ruby 1.9 or JRuby might be faster. For now, I’m sticking with REE, since Passenger doesn’t support JRuby yet, and Ruby 1.8.x is still what the Rails people recommends. I’m keeping my eye out though.

Monit

Monit is a monitoring tool for Unix systems, that can monitor processes, files etc. It can both alert the administrator when something is wrong, and in many cases automatically correct the problem.

I found Monit to be crucial when I had my application on Mongrel, because a Mongrel instance would crash once in a while. Monit would then automatically restart the instance and notify me, without the users experiencing any down-time. Now that Passenger takes care of always having running workers, Monit rarely has to restart anything, but I still count on it for notifying me if anything out of the ordinary happens.

I use monit for monitoring my system for abnormal load, checking that apache, mysql and my mailer deamon are running (and automatically restart them if they are not), and for making sure that I don’t accidentally run out of disk space (this has happened once or twice, and took me longer-than-I-liked to pinpoint). This is an example from my monit configuration file, which also shows some of Monits power:

set daemon  60 # Poll at 1-minute intervals

set alert erik@underbjerg.com

# Check for abnormal load
check system localhost
    if loadavg (1min) > 4 for 3 cycles then alert
    if loadavg (5min) > 3 for 3 cycles then alert
    if memory usage > 90% then alert
    if cpu usage (user) > 70% for 5 cycles then alert
    if cpu usage (system) > 30% for 5 cycles then alert
    if cpu usage (wait) > 20% for 5 cycles then alert

# Check apache2
check process apache with pidfile /var/run/apache2.pid
  start program = "/etc/init.d/apache2 start"
  stop program  = "/etc/init.d/apache2 stop"
  if cpu > 60% for 2 cycles then alert
  if cpu > 80% for 5 cycles then restart
  if totalmem > 200.0 MB for 5 cycles then restart
  if children > 250 then restart
  if loadavg(5min) greater than 10 for 8 cycles then stop
  if failed host localhost port 80 protocol http
     with timeout 15 seconds
     then restart
  if 3 restarts within 5 cycles then timeout
  group server

# Check that mysql is running
check process mysql with pidfile /var/run/mysqld/mysqld.pid
  start program = "/etc/init.d/mysql start"
  stop program = "/etc/init.d/mysql stop"

# Check that the ar_sendmail mailer daemon is running
check process ar_sendmail
  with pidfile /home/erik/apps/hojskole_sys/shared/log/ar_sendmail.pid
  start program = "/etc/init.d/ar_sendmail start" with uid erik and gid erik
  stop program = "/etc/init.d/ar_sendmail stop" with uid erik and gid erik
  if totalmem is greater than 65.0 MB for 2 cycles then restart      # eating up memory?
  if loadavg(5min) greater than 10 for 8 cycles then restart          # bad, bad, bad
  if 20 restarts within 20 cycles then timeout                        # something is wrong, call the sys-admin
  group ar_sendmail

# Check disk space
check device vzfs with path /
   if space usage > 85% then alert
   group server

So if you don’t already have a way to automatically check the health of server, I recommend giving Monit a try. Note that Monit is in no way specific to rails, so you should be able to use it for any kind of Unix server.

New Relic

The New Relic RPM performance monitoring tool is a real gem, that I wish I had when I was trying to optimize performance two years ago. It is an amazing piece of software, that plugs right in to your rails app and gives you detailed breakdowns, for each browser request, of where time is spent. It’s incredibly easy to install, and you can start using it in development mode right away.

It was never been this easy to figure out exactly where to optimize your rails app.

New Relic is free to use in development mode on your local machine, where you can get extremely detailed info, and in production, where the Lite-edition gives you basic metrics for the last 30 minutes. If you want more metrics or longer time periods in production, you will have to buy a licence, which I think is fair, since the Lite edition is already so useful. So far, I’ve gotten tremendous results out of using the Lite edition on my local machine, where I have been able to find exactly the places that were slow, and in production, where I’ve been able to see how the system was performing under real load.

If you are running a Rails application in production, and if you are just the slightest bit interested in how it performs (why wouldn’t you be?), download New Relic right now.

Thanks for reading

I hope you have found it useful. Please feel free to leave any comments, suggestions or questions you might have, and I’ll try to answer them.