• home
• about
• hosting
• faqs
• contact
• weblog
• forum
• help

Optimizing Rails Resource Usage

Hi, my name is Julik and I’m one of the new guest writers on the TextDrive weblog. Jason has given me the privilege to talk a little about performance-tuning Rails applications for deployment on TextDrive or anywhere.

I’ve been doing Rails development since April 2005 and have deployed 3 applications already, all for client use. Since then I have encountered quite some problems which are not covered well on the Rails wiki, partly because they pertain to running Rails on shared hosting accounts.

Rails is nuclear stuff, it allows you to do wonders in minutes but you have to be careful with it. The amount of magic contained in Rails imposes quite an overhead – that you, as a developer, are obliged to manage. These things not only make your application faster, scalable and tolerant, but also bearable for people living on the same web and application servers. Failures to do this in a shared heteregenous environment certifably leads to memory leaks and hard crashes. These tips are good standard procedures for a Rails application at any level (even if you are on a dedicated cluster setup).

So here it goes, a list of 10 things I recommend to do to streamline your application and make it more bearable for others.

In short,

1. Minimize the amount of FCGI listeners
2. Use caching
3. NEVER run “development” on FastCGI for more than 1-2 hours
4. Observe your memory consumption
5. Rotate your logs
6. Write and run unit tests
7. Check for memory leaks when you are developing
8. Be careful with iterations
9. Watch the Rails TRAC for bug reports
10. Be vigilant when restarting your server

And in detail

(1) Minimize the amount of FCGI listeners

As already stated, Rails is not PHP. It is persistent, meaning that it “hangs there” without connections and still consumes memory when there are no clients connected. You read it right – once you start a Rails app, it remains running forever.

Even though as something Jason says when working with the Litespeed server it’s developer, George Want, found that the FCGI interface for Ruby(OnRails) does not like trully persistent FCGI connections (it’s a design issue), and when running as a truly persistant FCGI there are serious file descriptor leaks and performance issues. However, it’s not something that has come up yet with Apache and lighttpd becaue they generally don’t take advantage of persistent FCGI connections (George was nice enough to make it an option in LiteSpeed 2.1.4 to turn off persistent connections. But that’s just an aside and something a bit more illustrative.

Normally, every dispatcher handles requests to your application in a linear queue (dispatchers are not threaded). It means, that when a user is requesting a page that makes 3 requests to the Rails app you made (for instance, for a dynamic image, an AJAX request and the page itself) it is going to invoke 3 runs of Rails processing cycle, in a queue. If you have more than 1 dispatcher the requests are going to be distributed between dispatcher processes by the web server. However, for an application undergoing development, 1 dispatcher is enough. 2 is quite some. If you need more dispatchers you should think about performance optimization. Remember as we said before, A List Apart runs on 4 FCGI dispatchers and is screaming fast. So putting in more dispatchers wil most likely not make your application faster, because each individual request is going to take the same amount of time – it will only allow you to scale to more users (which, for a little content site, is most likely not necessary). Remember alistapart.com, load of 0.01, 4 instances of dispatch.fcgi

You should also know that your application will consume 1 persistent database connection per dispatcher and won’t release it until the dispatcher is stopped, meaning that when you start 1+N dispatchers you consume +N more database connections (and this can make MySQL a tad slower for others). You can tune it so that it closes the connection when the controller is left (in an “after_filter” for instance) if you still want more dispatchers.

(2) Use caching

If you have a content site (not an application), take care to cache all the relevant pages, fragments and actions. It is very easy to do and very easy to tune the expirations, but you will need a line in your lighttpd configuration so that page caching still works.

(3) NEVER run “development” on FastCGI for more than 1-2 hours

The deal is simple. Under FastCGI, Rails ALWAYS leaks memory in “development” environment because of the class reloading.

There is no solution for this.

Granted, you can start your app in “development” for short debugging but pleae take care to turn it off (shutdown the server and kill off all of the FCGIs) after you’re done. A Rails app running in development for a day will consume as much memory as you have and will ask for more. One of my Rails apps grows up for 1.5 Mb RAM on every request, and Jason said that the large single application they recently saw was 4 processes from one guy’s application consuming 2.1 GIGABYTES of RAM. I personally had to restart my laptop a couple of times because of FCGI run amok.

(4) Observe your memory consumption

A Rails application usually consumes anywhere from 20 to 50-70 Mb per FastCGI dispatcher. If a Rails application is tuned properly this amount should stabilize after about 20 requests and stay the same. If it continues to bubble it means that your application leaks memory and is not production-ready.

Jason gave me an example of week old process from Strongspace

Yes, that’s right. Those processes are 5 days old, exchange TBs of data, they are all about ~30-35 MBs each.

My own apps consume:
1. Address book app (in development) – 31 MB per dispatcher
2. Radar (job posting) – 32 MB per dispatcher
3. A client’s CMS site for a design studio – 70 Mb (but it uses RMagick quite often).

None of the 3 go above these numbers, some are running for weeks. Persistently. 24/7.

(5) Rotate your logs

If your app is hit often, take time to cleanup your “log” directory from old logs. And you have some shrink-wrapped goodness to do it for you.

(6) Write and run unit tests

They are smart stuff and they are so baked into Rails (and they’re one of the things that makes Rails great) that it’s foolish not to spend a lot of time writing and running unit tests.

(7) Check for memory leaks when you are developing

This is simple.

First, switch your development environment (on your local machine, my god don’t do this on your TextDrive or any other hosting account) to

Then start your app and just hammer it’s most fragile (and heavy) controller actions with ab (apache benchmark), so that it rolls up to 400-500 requests. After that, start “top” and see where you sit. A normal Rails app (in my humble experience) never goes over 90 MB per dispatcher, and usually sits in the span between 20 MB and 70MB. Remember, that this RAM is shared by others and when you eat it away others will start to swap, and it hurts. If you’re not careful you can eat up all the server’s memory in an hour (or in a day – depending of how many hits your Rails app is serving).

And, and when you combine the resulting sluggishness with content-based checks, lighttpd restarts that don’t kill dispatch.fcgi and leaking memory, then you have a real server crasher on your hands.

(8) Be careful with iterations

ActiveRecord is smart, but don’t do

end


Remember that SQL is an ultimate and highly effective weapon, and ActiveRecord is very talky with the database. The less requests to the database the faster it works.

I don’t know if it relevant but for me using reject with AR objects has led to enormous memory consumption (looks like the objects being reject’ed from the enumerable were not garbage collected)

#or this
Post.find_all_by_comments_closed(false)


and implement relevant “finder” methods within your models themselves.

(9) Watch the Rails TRAC for bug reports

If you’re developing everyday, if you have an application that’s running everyday, then you should check it everyday

Since I have started I seen about 25 bugs pop up which were directly related to my own cases.

(10) Be vigilant when restarting your server

FastCGI is a bugger when restarts are needed. Currently, Rails dispatchers *won’t die* when you stop your lighttpd instance. It means that you have to do it in the following order when you need to restart:

Then start “top” or “ps ax” and look if the dispatchers are still running. Only when you are absolutely sure that there are no dispatchers hanging around anymore you can bring your lighttpd back up.

Using deadalus for this has led to immense overspawning of processes, and I hope Jason, Jan and Marten can invent something that would manage these “forceful” restarts for us. In the meantime, you have to take care of this yourself.

Thank you for your time, and I hope that helps some avoid many of the mistakes I have made myself and also help to decrease resource abuse. Recent server outages are, as it turns out, related to the fact that all of the above was not being done.

·:· Posted 25 October 2005, 16:15 by Julik to Rails  |  

1. you got some sup tags in the cache rewrite rule.
   
   
   
   — cch    25 October 2005, 01:36    #
2. maybe you could write more about the log rotation. mention some log rotation tools. I, for one, just run rake clear_logs daily via cron.
   
   — cch    25 October 2005, 01:41    #
3. Yes the sups should be
   
   ^
   
   — jason Hoffman    25 October 2005, 03:23    #
4. ————————-
   
   (5) Modify your dispatcher file
   
   garbage collection (memory cleanup) will be left at the discretion of Ruby, which most probably means “once a month”.
   
   Don’t do this.
   ———————————
   Does ‘don’t do this’ mean ‘ignore this bit’, or ‘don’t let ruby manage gc’ as I read it?
   
   Just thought I’d ask – I just made the change and davie has suddenly got very sluggish…. O_o
   
   — Dick Davies    25 October 2005, 09:31    #
5. Leave the default rails setting in dispatch.fcgi. Julik is is mistaken about this issue, as I have explained on my blog.
   
   — Dee Zsombor    25 October 2005, 10:25    #
6. I will try to integrate all the corrections into the post tonight, thanks to all who participated. Changing the dispatcher fixed one of my most terrible memory leaks on one of my apps, that’s why I’ve put it here. I’ll also contact Skaes and ask him about dispatcher configs in detail. Stay tuned!
   
   — Julik    25 October 2005, 13:31    #
7. Regarding the process! invocation from dispatch.fcgi:
   
   I tried both with a default (nil) period as well as a period of “10” and in both cases my dispatcher process(es) grow in memory consumption without bound, until the machine begins to swap.
   
   What gives? If I have a leak in my app, what are the possible causes? In the Rails model, is the controller instantiated once per request? If not, then I guess I might have instance variables that stick around with stale references following some of my action requests.
   
   Can someone clear this up so I don’t have to dig through Rails code?
   
   — Bosko    25 October 2005, 21:07    #
8. There is a script which allows you to look into your object heap after every request. You can then quickly see how the object counts are doubling and which objects are not cleaned up. I will try to lookup the script – in the meantime look for it (surprise) in Rails TRAC.
   
   — Julik    26 October 2005, 00:36    #
9. Step 5 has something wrong with it. If I leave dispatch.fcgi alone, Typo stops at 53MB of memory, even after many requests. If I follow step 5, however, the ruby fcgi process takes up more than 100MB on the first request, and gets killed because it’s over the per-process limit for ram on TextDrive. My memory usage was supposed to go down, not up. What’s the deal?
   
   — Andre    26 October 2005, 01:28    #
10. Ok, so I’m finally running EdgeRails out of vendor and I’ve been playing with this for the last few hours, including running with MemoryProfile.rb (http://rubyforge.org/snippet/detail.php?type=snippet&id=70) and analysing the output.
    
    Both Andre and Dee Zsombor are correct here: you generally do not want to change that process! line in dispatcher.fcgi. Ruby does a decent job collecting garbage on its own and by setting a period for GC, you effectively prevent it from doing so, making things a whole lot worse in most cases.
    
    If you observe otherwise, I suggest that you first make absolutely sure that you are in fact running in production mode. With 0.14.1 some things around config/ have changed and so this might not be immediately obvious, particularly if you were forcing production mode via some hack in environment.rb and friends before.
    
    — Bosko    26 October 2005, 03:20    #
11. The collection period in the dispatcher was inroduced to increase throughput for apps running on a dedicated server. Not for forcing GC in order to cut down on memory usage.
    
    And no, Ruby does not do a decent job on GC. GC performance is largely a matter of good luck, depending on the ratio of code size/allocated heap slots.
    
    Adding code may either speed GC up or slow it down, depending on whether GC decides to allocate additional heap slots. And the logic for this sucks majorly.
    
    However, in a shared environment you should leave the GC frequency parameter set to nil.
    
    — Stefan    26 October 2005, 04:56    #
12. Great advise! I’ve been struggling with these issues that last few days and should have went looking before figuring them out.
    
    — Chris Nolan.ca    26 October 2005, 09:16    #
13. Stefan: in my experience, setting the collection period even on a dedicated server can be bad.
    
    The fact is, once you provide a period argument for process!, Rails starts periodically invoking the GC. That means that unless you hit up one particular instance of the dispatcher with N requests, the GC does not run in between. This is evidenced by the run_gc! method in fcgi_handler.rb which does a GC_enable, starts the dispatcher, but then does a GC.disable, disabling it until the request count satisfies the period again.
    
    Whether Ruby does a good enough job on its own invoking the GC is another discussion altogether, but I, like Dee Zsombor, believe that in the majority of cases, not setting a period for GC invocations is preferable.
    
    If you are observing a per-request memory runoff for your dispatchers, you’re either running in ‘development’ mode and leaking memory on class reloads, or you’re leaking memory through an error in your use of global data.
    
    — Bosko    26 October 2005, 13:45    #
14. While I agree that not setting it will reduce the amount of memory used, on a dedicated server, that has RAM to spare, it can greatly enhance throughput.
    
    It all depends on the app of course. If you have requests of very different memory requirements it woill not be a good idea.
    
    Better use my GC patch
    
    — Stefan    26 October 2005, 17:26    #
15. Please see Safe Lighttpd Restarting Script and Daedalus Config for information about restarting lighttpd gracefully and not leaving zombie dispatchers, as well as a server-friendly daedalus.conf.
    
    — Gabe da Silveira    27 October 2005, 03:41    #
16. Everyone use fastcgi for RoR but there is also mod_ruby option…
    
    — Jaroslaw Zabiello    18 November 2005, 18:45    #
17. Of course there is. However I doubt it will be fit for shared environments.
    
    — Julik    25 November 2005, 01:12    #

© Copyright 2004–2005 TextDrive Inc. · TextDrive™ is a trademark of Joyent Inc.
Acceptable Use · Terms of Service · Privacy · Safe Harbor · All prices in US$