Weblog

What about Apache to Mongrel for Rails applications?

Just earlier I compared our lighttpd and litespeed serving up one of our Rails applications by proxying to mongrel

So what about Apache 2 using the prefork MPM?

This is our vanilla (and fat) Apache 2.0.x install.

Server Hostname: mongrel-ap.textdrive.com
Server Port: 80
Concurrency Level: 100
Complete requests: 10000
Requests per second: 431.57 [#/sec] (mean)

By fat, I mean this thing does mod_security, php, fcgi, DAV, SVN and has other authentication modules for MySQL and IMAP.

Let’s turn some stuff off

Let’s do a couple of things: let’s turn off mod_security as an input filter and let’s set KeepAlive Off

Server Hostname: mongrel-ap.textdrive.com
Server Port: 80
Concurrency Level: 100
Complete requests: 10000
Requests per second: 904.47 [#/sec] (mean)

Let’s start up more servers since we’re hitting it with a concurrency of 100

Was


<IfModule prefork.c>
StartServers 30
MinSpareServers 30
MaxSpareServers 55
ServerLimit 600
MaxClients 400
MaxRequestsPerChild 2500
ListenBacklog 1000
</IfModule>

Now


<IfModule prefork.c>
StartServers 100
MinSpareServers 10
MaxSpareServers 10
ServerLimit 10000
MaxClients 10000
MaxRequestsPerChild 2500
ListenBacklog 1000
</IfModule>

Server Hostname: mongrel-ap.textdrive.com
Server Port: 80
Concurrency Level: 100
Complete requests: 10000
Requests per second: 882.98 [#/sec] (mean)

Not too much of a change.

Let’s add some stuff

EnableSendfile On
EnableMMAP On

Server Hostname: mongrel-ap.textdrive.com
Server Port: 80
Concurrency Level: 100
Complete requests: 10000
Requests per second: 982.27 [#/sec] (mean)

OK that’s the best so far.

Hmm … what else can I do with this (without turning off our SVN, DAV etc)

Ah, I’ll turn off .htaccess and input filtering by mod_deflate as well

Server Hostname: mongrel-ap.textdrive.com
Server Port: 80
Concurrency Level: 100
Complete requests: 10000
Requests per second: 1070.59 [#/sec] (mean)

But I’ll turn those back on, no biggie.

The biggest hit actually comes from mod_security, restoring it after all the other changes pushed it back down to about 500 requests/second.

And this is very respectable by the way considering that just did a little better than lighttpd and litespeed for proxying to mongrel.

Joyent.com is a little PHP (mod_php) site with no database backend and under the same conditions it does

Requests per second: 588.36 [#/sec] (mean)

And the other and last point is that this is still a slow Apache because it’s prefork on FreeBSD and still doing rewrites, DAV, SVN and has several third party auth modules. I’m certain that it’s performance can be taken up several fold with a slim and trim proxy-only Apache 2.2 instance using the worker or event MPM.

There is also now a load-balancing extension for mod_proxy in Apache 2.2

There also wasn’t an appreciable load difference when using Apache and it’s nice to see the same mongrel processes stay at ~28MB (I used the same exact mongrel process with 4 threads for apache, lighttpd and litespeed).

·:· Posted 17 April 2006, 02:27 by Jason Hoffman to Rails  |  Got something to say? [3]

Lighttpd versus Litespeed with Mongrel as a backend for Rails applications

Done like always: client -> server over a gigabit connection, and both lighttpd and litespeed are proxying back to the same set of mongrel processes (in other words everything is exactly matched including file size and transfer rates).

This set is on FreeBSD with a ruby compile that has no pthreads.

Litespeed Enterprise 2.1.14 -> Mongrel 0.3.12.4

Concurrency Level: 100
Requests per second: 1008.04 [#/sec] (mean)

Lighttpd 1.4.10 -> Mongrel 0.3.12.4

Concurrency Level: 100
Requests per second: 1010.36 [#/sec] (mean)

This is forcing everything to be served through mongrel, and was perfectly comparable with fastcgi’s performance. Having the file be cached to the filesystem by rails and served as a static file by litespeed or via lighttpd’s mod_cml actually boosts it another 3-4 fold.

So …?

  1. Mongrel is good stuff.
  2. Lighttpd or litespeed for the simple purposes of being a load-balancing proxy capable of virtual hosting? Doesn’t matter, they’re comparable.

BTW, if you want to click around to the public sites, http://mongrel.textdrive.com and http://mongrel-lite.textdrive.com

·:· Posted 16 April 2006, 23:16 by Jason Hoffman to Rails  |  Got something to say? [1]

Two simple tips for freezing your rails

On the forum today Deirdre and Jordan reminded everyone of two easy and up to date tips for freezin’ your Rails.

Deirdre’s (from inside a working copy of your rails app)


$ export EDITOR=vi (or your editor of choice)
$ svn propedit svn:externals vendor

The contents should be


rails http://dev.rubyonrails.org/svn/rails/tags/rel_1-0-0/

Then commit it!


$ svn commit -m "Freezing to 1.0"

And you can go off to where it’s running from and svn update it.

Jordan’s (to freeze to 1.0)


$ rake freeze_edge REVISION=3303

·:· Posted 29 March 2006, 04:12 by Jason Hoffman to Rails  |  Got something to say? [6]

Playing with Lighttpd, Litespeed, FCGI and Mongrel

We’ve been big pushers (not advocates but downright pushers) of alternative web servers, primarily ones that at least support fastcgi and have proxying/load balancing capabilities. We do this despite being big fans and long time users of Apache (SVN and DAV in Apache is primarily what we still use it for. Of course the WebDAV implementation in the Joyent application is entirely in ruby but that’s a discussion for another time).

There’s lighttpd (endorsed and supported here), litespeed. We’ve been using litespeed since we got all the little things “fixed” in it to support ruby-fcgi back in October 2005. Fixed is in quotes because it’s not litespeed’s fault, the problem with it was that litespeed (unlike Apache or lighttpd) supports persistent fastcgi. Now I know what you’re asking, “I thought that fastcgi was always persistent?” Yes that’s correct, but the connection between the web server and the fastcgi processes are not. It’s the same as the difference between pconnect and connect in PHP with MySQl (which is a persistent server process). Litespeed happens to be the only one that also does persistent connections between itself and the fastcgi processes (Can you say “firehose”?). The easier solution was to introduce an option in litespeed to turn persistent fastcgi connections off (we’ll fix the ruby fastcgi connector later).

Then there’s mongrel, Zed Shaw’s ruby (and there rails) as HTTP.

So I thought I’d take two Dell 2850s, 4GB RAM, Dual 3.6 Ghz, 73GB SCSI drives, over a trunked 2x gigabit connection.

We’ll just look at Lighttpd with FCGI, Litespeed with FCGI and Lighttpd with Mongrel (Litespeed + Mongrel was comparable). I also won’t take it across the entire concurrency spectrum but will save that for another time.

Lighttpd-fcgi (mod_fastcgi)

ApacheBench


$ ab -n 1000 -c 10 http://private.textdrive.com/
This is ApacheBench, Version 2.0.41-dev <$Revision: 1.121.2.12 $> apache-2.0
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 1998-2002 The Apache Software Foundation, http://www.apache.org/

Benchmarking private.textdrive.com (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Finished 1000 requests

Server Software: lighttpd/1.4.9
Server Hostname: private.textdrive.com
Server Port: 80

Document Path: /
Document Length: 11047 bytes

Concurrency Level: 10
Time taken for tests: 6.318620 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 11228000 bytes
HTML transferred: 11047000 bytes
Requests per second: 158.26 [#/sec] (mean)
Time per request: 63.186 [ms] (mean)
Time per request: 6.319 [ms] (mean, across all concurrent requests)
Transfer rate: 1735.19 [Kbytes/sec] received

Connection Times (ms)

min mean[+/-sd] median max
Connect: 0 0 1.5 1 14
Processing: 15 61 32.0 55 356
Waiting: 14 58 31.9 52 353
Total: 15 62 32.1 56 356

Percentage of the requests served within a certain time (ms)

50% 56

66% 67

75% 74

80% 80

90% 98

95% 116

98% 133

99% 196

100% 356 (longest request)

Siege (light)


Lifting the server siege... done. Transactions: 2834 hits
Availability: 100.00 %
Elapsed time: 60.19 secs
Data transferred: 29.86 MB
Response time: 0.03 secs
Transaction rate: 47.09 trans/sec
Throughput: 0.50 MB/sec
Concurrency: 1.43
Successful transactions: 2834
Failed transactions: 0
Longest transaction: 0.15
Shortest transaction: 0.01

Lighttpd-mongrel (via mod_proxy)

ApacheBench


$ ab -n 1000 -c 10 http://private.mongrel.textdrive.com/
This is ApacheBench, Version 2.0.41-dev <$Revision: 1.121.2.12 $> apache-2.0
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 1998-2002 The Apache Software Foundation, http://www.apache.org/

Benchmarking private.mongrel.textdrive.com (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Finished 1000 requests

Server Software: lighttpd/1.4.9
Server Hostname: private.mongrel.textdrive.com
Server Port: 80

Document Path: /
Document Length: 11047 bytes

Concurrency Level: 10
Time taken for tests: 6.760181 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 11228000 bytes
HTML transferred: 11047000 bytes
Requests per second: 147.93 [#/sec] (mean)
Time per request: 67.602 [ms] (mean)
Time per request: 6.760 [ms] (mean, across all concurrent requests)
Transfer rate: 1621.85 [Kbytes/sec] received

Connection Times (ms)

min mean[+/-sd] median max
Connect: 0 0 1.5 0 9
Processing: 19 65 33.9 57 398
Waiting: 15 62 33.3 54 394
Total: 19 66 33.9 58 398

Percentage of the requests served within a certain time (ms)

50% 58

66% 71

75% 79

80% 85

90% 106

95% 124

98% 157

99% 177

100% 398 (longest request)

Siege


Lifting the server siege... done. Transactions: 2839 hits
Availability: 100.00 %
Elapsed time: 60.30 secs
Data transferred: 29.91 MB
Response time: 0.03 secs
Transaction rate: 47.08 trans/sec
Throughput: 0.50 MB/sec
Concurrency: 1.41
Successful transactions: 2839
Failed transactions: 0
Longest transaction: 0.13
Shortest transaction: 0.01

Litespeed-fcgi (non-persistent)

ApacheBench


$ ab -n 1000 -c 10 http://private.lite.textdrive.com/
This is ApacheBench, Version 2.0.41-dev <$Revision: 1.121.2.12 $> apache-2.0
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 1998-2002 The Apache Software Foundation, http://www.apache.org/

Benchmarking private.lite.textdrive.com (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Finished 1000 requests

Server Software: LiteSpeed
Server Hostname: private.lite.textdrive.com
Server Port: 80

Document Path: /
Document Length: 11047 bytes

Concurrency Level: 10
Time taken for tests: 5.987303 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 11193896 bytes
HTML transferred: 11049752 bytes
Requests per second: 167.02 [#/sec] (mean)
Time per request: 59.873 [ms] (mean)
Time per request: 5.987 [ms] (mean, across all concurrent requests)
Transfer rate: 1825.70 [Kbytes/sec] received

Connection Times (ms)

min mean[+/-sd] median max
Connect: 0 0 1.3 0 6
Processing: 20 58 23.2 54 164
Waiting: 14 51 20.5 47 141
Total: 22 59 23.2 55 164

Percentage of the requests served within a certain time (ms)

50% 55

66% 63

75% 68

80% 74

90% 91

95% 105

98% 126

99% 137

100% 164 (longest request)

Mild Siege


Lifting the server siege... done. Transactions: 3051 hits
Availability: 100.00 %
Elapsed time: 60.21 secs
Data transferred: 32.14 MB
Response time: 0.03 secs
Transaction rate: 50.67 trans/sec
Throughput: 0.53 MB/sec
Concurrency: 1.46
Successful transactions: 3051
Failed transactions: 0
Longest transaction: 0.16
Shortest transaction: 0.01

Final thoughts

Then I nuked them with a high concurrency and alternating bursts of requests (using a very fun config for …) and while litespeed did in fact crank out at a high concurrency, lighttpd 1.4.9 went into a “D state” and it’s processes become unkillable, so the server machine had to be rebooted. It’s likely something that’s fixed in 1.4.10 (which at first release didn’t compile on FreeBSD or Solaris and I didn’t fix that compile).

Mongrel as a proxy performs just as well as fcgi does, either with lighttpd or litespeed, and could easily become an Adios FCGI thing.

Litespeed didn’t make me have to reboot a server (but it was the paid enterprise version) and combined with a few other features that I’ll cover later did make me do


$ curl -I http://textdrive.com/
HTTP/1.1 200 OK
Server: LiteSpeed
Date: Sat, 18 Feb 2006 20:01:53 GMT
Connection: close
Content-Type: text/html
Cache-Control: no-cache

You’ll also notice that Wordpress.com uses Litespeed

And dude, static files. I love ‘em


Server Software: LiteSpeed
Server Hostname: private.lite.textdrive.com
Server Port: 80

Document Path: /images/hero/2006-02.jpg
Document Length: 12973 bytes

Concurrency Level: 200
Time taken for tests: 6.289341 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 132886064 bytes
HTML transferred: 129753247 bytes
Requests per second: 1589.99 [#/sec] (mean)
Time per request: 125.787 [ms] (mean)
Time per request: 0.629 [ms] (mean, across all concurrent requests)
Transfer rate: 20633.48 [Kbytes/sec] received

In the next post, I’ll show you how Litespeed on Solaris 10 using both devpoll and sendfile() on a ZFS file system absolutely cranks.

Cranks

·:· Posted 18 February 2006, 20:03 by Jason Hoffman to Web servers  |  Comments? [2]

On Rails sessions

Like everything else in Rails, sessions are pretty easy to use. Unfortunately, the default session configuration isn’t particular well suited for a shared host.

You see, by default CGI::Session::PStore (the default session store in Rails) will use the operating system’s temp dir, which on nix-like systems means /tmp. Which is fine, the /tmp dir well suited for small, temporary files that will get cleaned out, which is also why the /tmp partition is traditionally being kept small (it’s actually quite big here at TextDrive compared to traditions).

But here’s the problem, Rails loves to create new session files, one particular case where you’ll end up with an abnormal number of “useless” sessions is RSS/Atom feeds. See, most feedreaders just request a new session every time they hit a feed, let’s say your users are nice and only refreshes their readers once every hour, that’s 24 new sessions per day per user. Let’s say you have a 100 users loyally refreshing their feeds every hour, every day; that’s 2400 new sessions. If you suddenly get popular and get a whooping 250 loyal visitors a day, then that’s 6000 new sessions. And that’s only for the feedreaders, your normal visitors will create a good amount of new sessions as well (even though browsers are a bit smarter about being able to “reuse” a previously created session, thanks to cookies).
Now all that activity will end up with a decent amount of session files, which is fine. The problem is that your site is hosted on a shared server, where you’ll have plenty of neighbors doing the same thing, and /tmp will quickly end up looking like this:


ls -l /tmp/ | grep -c ruby
79962

That’s eighty-thousand ruby sessions, let’s say all those users are nice and don’t keep anything big in their sessions, so they’re about four kilobytes each, that’s still about 320 megabytes worth of session files. Enough to fill up most /tmp partitions based on traditional partition schemes. And that is bad, since things like MySQL stores its socket file in there.

OK, enough 3rd grade mathematics. Luckily Rails has alternatives, CGI::Session::PStore allows you to set an alternative “tmpdir” by sending it the ‘tmpdir’ option key, within Rails you could do it like this, in your environment.rb:
ActionController::CgiRequest::DEFAULT_SESSION_OPTIONS.update(:tmpdir => File.join(RAILS_ROOT, '/tmp'))
which would set it to /path/to/your/railsapp/tmp, very easy and unobtrusive to the real /tmp partition.

That’s one way, even better would be to use ActiveRecordStore to store the sessions in your database. Not only does it not involve files on the filesystem, it’ll also make your application faster. Just look and uncomment this line in your environment.rb:
config.action_controller.session_store = :active_record_store
and run rake create_sessions_table to create the sessions table.

Once upon a time Scott Barron wrote this little thing called Session Container Performance in Ruby on Rails, and if you look at the graphs you’ll see how performance is slowly degrading over time using the default PStore, while staying constant using ActiveRecordStore. Scott wrote that thing late 2004, so the numbers may or may not be the same, but the very same principles and conclusions still apply.

·:· Posted 23 January 2006, 10:11 by Johan Sørensen to Upon our Recommendation  |  Got sessions? [4]

← Previously