Weblog

I made some other web server points over at Joyeur

It’s here and it’s primarily about putting a web server in the context of the network stack and connection (with a hat tip to an old post from Querna that’s been sitting in the brain for nearly a year).

·:· Posted 17 April 2006, 23:37 by Jason Hoffman to Web servers  |  Got something to say?

Beware the trailing slash in Apache's proxy balancer

This is correct


ProxyPass / balancer://app
<Proxy balancer://app>
BalancerMember http://10.0.0.166:8181
BalancerMember http://10.0.0.166:8182
BalancerMember http://10.0.0.166:8183
BalancerMember http://10.0.0.166:8184
BalancerMember http://10.0.0.166:8185
BalancerMember http://10.0.0.166:8186
BalancerMember http://10.0.0.166:8187
BalancerMember http://10.0.0.166:8188
BalancerMember http://10.0.0.166:8189
BalancerMember http://10.0.0.166:8190
</Proxy>

This is not


ProxyPass / balancer://app
<Proxy balancer://app>
BalancerMember http://10.0.0.166:8181/
BalancerMember http://10.0.0.166:8182/
BalancerMember http://10.0.0.166:8183/
BalancerMember http://10.0.0.166:8184/
BalancerMember http://10.0.0.166:8185/
BalancerMember http://10.0.0.166:8186/
BalancerMember http://10.0.0.166:8187/
BalancerMember http://10.0.0.166:8188/
BalancerMember http://10.0.0.166:8189/
BalancerMember http://10.0.0.166:8190/
</Proxy>

This is also not correct


ProxyPass / balancer://app/
<Proxy balancer://app/>
BalancerMember http://10.0.0.166:8181/
BalancerMember http://10.0.0.166:8182/
BalancerMember http://10.0.0.166:8183/
BalancerMember http://10.0.0.166:8184/
BalancerMember http://10.0.0.166:8185/
</Proxy>
Made purty over there

And this is neat

Apache 2.2 Load Balancer Manager

·:· Posted 17 April 2006, 22:50 by Jason Hoffman to Web servers  |  Got something to say? [2]

What about Apache to Mongrel for Rails applications?

Just earlier I compared our lighttpd and litespeed serving up one of our Rails applications by proxying to mongrel

So what about Apache 2 using the prefork MPM?

This is our vanilla (and fat) Apache 2.0.x install.

Server Hostname: mongrel-ap.textdrive.com
Server Port: 80
Concurrency Level: 100
Complete requests: 10000
Requests per second: 431.57 [#/sec] (mean)

By fat, I mean this thing does mod_security, php, fcgi, DAV, SVN and has other authentication modules for MySQL and IMAP.

Let’s turn some stuff off

Let’s do a couple of things: let’s turn off mod_security as an input filter and let’s set KeepAlive Off

Server Hostname: mongrel-ap.textdrive.com
Server Port: 80
Concurrency Level: 100
Complete requests: 10000
Requests per second: 904.47 [#/sec] (mean)

Let’s start up more servers since we’re hitting it with a concurrency of 100

Was


<IfModule prefork.c>
StartServers 30
MinSpareServers 30
MaxSpareServers 55
ServerLimit 600
MaxClients 400
MaxRequestsPerChild 2500
ListenBacklog 1000
</IfModule>

Now


<IfModule prefork.c>
StartServers 100
MinSpareServers 10
MaxSpareServers 10
ServerLimit 10000
MaxClients 10000
MaxRequestsPerChild 2500
ListenBacklog 1000
</IfModule>

Server Hostname: mongrel-ap.textdrive.com
Server Port: 80
Concurrency Level: 100
Complete requests: 10000
Requests per second: 882.98 [#/sec] (mean)

Not too much of a change.

Let’s add some stuff

EnableSendfile On
EnableMMAP On

Server Hostname: mongrel-ap.textdrive.com
Server Port: 80
Concurrency Level: 100
Complete requests: 10000
Requests per second: 982.27 [#/sec] (mean)

OK that’s the best so far.

Hmm … what else can I do with this (without turning off our SVN, DAV etc)

Ah, I’ll turn off .htaccess and input filtering by mod_deflate as well

Server Hostname: mongrel-ap.textdrive.com
Server Port: 80
Concurrency Level: 100
Complete requests: 10000
Requests per second: 1070.59 [#/sec] (mean)

But I’ll turn those back on, no biggie.

The biggest hit actually comes from mod_security, restoring it after all the other changes pushed it back down to about 500 requests/second.

And this is very respectable by the way considering that just did a little better than lighttpd and litespeed for proxying to mongrel.

Joyent.com is a little PHP (mod_php) site with no database backend and under the same conditions it does

Requests per second: 588.36 [#/sec] (mean)

And the other and last point is that this is still a slow Apache because it’s prefork on FreeBSD and still doing rewrites, DAV, SVN and has several third party auth modules. I’m certain that it’s performance can be taken up several fold with a slim and trim proxy-only Apache 2.2 instance using the worker or event MPM.

There is also now a load-balancing extension for mod_proxy in Apache 2.2

There also wasn’t an appreciable load difference when using Apache and it’s nice to see the same mongrel processes stay at ~28MB (I used the same exact mongrel process with 4 threads for apache, lighttpd and litespeed).

·:· Posted 17 April 2006, 02:27 by Jason Hoffman to Rails  |  Got something to say? [3]

Lighttpd versus Litespeed with Mongrel as a backend for Rails applications

Done like always: client -> server over a gigabit connection, and both lighttpd and litespeed are proxying back to the same set of mongrel processes (in other words everything is exactly matched including file size and transfer rates).

This set is on FreeBSD with a ruby compile that has no pthreads.

Litespeed Enterprise 2.1.14 -> Mongrel 0.3.12.4

Concurrency Level: 100
Requests per second: 1008.04 [#/sec] (mean)

Lighttpd 1.4.10 -> Mongrel 0.3.12.4

Concurrency Level: 100
Requests per second: 1010.36 [#/sec] (mean)

This is forcing everything to be served through mongrel, and was perfectly comparable with fastcgi’s performance. Having the file be cached to the filesystem by rails and served as a static file by litespeed or via lighttpd’s mod_cml actually boosts it another 3-4 fold.

So …?

  1. Mongrel is good stuff.
  2. Lighttpd or litespeed for the simple purposes of being a load-balancing proxy capable of virtual hosting? Doesn’t matter, they’re comparable.

BTW, if you want to click around to the public sites, http://mongrel.textdrive.com and http://mongrel-lite.textdrive.com

·:· Posted 16 April 2006, 23:16 by Jason Hoffman to Rails  |  Got something to say? [1]

Moving a ZFS filesystem and all of its snapshots from one zpool to another

This is how you can move all the contents of a ZFS filesystem from one zpool of block storage to another zpool of block storage. This example was done from one zpool of fiber attached LUNs to another. This can also be done from on- to off-site.

This is over in textsnippets as well.

Go into the root directory


[hostname:/] root# cd /

You can look in the filesystem and you’ll see its contents and the .zfs folder


[hostname:/] root# ls -l old-pool/filesystem
total 24
dr-xr-xr-x 3 root root 3 Mar 31 00:34 .zfs/
drwxr-xr-x 18 root root 101 Mar 11 21:45 etc/
drwx------ 4 root root 16 Feb 19 08:07 root/
drwxr-xr-x 3 root root 3 Feb 19 21:18 usr_local_etc/
drwxr-xr-x 3 root root 3 Feb 19 21:18 usr_local_var_db_mysql/

Make a final “migration” snapshot that represents the latest old-pool/filesystem


[hostname:/] root# zfs snapshot old-pool/filesystem@migration

You can see there are 3 snapshots in there


[hostname:/] root# ls -l old-pool/filesystem/.zfs/snapshot/
20060329/ 20060330/ migration/

Do a zfs backup of the oldest snapshot and pipe that into a zfs restore. This is will make the filesystem in the new-pool. You could also do this over ssh.


[hostname:/] root# zfs backup old-pool/filesystem@20060329 | zfs restore new-pool/filesystem@20060329

Now you do an incremental (-i) backup and restore using the first snapshot you used above and the one that comes after it. The key here is that incremental backups expect there to be a pre-existing new-pool/filesystem, this is how it is diffrent from the non-incremental backup above.


[hostname:/] root# zfs backup -i old-pool/filesystem@20060329 old-pool/filesystem@20060330 | zfs restore new-pool/filesystem

Do an incremental on the next pair. This happens to be with the last “migration” snapshot.


[hostname:/] root# zfs backup -i old-pool/filesystem@20060330 old-pool/filesystem@migration | zfs restore new-pool/filesystem

When you look in the new-pool/filesystem you’ll see that it has been populated from the last migration snapshot.


[hostname:/] root# ls -l new-pool/filesystem
total 24
dr-xr-xr-x 3 root root 3 Mar 31 00:34 .zfs/
drwxr-xr-x 18 root root 101 Mar 11 21:45 etc/
drwx------ 4 root root 16 Feb 19 08:07 root/
drwxr-xr-x 3 root root 3 Feb 19 21:18 usr_local_etc/
drwxr-xr-x 3 root root 3 Feb 19 21:18 usr_local_var_db_mysql/

And all three snapshots are present


[hostname:/] root# ls -l new-pool/filesystem
20060329/ 20060330/ migration/

Yes it scripts up fine as well.

·:· Posted 16 April 2006, 02:48 by Jason Hoffman to ZFS  |  Got something to say?

← Previously