The weblog is heading over to Joyeur.com

I’m going to be “blogging” quite a more again but all the new content is going to be over at our Joyeur weblog, so please redirect your Atom or RSS feeds.

All the older content is staying right here because well … it’s linked to a bit (the first lighttpd for rails and php article from Feb 2005 for example) and google loves it.

·:· Posted 864 days ago by Jason Hoffman to About This Particular Weblog  |  Got something to say? [2]

The Scale with Rails workshops

I wrote a little piece on the Scale with Rails workshop over at Joyeur

“I thought why not take all of the questions and concerns that I’ve heard over the last two years and all of the issues we’ve studied and addressed ourselves, and present a platform?”

“That platform would not only be best practices for even the largest of companies (because we’ve all been in large Fortune 500 companies), but then we’ll frame the entire discussion in a set of questions that should be and likely are on anyone’s mind. A set of questions that in fact are unlikely to change. What you’ll get then is not blind Rails evangelism, but the perspective of an architect, engineer and executive who currently thinks that the answer to one of those questions was to use the Ruby and then the Rails framework (and sometimes it isn’t, sometimes C++ is).”

Tickets are available at the store

·:· Posted 937 days ago by Jason Hoffman to Workshops  |  Got something to say? [2]

By popular demand, the plain FSCK You shirts are out

At the store

·:· Posted 942 days ago by Jason Hoffman to Server geek  |  Got something to say? [2]

Apache 2.2, mod_proxy_balancer and Mongrel

Jonathan Weiss has a great little write up on their use of Apache 2.2, mod_proxy_balancer and mongrel

I couldn’t agree more.

As is here, here and here

I’m telling you.

I think it’s Mongrel -> Apache 2.2 event with mod_proxy_balancer -> k-ssl on Solaris.

{k-ssl is Solaris’s Kernel level SSL proxy.}

·:· Posted 953 days ago by Jason Hoffman to   |  Got something to say? [3]

Apache 2.2 worker on solaris to a remote mongrel

Zoned to 1/8th of a solaris machine but the rest is the same as the others further down.

Was

FreeBSD client -> FreeBSD server (web -> mongrel)

and

FreeBSD client -> FreeBSD server (web) -> FreeBSD server (mongrel)

This is

FreeBSD client -> Solaris server (web) -> FreeBSD server (mongrel)

Concurrency Level: 100
Complete requests: 10000
Requests per second: 1452.23 [#/sec] (mean)

So it wins.

FreeBSD client -> Solaris server (web) -> FreeBSD server (web) -> FreeBSD server (mongrel) also turned out fine by the way.

The bottleneck actually is how many backends and you can see that I’m unimpressed with FreeBSD network pumping out abilities even when tuned up so the entire chain will be tuned up on Solaris.

And I tell you, I actually turned out to be quite impressed with


<VirtualHost 10.0.0.247:80>
ServerName mongrel-worker-solaris.textdrive.com
ProxyPass / balancer://app
<Proxy balancer://app>
BalancerMember http://10.0.0.166:8181
BalancerMember http://10.0.0.166:8182
BalancerMember http://10.0.0.166:8183
BalancerMember http://10.0.0.166:8184
BalancerMember http://10.0.0.166:8185
BalancerMember http://10.0.0.166:8186
BalancerMember http://10.0.0.166:8187
BalancerMember http://10.0.0.166:8188
BalancerMember http://10.0.0.166:8189
BalancerMember http://10.0.0.166:8190
BalancerMember http://10.0.1.166:8181
BalancerMember http://10.0.1.166:8182
BalancerMember http://10.0.1.166:8183
BalancerMember http://10.0.1.166:8184
BalancerMember http://10.0.1.166:8185
BalancerMember http://10.0.1.166:8186
BalancerMember http://10.0.1.166:8187
BalancerMember http://10.0.1.166:8188
BalancerMember http://10.0.1.166:8189
BalancerMember http://10.0.1.166:8190
</Proxy>
</VirtualHost>

In a slightly related note, I’m sick of doing configs in flat files on the same machines.

·:· Posted 959 days ago by Jason Hoffman to   |  Got something to say? [2]

I made some other web server points over at Joyeur

It’s here and it’s primarily about putting a web server in the context of the network stack and connection (with a hat tip to an old post from Querna that’s been sitting in the brain for nearly a year).

·:· Posted 959 days ago by Jason Hoffman to Web servers  |  Got something to say?

Beware the trailing slash in Apache's proxy balancer

This is correct


ProxyPass / balancer://app
<Proxy balancer://app>
BalancerMember http://10.0.0.166:8181
BalancerMember http://10.0.0.166:8182
BalancerMember http://10.0.0.166:8183
BalancerMember http://10.0.0.166:8184
BalancerMember http://10.0.0.166:8185
BalancerMember http://10.0.0.166:8186
BalancerMember http://10.0.0.166:8187
BalancerMember http://10.0.0.166:8188
BalancerMember http://10.0.0.166:8189
BalancerMember http://10.0.0.166:8190
</Proxy>

This is not


ProxyPass / balancer://app
<Proxy balancer://app>
BalancerMember http://10.0.0.166:8181/
BalancerMember http://10.0.0.166:8182/
BalancerMember http://10.0.0.166:8183/
BalancerMember http://10.0.0.166:8184/
BalancerMember http://10.0.0.166:8185/
BalancerMember http://10.0.0.166:8186/
BalancerMember http://10.0.0.166:8187/
BalancerMember http://10.0.0.166:8188/
BalancerMember http://10.0.0.166:8189/
BalancerMember http://10.0.0.166:8190/
</Proxy>

This is also not correct


ProxyPass / balancer://app/
<Proxy balancer://app/>
BalancerMember http://10.0.0.166:8181/
BalancerMember http://10.0.0.166:8182/
BalancerMember http://10.0.0.166:8183/
BalancerMember http://10.0.0.166:8184/
BalancerMember http://10.0.0.166:8185/
</Proxy>
Made purty over there

And this is neat

Apache 2.2 Load Balancer Manager

·:· Posted 959 days ago by Jason Hoffman to Web servers  |  Got something to say? [2]

What about Apache to Mongrel for Rails applications?

Just earlier I compared our lighttpd and litespeed serving up one of our Rails applications by proxying to mongrel

So what about Apache 2 using the prefork MPM?

This is our vanilla (and fat) Apache 2.0.x install.

Server Hostname: mongrel-ap.textdrive.com
Server Port: 80
Concurrency Level: 100
Complete requests: 10000
Requests per second: 431.57 [#/sec] (mean)

By fat, I mean this thing does mod_security, php, fcgi, DAV, SVN and has other authentication modules for MySQL and IMAP.

Let’s turn some stuff off

Let’s do a couple of things: let’s turn off mod_security as an input filter and let’s set KeepAlive Off

Server Hostname: mongrel-ap.textdrive.com
Server Port: 80
Concurrency Level: 100
Complete requests: 10000
Requests per second: 904.47 [#/sec] (mean)

Let’s start up more servers since we’re hitting it with a concurrency of 100

Was


<IfModule prefork.c>
StartServers 30
MinSpareServers 30
MaxSpareServers 55
ServerLimit 600
MaxClients 400
MaxRequestsPerChild 2500
ListenBacklog 1000
</IfModule>

Now


<IfModule prefork.c>
StartServers 100
MinSpareServers 10
MaxSpareServers 10
ServerLimit 10000
MaxClients 10000
MaxRequestsPerChild 2500
ListenBacklog 1000
</IfModule>

Server Hostname: mongrel-ap.textdrive.com
Server Port: 80
Concurrency Level: 100
Complete requests: 10000
Requests per second: 882.98 [#/sec] (mean)

Not too much of a change.

Let’s add some stuff

EnableSendfile On
EnableMMAP On

Server Hostname: mongrel-ap.textdrive.com
Server Port: 80
Concurrency Level: 100
Complete requests: 10000
Requests per second: 982.27 [#/sec] (mean)

OK that’s the best so far.

Hmm … what else can I do with this (without turning off our SVN, DAV etc)

Ah, I’ll turn off .htaccess and input filtering by mod_deflate as well

Server Hostname: mongrel-ap.textdrive.com
Server Port: 80
Concurrency Level: 100
Complete requests: 10000
Requests per second: 1070.59 [#/sec] (mean)

But I’ll turn those back on, no biggie.

The biggest hit actually comes from mod_security, restoring it after all the other changes pushed it back down to about 500 requests/second.

And this is very respectable by the way considering that just did a little better than lighttpd and litespeed for proxying to mongrel.

Joyent.com is a little PHP (mod_php) site with no database backend and under the same conditions it does

Requests per second: 588.36 [#/sec] (mean)

And the other and last point is that this is still a slow Apache because it’s prefork on FreeBSD and still doing rewrites, DAV, SVN and has several third party auth modules. I’m certain that it’s performance can be taken up several fold with a slim and trim proxy-only Apache 2.2 instance using the worker or event MPM.

There is also now a load-balancing extension for mod_proxy in Apache 2.2

There also wasn’t an appreciable load difference when using Apache and it’s nice to see the same mongrel processes stay at ~28MB (I used the same exact mongrel process with 4 threads for apache, lighttpd and litespeed).

·:· Posted 960 days ago by Jason Hoffman to Rails  |  Got something to say? [3]

Lighttpd versus Litespeed with Mongrel as a backend for Rails applications

Done like always: client -> server over a gigabit connection, and both lighttpd and litespeed are proxying back to the same set of mongrel processes (in other words everything is exactly matched including file size and transfer rates).

This set is on FreeBSD with a ruby compile that has no pthreads.

Litespeed Enterprise 2.1.14 -> Mongrel 0.3.12.4

Concurrency Level: 100
Requests per second: 1008.04 [#/sec] (mean)

Lighttpd 1.4.10 -> Mongrel 0.3.12.4

Concurrency Level: 100
Requests per second: 1010.36 [#/sec] (mean)

This is forcing everything to be served through mongrel, and was perfectly comparable with fastcgi’s performance. Having the file be cached to the filesystem by rails and served as a static file by litespeed or via lighttpd’s mod_cml actually boosts it another 3-4 fold.

So …?

  1. Mongrel is good stuff.
  2. Lighttpd or litespeed for the simple purposes of being a load-balancing proxy capable of virtual hosting? Doesn’t matter, they’re comparable.

BTW, if you want to click around to the public sites, http://mongrel.textdrive.com and http://mongrel-lite.textdrive.com

·:· Posted 960 days ago by Jason Hoffman to Rails  |  Got something to say? [1]

Moving a ZFS filesystem and all of its snapshots from one zpool to another

This is how you can move all the contents of a ZFS filesystem from one zpool of block storage to another zpool of block storage. This example was done from one zpool of fiber attached LUNs to another. This can also be done from on- to off-site.

This is over in textsnippets as well.

Go into the root directory


[hostname:/] root# cd /

You can look in the filesystem and you’ll see its contents and the .zfs folder


[hostname:/] root# ls -l old-pool/filesystem
total 24
dr-xr-xr-x 3 root root 3 Mar 31 00:34 .zfs/
drwxr-xr-x 18 root root 101 Mar 11 21:45 etc/
drwx------ 4 root root 16 Feb 19 08:07 root/
drwxr-xr-x 3 root root 3 Feb 19 21:18 usr_local_etc/
drwxr-xr-x 3 root root 3 Feb 19 21:18 usr_local_var_db_mysql/

Make a final “migration” snapshot that represents the latest old-pool/filesystem


[hostname:/] root# zfs snapshot old-pool/filesystem@migration

You can see there are 3 snapshots in there


[hostname:/] root# ls -l old-pool/filesystem/.zfs/snapshot/
20060329/ 20060330/ migration/

Do a zfs backup of the oldest snapshot and pipe that into a zfs restore. This is will make the filesystem in the new-pool. You could also do this over ssh.


[hostname:/] root# zfs backup old-pool/filesystem@20060329 | zfs restore new-pool/filesystem@20060329

Now you do an incremental (-i) backup and restore using the first snapshot you used above and the one that comes after it. The key here is that incremental backups expect there to be a pre-existing new-pool/filesystem, this is how it is diffrent from the non-incremental backup above.


[hostname:/] root# zfs backup -i old-pool/filesystem@20060329 old-pool/filesystem@20060330 | zfs restore new-pool/filesystem

Do an incremental on the next pair. This happens to be with the last “migration” snapshot.


[hostname:/] root# zfs backup -i old-pool/filesystem@20060330 old-pool/filesystem@migration | zfs restore new-pool/filesystem

When you look in the new-pool/filesystem you’ll see that it has been populated from the last migration snapshot.


[hostname:/] root# ls -l new-pool/filesystem
total 24
dr-xr-xr-x 3 root root 3 Mar 31 00:34 .zfs/
drwxr-xr-x 18 root root 101 Mar 11 21:45 etc/
drwx------ 4 root root 16 Feb 19 08:07 root/
drwxr-xr-x 3 root root 3 Feb 19 21:18 usr_local_etc/
drwxr-xr-x 3 root root 3 Feb 19 21:18 usr_local_var_db_mysql/

And all three snapshots are present


[hostname:/] root# ls -l new-pool/filesystem
20060329/ 20060330/ migration/

Yes it scripts up fine as well.

·:· Posted 961 days ago by Jason Hoffman to ZFS  |  Got something to say?

RSS / Atom

Related