Fun

Rails got upgraded, people tool around with their typos on there, then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more, then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more, then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more, then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more, then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more, then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more, then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more, then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more, then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more, then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,then they had their daedalus hit the site, site a little sluggish, so it restarts lighttpd, but see that doesn’t touch the dispatch.fcgi files, it just launches 2-4 more,

And see

When you combine with a big fat-ass fscking memory leak

What happens?

It completely fried a server.

Yes in fact, it fried two servers and nearly some more. In the last week, we’ve have $1500/each procs, a stick of RAM and a raid controller get completely fried in two $5000 servers. So bad that the drives had to be pulled (seriously that’s what the pictures on flickr are ). And guess what the long repetitive string of entries in the process accounting logs where? Wanna guess? Wanna? Go ahead. Guess. The retrieved core dump?

The last time I saw a server get fried that like I was tooling around with lighttpd’s config (seriously I did completely fry two drives putting … well maybe I shouldn’t say).

So we put in all that hard (used to be soft) memory, process, open file limits etc.

And what happened, half a dozen people on barclay hit them. All typo users, and then it exposed that memory leak in rails itself.

So … FUN FUN!

And Scott and Scott, and the entire teams, were beyond perfect in their handling of it. Worked just like an open source project should (actually better than just about any professional operation).

And hey don’t think for one second this is critical of Rails, it isn’t. I find someone posting on their blog that it is, and I’ll personally kick you in the nards.

Stuff is better than Tacos and that’s sayin’ somethin’.

·:· Posted 26 October 2005, 05:33 by Jason Hoffman to Stuff  |  

  1. Are you seriously implying that a piece of software can result in ram, processors, and a raid controller being fried or are you just having fun (as the title implies)? Inadequate cooling coupled with an over-active processor and drive can do that, but software can’t do that. Ram doesn’t wear out that much quicker if you fill it up too much! :-) I, for one, would love it if the number of reboots were reduced so I’m a fan of what you’re prescribing, but if your new hardware is going bad, I’d seriously check ambient temperatures in the rack ;-)

    — scott    26 October 2005, 06:10    #
  2. Sure it’s Fun. And we're talking figurative frying, not literally fried fried. And don’t you start pulling temperature. Each component of every server has a temperature time course log on it, and the ambient temperature in front and back is measured as well.

    — jason Hoffman    26 October 2005, 06:38    #
  3. That sucks. However…

    “The last time I saw a server get fried that like I was tooling around with lighttpd’s config (seriously I did completely fry two drives putting … well maybe I shouldn’t say).”

    Yes, yes, you should. I sure don’t want to be the one accidentally causing this, be hung out to dry in the forums, and have to pay hefty recovery fees. However, if I have no clue what to avoid, that might just be next time.

    Are there any actual recommendations or just ranting? Don’t use Rails? Don’t use Typo? Don’t use FCGI? Don’t use Daedalus? Damn guys, you need to throw us a bone here and give us specific guidelines and/or advice – especially when it turns out the previous specific guidelines you’ve given are makihng us fry servers.

    — Jakob S    26 October 2005, 07:11    #
  4. I’m curious too about how misbehaving client software is supposed to have fried (do you literally mean “burn up”—overheat?) RAM, hard drives and a RAID controller. Please expand on this! Thanks.

    — Eric O'Brien    26 October 2005, 07:46    #
  5. Scott seems rather right. A software frying hardware ? Hmmmm…

    A statistic I would be curious to see… how many outages/downtimes are related to Ruby, compared to other causes ?

    — Jérémie    26 October 2005, 07:52    #
  6. It’s easy: don’t configure your daedalus to check every 10 seconds and then if it fails, give it more than 2 seconds for the app to start up again.

    — jason Hoffman    26 October 2005, 09:34    #
  7. Thanks Jason, that’s exactly the kind of speficics that are nice to know. So are there any recommendations for values? I doubt checking every 11 seconds with 3 seconds of delay on failure is okay?

    — Jakob S    26 October 2005, 09:39    #
  8. Jan posted some hard numbers for Daedalus in the forums just now. Slightly longer intervals than 11 and 3 seconds, though.

    — Jakob S    26 October 2005, 10:29    #
  9. I have four Dell PE6850’s, each of them have already gone through two sets of memory, a processor card, and a RAID controller.

    I was originally going to go with IBM x445’s but decided that Dell was the better option due to price. Never again!

    — Brian    4 November 2005, 04:03    #
  10. Oh Rails, you are so perfect, so spectacularrr in every way
    You bring light into my life, Rails. You almost make me forget all about…
    Tacos! Oh, tacos, so good in my tummy yummy yummy give me more.
    I love you, Rails. You almost make me forget about…
    Tacos…...

    — slumos    4 November 2005, 19:37    #
  11. You might also want to upgrade to a newer version of Daedalus. It’s at 2.something now. You can snag if from http://wrath.grayskies.net:8080/view/Daedalus or gem install daedalus

    — Ralph D    28 November 2005, 20:27    #