Esoteric Curio

Irony: The website is what?

nospam@example.com (Theo Schlossnagle) — Sat, 20 Sep 2008 10:40:49 -0400

The Internet is not without a sense of irony:

The website is up.

In case they fix it They fixed it... thewebsiteisup.com just spewed PHP errors at the time I wrote this.

Zetaback. Respect.

nospam@example.com (Theo Schlossnagle) — Fri, 19 Sep 2008 10:02:15 -0400

With the release of ZFS on Solaris 10, I sat down and marveled at the opportunities for off-site backups. I have already written a bit about ZFS detailing why I think it kicks so much ass. With zfs send and zfs receive, one can manage block-level incremental backups and restores. What's missing? An elegant hack leveraging that to provide a simple and reliable backup infrastructure for a network of ZFS capable machines (including Mac OS X and FreeBSD now, BTW).

So, I sat down and wrote Zetaback -- which is currently 1032 lines of perl code (including complete documentations) plus a thin agent on remote machines that is 290 lines of perl code (including complete documentation). I'd like to note that the only reason there is documentation, let alone complete documentation, is because of Eric Sproul. This really demonstrates to me that "Keep It Simple Stupid" still works for important tasks.

Zetaback is a rather full features backup and restore system. It can manage multiple hosts, multiple ZFS per host, both frequency and retention policies on full and incremental backups. It can report policy violators (things that haven't been backed up within the policy). It can manage the archiving of backups. It provides both non-interactive and interactive restores. It has an excellent command line syntax. And most importantly, it has saved my ass more times than I can count.

I'm not usually big on awards... I find the single unexpected email from someone saying: "damn that was useful, thanks!" to be more gratifying most of the time. However, Zetaback was one of the first projects we put up on labs, so being a 3rd place winner in the OpenSolaris Community Innovation Awards is pretty exciting.

Last second scaling hack

nospam@example.com (Theo Schlossnagle) — Tue, 16 Sep 2008 00:17:20 -0400

So, you have an app. You can't change the code. Now this isn't the common case when I try to scale things. I usually roll up my sleeves and ignore application stack boundaries. This is a unique case where for political reasons, I can't touch the app. So.. the app was a tiny little site, then it got popular on facebook and collegehumor and instead of pushing 5-10 megabits, it was falling apart at around 105 megabits due to resource saturation (one box wasn't enough) and ended up needing to push 200 megabits.

200 megabits isn't all that much traffic anymore, but when the application wasn't written to scale horizontally, you are at the mercy of its raw performance and must scale vertically. If the application hasn't had a lot of focus on profiling and performance tuning, it means you are going to hit that extremely painful price point of vertical scaling. In this case, the architecture went live with an expectation of a 20Mbit/s peak and BOOM. Because it needed to be fixed quickly, purchasing new hardware is now a problem for scheduling reasons more than financial ones, we have plenty of similar hardware available, just nothing with twice the RAM and twice the cores and twice the disks.

The reason that this app couldn't scale is because it used not only a shared DB (which is very very common) it required filesystem use and thus needed a shared filesystem. So, how do you fix that without modifying the app? You study the app and look for patterns of use that can be exploited.

First we looked at the database. In this case, it was not being pushed very hard. We could easily handle a tenfold increase in traffic without exhausting database resources... That was a relief, because scaling a database "behind the scenes" without any application access can be more than a few hour exercise. Next we found that the app itself (PHP) was taxing memory, CPUs and disk I/O pretty heavily. The most important was memory and CPU, but disk I/O was a close second. This meant that if we just installed the app on another machine and NFS exported the first machine's mounts, it would "work" but not achieve out performance requirements because of I/O saturation. Quick testing in this arena showed about 15% increase in capacity -- just not enough.

So, this app needs a shared FS. Why? Well the user uploads assets, and then through the life of their session, the app serves them back to that user. EASY, session sticky load balancing (by source IP or by introduced cookie on the load balancer). Because of the nature of this app, session sticky load balancing produced extremely inequitable load distribution and we would have had to bump up to three servers. Not ideal, but acceptable -- this is triage. One step forward, flat on our face: it appears that under certain circumstances, the images I upload are served to another.

So, basically, all I need is to glue the static assets (uploaded by users) together under a common URL (and push 200Mbs or so). Some assets are on one server, some on another, and I have no way of knowing which server owns the asset without looking in the FS... or asking over HTTP and getting a 404 back.

I just happen to have a Varnish instance to provide content acceleration for other bits of infrastructure. And Varnish has (as its major selling point, IMO) the VCL language that allows me to script how it handles requests and satisfies them.

If I get a request, I want to try server one, if I get a 404, I'd like to retry the request against server two. As the number of servers goes up, this solution completely falls apart as the 404 isn't that cheap. I want it fast, efficient, and it'd be great to cache it. If it isn't fast and efficient, I've simply moved my problem instead of addressing it. This works well because serving a 404 on server one is cheap. Remember, triage.

backend obscuredserver1 {
  .host = "10.225.209.89";
  .port = "80";
}
backend obscuredserver2 {
  .host = "10.225.209.90";
  .port = "80";
}

sub vcl_recv {
  if (req.http.host ~ "^fqdn\.of\.caching\.server$") {
    if (req.restarts == 0) {
      set req.backend = obscuredserver1;
    } else {
      set req.backend = obscuredserver2;
    }
  }
  if (req.request != "GET" && req.request != "HEAD") {
    pipe;
  }
  lookup;
}

sub vcl_fetch {
  if (req.http.host ~ "^fqdn\.of\.caching\.server$" &&
      req.restarts == 0 && obj.status == 404) {
    restart;
  }
  if (!obj.cacheable) {
    pass;
  }
  if (obj.http.Set-Cookie) {
    pass;
  }
  set obj.prefetch = -30s;
  deliver;
}

Now, this is a excerpt, my varnishes here have some other logic for other services that I can't share... However, they are rather lightly used. That particular instance went from serving an average of 6 Mbits/second to peaking at 200 Mbits/second. And the system load jumped from 0.01 to 0.06. It's nice when a triage exercise results in a quick hack that doesn't bust at the seams -- we've got plenty of headroom.

While I, in no way, consider this successful scaling. I consider it successful triage by creative engineering (a.k.a. hack). And for those that like pretty pictures, these demonstrate that when you encounter capacity issues, it isn't always pretty and graceful. Queueing theory is complicated and sometimes results in everyone getting screwed. Here's a visualization of queueing theory making trouble.

Queueing theory rears its ugly head

What it looks like with some headroom.

OpenSSH and SecurID, still a good choice.

nospam@example.com (Theo Schlossnagle) — Tue, 02 Sep 2008 11:01:47 -0400

A long time ago, I wrote integration into the portable version of OpenSSH to allow direct authentication against an RSA ACE (SecurID) server. I've received many thanks over time for the work and I'm aware that it is used at some (very large) organizations. However, as with most security related things, people tend not to talk about what they do. As it is open source and no registration is required to download the patch, I think I might have underestimated the deployments.

Quite some time ago, Jim Matthews over at NASA took over maintenance of the patch. This sort of seamless transition of ownership is why I really love open source. Jim does a great job.

Since that patch's inception, it has been hosted on my old static projects page. That meant that James has to send me a copy to post every time a new version of the patch came out. How 1998. Anyway, since we went through all the effort of setting up open source hosting, how about I use it! The OpenSSH+SecurID integration effort has moved to labs! Get your one-time-password, two-factor security while it's hot!

XML/XSLT and DocBook for docs

nospam@example.com (Theo Schlossnagle) — Sun, 31 Aug 2008 12:01:28 -0400

I've been writing docs for Reconnoiter. I selected DocBook for two reasons. First, I hoped that number of polished documents I've seen written in DocBook would mean that if this manual grows in size and usefulness we might be able to achieve some polish "on the cheap." Second, our open-source site has a really nice automated systems for auto-publishing project documentation... if it is in DocBook. That said, DocBook is a complete pain in the ass. It isn't broken or bad, but it really gets in the way of writing the documentation. There are so many marks, and their use is specific and contextual. I understand that there is a reason for all the marks (provide semantic meaning to what you write), but you must be fluent with the entire specification and practiced to achieve anything useful with DocBook.

The fact that DocBook uses XML is tiresome. XML itself isn't bad. Despite my extreme ineptitude and writing XML that will lint, I'm rather fond of XML. It just so happens that because I'm fond of XML, the software product I'm using leverage XML in configuration files and support files. This leaves me in the painful position of documenting XML in XML and now I have all sorts of escaping requirements that I'd like to not use CDATA for (cause I want to lint them).

Now, enter the next phase of inconvenience (read: torture). Because Reconnoiter is modular, the documentation for the modules needs to be programmatically accessible to assist with online configuration validation. To achieve this, we choose to put the module docs right next to the module itself and from there we produce DocBook snippets for inclusion in the reference manual. The module documentation is in XML, the configuration of the module is in XML, the configuration of Reconnoiter is in XML, DocBook is in XML... We just need to take them all and compose documentation (with all the right escaping). Enter XSLT: a tool of torture. Now, I'm intimately familiar with the inner workings of XSLT transforms from the C side hving worked extensively on FastXSL for PHP, but actually writing XSL documents is something I strive to avoid. This conflicts with one of my core principles: "use the right tool for the job." The entire purpose of XSL is to translate XML documents into other documents (XML being a first-class target). XSLT it is.

I'm left in a position of documenting a component of the system in XML (easy enough) and then writing XML to run on the first XML (which contains both raw docbook XML snipets that cannot be escaped as well as XML configuration snipets which must be escaped) to produce a second XML to be included by a larger XML DocBook document. This now conflicts with another of my core principles: "K.I.S.S." (I'll note the wikipedia entry says it's used on the Internet in a way that implies it started there. This term far far predats the Internet. Oh, and the last S is definitely "Stupid").

Now, I don't much like DocBook because of my lack of fluency and its interruptive influence on the process of writing documentation. This is mostly my shortcoming, not DocBook's. I never can seem to write an XML document that lints the first time around... Again, my shortcoming, not XMLs. XSLT is an "ornate, complexly syntaxed, functionally limited shortcoming." I really hate XSLT. Here are someone else's reasons. They are good, but mine reason is that my tasks are usually simple and there is no simple variant of XSLT that can do the job; i.e. the jobs I do are simple and I think XSLT's cost (learning) rarely outweighs the value.

This makes me think that most documentations should be written by a tag-team. A documentation writer that writes in something like POD (the ultimate in K.I.S.S.) and a documentation compiler that consumes that and produces all the required DocBook stuff. Within a company, it's easy to put this process in place, but it's hard to get this sort of collaboration on open-source projects.

It's hard enough to get an engineer to write good documentation. Painful, persistent and minor technical obstacles really don't help. There should be a really kick-ass opensource docbook editor for technical manuals that integrates with subversion and mult-file layouts. This won't ever make XSLT better, but it would sure allow engineers to concentrate on content instead of markup. Anyone know of one?

My favorite cookbook

nospam@example.com (Theo Schlossnagle) — Wed, 27 Aug 2008 17:43:57 -0400

Many people say that their favorite cookbook is Joy of Cooking. I love that cookbook, it is excellent. However, when it comes to just getting the job done, I find the ultimate reference manual to be The Original Fannie Farmer 1896 Cookbook: The Boston Cooking School

Just thought I'd share.

Sweet Shrimp Goulash

nospam@example.com (Theo Schlossnagle) — Wed, 27 Aug 2008 15:11:25 -0400

Okay, it's not quite Goulash, but I'll call it what I want. If you want it red, add some paprika.

I made this the other night and it worked well for me.

Glaze:

1/2 cup of rice wine vinegar
2 tbsp of honey
1/3 tbsp of lemon juice

Bring to boil and reduce to coat the back of a spoon.

Goulash

1 red onion chopped large
6 cloves are garlic shredded or pressed
3 jalepenos sliced (seeds in)
2 thai chilis sliced (seeds in)
3 table spoons of vegetable oil

Sweat the the above until onions loose their sharpness (not yet translucent). Immediately add:

4 cups of assorted exotic mushrooms (shitake and others to liking)

Cook until mushrooms take on juices and onions are translucent. Immediately add:

1 lbs of 26-30 count shrimp, pealed, tails off

Once shrimp are pink evenly add glaze. Cook until shrimp are done (30 seconds or so past pink).

Enjoy! It has some bite.

Varnish, get your patch on.

nospam@example.com (Theo Schlossnagle) — Sun, 10 Aug 2008 21:50:00 -0400

Varnish is a "bad ass" new HTTP caching accelerator. It's developed by some crufty old BSD hacker and has a lot of Linux users. By and large, it has ignored Solaris. This sort of neglect isn't malicious, it is just neglect... you know: "out of sight, out of mind."

Well, check out Varnish trunk and give this patch a spin. Let me know what you think.

Perhaps one day, the Solaris networking team (or someone else) will satisfy this pretty abysmal shortcoming: BugID 4641715.

BWPUG: The essential PostgreSQL.conf

nospam@example.com (Theo Schlossnagle) — Fri, 08 Aug 2008 13:55:44 -0400

Howdy folks,

This is a reminder that our monthly meetup is scheduled to take place this coming MONDAY, August 11th. As requested, we've moved the meetings from Wednesday to Monday to facilitate some of the would-be-attendees that have contacted me out of band.

This month's presentation is titled "The essential PostgreSQL.conf". With almost 200 configuration parameters, some people might think the postgresql.conf is a bit heady, but the truth is there are only about 2 dozen that you really need for everyday use. This talk will discuss the different types of configuration settings, and give an overview of the ones you'll want to know when running PostgreSQL. Speakers for the talk are Greg Smith, Software Engineer at Truviso, and Robert Treat, Database Architect at OmniTI.

Look forward to seeing you all there!

2008-08-11 @ 6:30pm
OmniTI
7070 Samuel Morse Dr. Ste 150
Columbia, MD 21046

Best regards,

Theo

OSCON2008 Presentation

nospam@example.com (Theo Schlossnagle) — Thu, 24 Jul 2008 03:58:09 -0400

Hello from OSCON. I gave my full-stack introspection crash course talk today. It has been quite a while since I've presented anything in a 40 minute format, but I think the talk went quite well. I got a lot of positive feedback.

I decided to take a risky approach inspired by dtrace.conf(08) by demonstrating dtrace on a live, mission-critical system we run at OmniTI. The risks of this are: network connections flake out, dtrace doesn't work correctly or I do something stupid and cause some service unavailability. Well, as I use dtrace just about every day, I wasn't worried about the breaking things. And while my network connection winked out for about one minute and dtrace has some annoying sub-second aborts, I think the demonstration was quite effective.

Many people gave positive commentary at the end and afterward. People asked for the slide to be put online... and while they have no real content of value (as the demo was everything), I put them here anyway:

OSCON2008 Full-stack Introspection Crash Course

view presentation (tags: oscon dtrace)

In addtion to the slide stack, I've included the simple scripts that I used during the demonstration. I ran qps.d, query_speed.d, and query_speed2.d on the database server and r3.sh, perl.d, pcpu.d, papcpu.d, and papcpu2.d on the web server. These require The Devel::DTrace perl module, PostgreSQL patches, and Apache 2.2.8 patches. Some of them are approximations of correctness, so weigh the output appropriately (the perl ones). Enjoy!

Scalability and concessions

nospam@example.com (Theo Schlossnagle) — Wed, 02 Jul 2008 11:52:03 -0400

Oren Hurvitz has a great post about LinkedIn's architecture. It's well-written and well thought out. Their architecture has evolved on what appears to be a steady and safe path of improvement. It is well worth a read.

I would like to comment on something I see repeated again and again and is likely misinterpreted by young scalability architects. The statement of what you should expect to lose when you scale up/out. Oren writes:

The presentation ends with some tips about scaling. These are oldies but goodies:
Can’t use just one database. Use many databases, partitioned horizontally and vertically.
Because of partitioning, forget about referential integrity or cross-domain JOINs.
Forget about 100% data integrity.
At large scale, cost is a problem: hardware, databases, licenses, storage, power.
Once you’re large, spammers and data-scrapers come a-knocking.
Cache!
Use asynchronous flows.
Reporting and analytics are challenging; consider them up-front when designing the system.
Expect the system to fail.
Don’t underestimate your growth trajectory.

Now, I agree with much of that. The spammers comment should be revised to "Fraud happens and the bigger you are, the bigger the bullseye." Be aware and protect your assets. Everything from Cache! on down: hard and fast rules. The cost argument is odd. While it is completely correct, it's also rather obvious. If your business model ties audience size and site use to revenue (which it should), then the cost should simply scale sub-linearly w.r.t. revenues (i.e. no big deal). However, there are a few that remain on that list that should be cherished and the loss of them should pain you.

"[You] Can't use just one database" -- this is a conclusion you should arrive at after analysis. We have one client that supports 10 million users on a cluster of partitioned databases. We have another that supports 35 millions users on one database without issue and room for growth.

"Because of partitioning, forget about referential integrity or cross-domain JOINs." Think. Think hard. Think harder. Sometimes it is possible to partition in a fashion that allows for integrity. While I'm sure (or at least hope) that the LinkedIn guys had some sleepless nights making the decision to break foreign constraints, it isn't conveyed. You should absolutely have some sleepless nights over a decision like that. My bank supports many more users and transactions than LinkedIn -- and it damn well better have FKs and 100% integrity. So, while you still may partition in such a fashion that requires a loss of enforced integrity, the decision should be a heavy one.

"Forget about 100% data integrity." WTF? While I'm sure it was the end of the post and he was being smart, someone somewhere might actually take the advice to forget about data integrity. You never, ever, ever forget about it. We have some "one big database" architectures where data integrity has been an issue due to memory bit-flips (corrupt data on disk) -- it's a BFP (big f@#$ing problem) and we treat it that way. Sometimes you make an architectural decision that will make the loss of integrity much more probable (partitioning and losing FK constraints is a ripe example). It's still something that should be attended to with great attention and diligence. you should never forget about data integrity and always put forth the effort required to reach as close to 100% as possible. When you lose data integrity you end up with a big pile of shit in your database. I'll leave you with a rather crass metaphor:

There's an expectation that there is no shit on your living room floor. Don't shit in your living room. Don't let your dog shit in your living room. If you're a dog owner, you know your dog could have an accident. You bought the dog. You chose to increase the probability of finding shit in your living room. Don't ignore it or forget it. Clean up the shit when it happens. If you get suddenly ill while playing your Wii naked and shit on your living room floor (be it probable or improbable)... respect yourself -- clean it up. Never forget the goal: a 100% shit-free living room.

Reconnoiter and another platform

nospam@example.com (Theo Schlossnagle) — Fri, 27 Jun 2008 17:27:27 -0400

Reconnoiter is coming along. Unlike most open source project, I tend not to talk about mine until their are really useful to people. Over the last year, I've adopted the unhealthy attitude that useful means "shiny front-end." So, I'm blogging to break that attitude and talk a bit about project that doesn't have a shiny front-end... yet.

Reconnoiter is built out of years of frustration using tools like RRDTOOL, Munin, Cacti, ZenOSS, Nagios, etc. etc. I have a lot of problems with these tools. First, they are not efficient. I need a powerful machine to monitor a mere 10k services. And it actually gets to be an engineering challenge to monitor 100k services with these tools. Also, the graphs are about 10 years old with respect to design and usability. I want something new, something fresh, and something that doesn't need a damn web UI to configure. Several people have asked, why are you reinventing the wheel? Why don't you just improve an existing product? My answer is that I want a well-thought-out product foundation so that I can trust all the bits. I want reponsibilities decoupled at the right spots. I want data in a form that the world can query and run reports the likes of which I have not concieved. I don't want the load on my monitoring machines to be 8. I want my monitoring system to check services and metrics when it planned to, not several minutes (or event 2 seconds) after it told me it would. Simply put, I expect it to work well, all the time. And, of course, I want it to work how I would expect it to work.

Reconnoiter was born out of the need to monitor the internals of many disconnected data centers with between 10 and 1000 machines in each facility. Monitoring can mean a lot of things, here I consider it to be the collection of metrics and awareness of their availability. In and of itself, monitoring is pretty useless, but it is the foundation for two critical pursuits in Internet infrastructure and business management: fault detection and trending.

Fault detection is as simple as understanding when something has faulted. However, knowing something is broken is easier than knowing something is about to break. Is it better to know that your machine just crashed because the chip slagged to the motherboard, or that the temperatures in rack 043 are rising unexpectedly? Answer: both, but I hope I only learn the latter and not the former. Truly, there are too many things to monitor... hundred or thousands of metrics on each piece of equipment. I can't reasonable go in and configure good/bad thresholds on each one. I want anomaly detection. I want a system that I can say: "this looks right, tell me when it stops looking right." That, to me, is a much need companion to tradition fault detection.

To me, trending is much more than drawing graphs... it is about intelligent data correlation, regression analysis/curve fitting and looking into the past to see how much you fucked up getting where you are now -- in the vain hope that you learn from your mistakes and plan better next time.

Reconnoiter is an attempt to build these things. Building a system requires starting with pain (need), solid structure and plumbing (good engineering). So, reconnoiter is underway. And this post is in mid-step:

It started on OpenBSD, and added support for FreeBSD, Mac OS X, Linux.

As of changeset [292], we have Solaris/OpenSolaris support.

We have a pretty nice front-end for trending under construction, but it isn't there yet. We'll have numeric data combined with textual "event" data on the same graphs. All that convenient stuff. Here's the rather plain-Jane graph you get now (because some people won't even read a post if it doesn't have a pretty graph):

Honestly, I don't know what the value of this post is, but people around here keep telling me that people should be aware of an open-source tool like this, even if it isn't finished (read: usable) yet. I say it isn't usable yet, but on our development instances here, we monitor 2892 production metrics across two data centers and the load never peaks past 0.10. I'm pretty excited about where this is going. Honestly, my favorite part right now is that I can configure and control the noitd checking nodes via a telnet console and it acts as if it is a piece of network equipment rather than an "application" -- as it should be IMHO.

Dissecting today's Internet traffic spikes

nospam@example.com (Theo Schlossnagle) — Tue, 20 May 2008 13:56:59 -0400

Today's Internet has changed quite a bit from the Internet I used to know. The Internet has always been successful because of net neutrality. What's net neutrality? It's complicated, but essentially it means that anyone anywhere can publish with equal rights. These aren't the kind of rights people usually talk about... I'm not speaking of freedom of speech. Instead, I'm talking about content being simply bits. It doesn't matter if it comes from CNN or this blog, you as a reader can download the bits that make up the pages you see without bias or preferential treatment. This makes it darn easy to be a publisher and leads to a fabulous ecosystem with an overwhelming amount of varied content. However, with more content it is easy to recognize that much of it is utter trash. Yes. Yes. I know that one man's trash is another man's treasure. However, it presents opportunities for sites that help you navigate the wasteland.

Many popular sites today are popular because they link to articles and news items and photographs and movies all over the Internet; they are "interest aggregation services." And while the Internet has (for now) a decent preservation of net neutrality when it comes to simple web content, not all publishers are on equal footing. Not long ago, anyone could run a server anywhere (their basement) with DSL or cable or (gasp) dial-up -- now, the challenge is coping with unexpected attention.

Years ago, the site slashdot coined a term "slashdotted" which meant that a site received so much sudden traffic that service degraded beyond an acceptable point and the site was effectively unavailable. This often happened to sites that were at the end of small pipes (DSL, T1, etc.) and occasionally (though rarely) due to bad engineering. While slashdot might have coined the term, they simply don't have the viewership numbers that other large sites today have.

At the $DAYJOB, I work on sites that aren't on the end of T1 lines. Sites with gigabits or tens of gigabits of connectivity. Sites with 50 millions users. Sites powered by thousands of machines. I also work on sites that service millions of people from just a handful of machines (efficiency certainly has its advantages sometimes). I find it particularly interesting that already popular sites (with significant baseline bandwidth) are seeing these unexpected surges. For a long time, my blog has been on this same machine which is a vhost for several other web sites. I've had traffic spikes from places like slashdot, reddit, digg, etc. And, no surprise, I couldn't actually see the bandwidth jump on the graphs... 10Mbits to 11Mbs? That's not a spike.

Things are changing. Sites like Digg are becoming ever more popular and people are drawn to them as a means of sifting the waste of the Internet. This means as more people rely on Digg and Reddit and other similar sites, the number of unexpected viewers of your content can rise more sharply.

What does all of this mean? It means that the old rule of thumb that your infrastructure should see 70% resource utilization at peak is starting to falter. The typical trends used to look like this (this is last week's graph from a retail client with a user base of 3 million):

We see a nice peak, a nice valley. Thursday afternoon, we see a nice traffic spike. Well, this used to be what I called a traffic spike. Now, different services have different spike signatures. It resembles traffic model of classic Internet advertising, except that there is genuine interest and thus dramatically higher conversion rates. It's a simple combination of placement, frequency and exposure. Because content, unlike ad banners, exists for an extended period of time (sometimes forever), the frequency is very high. Digg and Reddit have excellent placement with very little exposure (things move out quickly). A site like CNN or NYTimes usually provides mediocre placement (unless you are on the front page) and excellent exposure.

Lately, I see more sudden eyeballs and what used to be an established trend seems to fall into a more chaotic pattern that is the aggregate of different spike signatures around a smooth curve. This graph is from two consecutive days where we have a beautiful comparison of a relatively uneventful day followed by long-exposure spike (nytimes.com) compounded by a short-exposure spike (digg.com):

The disturbing part is that this occurs even on larger sites now due to the sheer magnitude of eyeballs looking at today's already popular sites. Long story short, this makes planning a real bitch.

And the interesting thing is perspective on what is large... People think Digg is popular -- it is. The New York Times is too, as is CNN and most other major news networks -- if they link to your site, you can expect to see a dramatic and very sudden increase in traffic. And this is just in the United States (and some other English speaking countries)... there are others... and they're kinda big.

What isn't entirely obvious in the above graphs? These spikes happen inside 60 seconds. The idea of provisioning more servers (virtual or not) is unrealistic. Even in a cloud computing system, getting new system images up and integrated in 60 seconds is pushing the envelope and that would assume a zero second response time. This means it is about time to adjust what our systems architecture should support. The old rule of 70% utilization accommodating an unexpected 40% increase in traffic is unraveling. At least eight times in the past month, we've experienced from 100% to 1000% sudden increases in traffic across many of our clients.

I talk about scalability a lot. It's my job. It's my passion. I regularly emphasize that scalability and performance are truly different beasts. One key to scalability is that a "systems design" scales. Architectures are built to be able to scale, they are not built "at scale." It's just too expensive to build a system to serve a billion people (until you have a billion people). It's cheap to design a system to serve a billion people. Once you have a billion people accessing your site, you can likely justify executing on your design. Google is successful for this reason: their ideas scale and they can build into them as demand rises. On the flip side, traffic anomalies in the form of spikes are unexpected (by their definition) and scaling a system out to meet the unexpected demand is almost unreasonable. I would even argue that it is more of a performance-centric issue. I want every asset I serve to be as cheap to serve as possible allowing me to handle larger and larger spikes.

The reason I find all of this stuff interesting is that understanding performance and scalability, understanding the principles of scalable systems design and having sound and efficient processes for handling performance issues is becoming crucial for sites regardless of their size. This takes insight and practice and it reminds me of Knuth's famous saying:

We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.

That's all well and good, but which 97% of the time? My response to Knuth's statement (with which I completely agree) is:

Understanding what is and isn't "premature" is what separates senior engineers from junior engineers.

Let's add perspective on the word "sudden." Most network monitoring systems poll SNMP devices (like switches, load-balancers, and hosts) once every five minutes (we do this every 30 seconds in some environments). Some people say, "my site scales! bring it on." We see these spikes happen inside 60 seconds and they occasionally induce a ten-fold increase over trended peaks. Often times, this spike can be well underway for several minutes before your graphing tools even pick up on it. Then, before you have time to analyze, diagnos and remediate... poof... it's gone. Be careful what you wish for.

This, in many ways, is like a tornado. Our ability to predict them sucks. Our responses are crude and they are quite damaging. However, predicting these Internet traffic events isn't even possible -- there are no building weather patterns or early warning signs. Instead we are forced to focus on different techniques for stability and safety. The idea of a DoS, a DDoS or the sometimes similar signature of a sudden popularity spike doesn't increase my heart rate anymore -- it's just another day on the job. However, I thought I'd share the four guidelines that I believe are key to my sanity in these situations:

Be Alert: build automated systems to detect and pinpoint the cause of these issues quickly (in less than 60 seconds).
Be Prepared: understand the bottlenecks of your service systemically. Understanding your site inside and out. Contemplate how you would respond if a specific feature or set of features on your site were to get "suddenly popular."
Perform Triage: understand the importance of the various services that make up your site. If you find yourself in a position to sacrifice one part to ensure continued service of another, you should already know their relative importance and not hesitate in the decision.
Be Calm: any action that is not analytically driven is a waste of time and energy. be quick, not rash.

Back to those other countries... Enter China and their recently lessened censorship and we have a looming tidal wave for smaller sites that achieve sudden popularity. Spikes of several hundred megabits per second are difficult to account for when your normal trend is around twenty megabits per second. The following graph is traffic induced from a link from a popular foreign news site (that I can't read). I call it: "ouch:"

BWPUG Meetup Reminder

nospam@example.com (Theo Schlossnagle) — Tue, 13 May 2008 20:49:56 -0400

Hi all!

Just a friendly reminder that we'll be having our first meetup tomorrow as planned. I thought as a good kick-off we could all collaboratively share what we do with PostgreSQL. We'll start off with a whirlwind tour of how OmniTI uses PotsgreSQL, taking a brief look at ZFS, DTrace and large datasets. After that I think it would be good to get to know each other -- maybe we'll hit a local pub afterwards!

I look forward to seeing you there!

Meetup starts at 6:30pm
7070 Samuel Morse Dr. Ste 150
Columbia, MD 21046

If you have issues getting in the building, ring me on my cell -- it will be posted on the doors.

Best regards,

Theo

OSCON 2008: And now for something completely different.

nospam@example.com (Theo Schlossnagle) — Sun, 27 Apr 2008 22:02:28 -0400

I just registered for OSCON. They say I should advertise that I am a speaker. Here goes.

For the last several years, I've presented multiple talks at the O'Reilly Open Source Conference. My Scalable Internet Architectures talk has been quite popular and drawn large crowds. It is an interesting talk as it doesn't really change with time. As I say, "if principles of good engineering changed frequently, I'd never drive on bridges." The talk is about sound engineering approaches to building really large consumer-facing websites. Almost all of it is open-source centric, which is why it fits so well at OSCON. While my Scalable talk was not accepted this year, I've got another talk lined up that will rock your world.

I am quite excited that my other proposal was accepted. This year I will be giving a session about using DTrace to perform "full-stack" introspection.

Using DTrace we will deep dive into the amazingly cool questions one can ask. Is my application really hitting disk? If so, what line of code is causing it? My process is being descheduled by the kernel, why? I have 100 Apache process and some randomly segfault, how do I get a stack trace when that happens? The app I am running doesn’t have the right debugging output, I need to know more!

DTrace is an oracle. The value of the answers depends on the quality of the questions. Learn to ask good question and prepare to be amazed at the possibilities.

I've given a variation on this presentation at a few places now (both internal to OmniTI and external) and had really positive feedback. I'll be taking these prior presentations and polishing them up for a 45 minute escapade that will open your eyes to new possibilities. DTrace is an amazing tool and once you get used to it, you can really take it for granted. I do. When people watch the presentation and say "by the power of Greyskull," I know I've made my point.

Come to OSCON. Immerse yourself in technology.