Skip to: categories | main content
Theo's Contributions to Technological Surreality
About meDTrace kicks ass. Everyone knows I love ZFS, but in my opinion DTrace is the most significant and disruptive systems technology in at least 10 years.
Today I gave my talk at ApacheCon; it was well received. I hope that I inspired some people to go use DTrace and look in their systems' underpants.
Here are my slides and the scripts I used for my live demo.
I'll update this entry with the video as soon as I have the link.
I mentioned a Linux port underway. I got the name wrong when I talked about it (sorry Paul!). The person working on the port is Paul Fox, the author of the CRiSP editor. For more information, check out his blog.
I met Bryan Cantrill a few years back at OSCON where he gave a session on DTrace. As any reader of my blog would know, I'm a big DTrace fan. Bryan's presentation style was passionate and animated and I was quite convinced there was an illegal substance involved. I've come to learn that style is Bryan's style.
Since then, I've had a few opportunities to dialogue with Bryan and on every occasion he reinforced my opinion that he is one of the more insightful minds in today's computer engineering field. His critique and vision has an element of clarity and completeness to it that is far too rare. Combine that with truly elegant written prose and I am unable to resist consuming what he writes.
Bryan's latest communication on transactional memory is an exquisite example of poise in both thought and communication. At one point he states that "experience has taught me to be wary of crooked problem statements." I must admit that in my efforts to play devil's advocate I often construct straw-man arguments and unjustified counterpoints. I do this in an effort to force lead my counterpart to conciseness and deeper well-founded justification in their argument. Backing out of a solid justification is, at the very least, revealing.
I took two things away from his prose. First, his output is very worth digesting. Second, when I present solutions to problems I should vehemently avoid perfunctory problem definition and invest equal efforts in the completeness of both the problem statement and the solution proposed.
There is a pleasant dream where the world is black and white, problems are discreet, and solutions are straightforward. People that live in this dream are an obstacle unto themselves. The harsh reality is that we struggle daily to simplify things because they tend to be so complex. Most choices lead to outcomes that have both positive and negative aspects. This includes the choice of inaction (wrongfully referred to as "not choosing"). When presented with options, by and large people tend to make decent choices. The conundrum is identifying good options when none are presented. This skill, in many people, just plain sucks.
There is an old proverb: "Give a man a fish and you have fed him for today. Teach a man to fish and you have fed him for a lifetime." Perhaps I am oversimplifying fishing, but I struggle to adapt this proverb to systems operations. Perhaps it is due to the intangible nature of computing, but the one-step-closer proverb fits better (at least in my twisted brain): "Fix a man's car, and he can drive to the repair shop. Teach man to fix his car and he can go where he pleases." Why is this better? Perhaps it is just how my mind works. Perhaps it is because of the way symptoms present:
Dave: my car has issues.
Carl: what's wrong with it?
Dave: not sure.
Carl: why do you think it has issues?
Dave: It seems sluggish when pulling onto the interstate.
Carl: so it accelerates inadequately?
Dave: I guess.
Carl: What kind of car do you have again?
Dave: a brown one.
Carl: Uh, I mean make and model.
Dave: a Ronda Jackel.
Carl: is that carbureted or fuel injected?
Dave: um... um... it's got carbs.
Carl: really? what year?
Dave: oh, 2007.
Carl: okay, it's fuel injected.
Dave: oh, yeah.. That's right.. it's the Jackel GT, with the sports pack.
Carl: That has low-profile tires; are they properly inflated?
Dave: I think so, they look good.
Carl: when was the last time you checked?
Dave: last week, when I filled up. It costs like $58 to fill up. Gas prices are crazy!
Carl: *thinks*
Carl: Wow, where did you find high-test gas at that price?
Dave: Oh, I put 87 octane in that... that expensive stuff is a sham...
Carl: really? who told you that?
Dave: Alan
Carl: Who's Alan?
Dave: This great guy who taught me to fish.
Most people simply have no idea how their systems operate. They just know they are supposed to "go." Now, the purpose of this rant is really not to complain about car owners and their lack of knowledge (I'm one of them!), but rather to ask a wider audience about techniques to make the Carl's of the world better at their jobs.
Computing systems, like cars, can be immensely complex systems. When things go wrong, as they so often do, I feel you need someone who knows every single part. Not only should they know the part, they should know who built the part, why they built the part, when it was introduced into the system, what component it obsoleted, and what general advantages (and disadvantages) the newer part has over the older part. In car-speak, you should know who invented fuel-injection and when the industry transitioned, what advantages (and disadvantages) FI has over carburetor-based design. That should extend to every other part of the car down to the rubber on the tires and the stitching method used on the seat upholstery.
Note that I said "I feel you need someone..." I know a few people that can walk into a room with systems engineers and (knowing nothing about systems operations) assess a situation, ask a handful of high-level questions that results in at least two engineers slamming their heads into their desks. I call these people "Critical Thinkers." Critical Thinkers have the ability to critically think (surprise surprise), as well as understand other people's thought processes and deduce where they did not think critically.
Critical Thinkers are hard to come by, but they are out there. People with experience in computer science and operations are a little less hard to come by. It would make sense to take a Critical Thinker and teach them systems. However, to be really good at operations work, it takes years (probably five to ten). What is more challenging is finding a good Critical Thinker that actually wants to work in systems operations. What I constantly strive to do is mold people with solid technical skills and knowledge into critical thinkers. This brings us to the heart of this post.
I can hardly articulate my method of teaching critical thinking skills. Much of it is teach-by-example, but that has serious shortcomings in critical thinking scenarios. I could really use some breadth in my teaching/mentoring techniques. Does anyone have any good training materials that could be used to develop critical thinking skills? How do you cultivate and improve intuition?
I'm exciting that the Universidade Federal de Goiás or Brazil has a course called: "Scalable Architectures for Large Scale Applications" in their Computer Science department.
What's more? I'm very honored to learn that they are using my book as a part of their curriculum. I've added a venue for recruiting Site Reliability Engineers at OmniTI.
Two weeks ago was a whirlwind. I thought I'd catch up last week and blog about it, but it was so taxing that my fatigue combined with three germ monsters at home resulted in a severe cold that knocked me out.
4:28am Amtrak BWI to NYP. I had the privilege and honor of visiting Club 101 in New York, NY to speak to the New York CTO Group. I spoke on the topics of (you guessed it) scalability and performance in Internet applications. I did my best to integrate some thoughts on cloud computing into the discussion. As is typical with a short time frame and a first presentation -- I have a lot of things I'd change. For one, I believe I came across as far too negative toward cloud computing. I considered the amount of published hype around cloud computing and I wanted to provide a reality check. Many thinks to Geir Magnusson and Igor Shindel. This was a truly wonderful experience and I can only hope that the audience was as impressed with me as I was with them.
6:50am flight from BWI to BUF. I went on-site with Ciprian to a client in Buffalo. Various documents prevent me from discussing any details. Suffice it to say, it is an interesting problem and truly challenging technical and business problem solving on a day-to-day basis. The point? I leave there tired. Heading back two days this coming week.
2:30pm Amtrak from BWI to NYP. Jason Dixon and I went to NYCBSDCON. I had a speaking engagement on Sunday morning to reveal (first public presentation of) Reconnoiter. I think it went really well. I enjoyed meeting a lot of new people with different perspectives. It was an interesting crowd and a refreshing experience. Being as tired as I was and around so many chain smokers made me ill on Sunday -- that was no fun.
I was hoping I could catch up on things last week. I'm preparing for a lot of action on the Buffalo front, a new person is starting (tomorrow) that will head up OmniTI's internal accounting and I toured many office spaces in search of new offices for the ever growing OmniTI. Running OmniTI takes pretty much all of my time away from home. I need to learn that adding more things just hurts.
Murphy certainly had his sights on me. I promised my wife I'd take a day of much needed vacation on Friday. I managed to get sick on Thursday which climaxed on Saturday morning. On top of that, I did my business real-estate tour on Friday morning -- my sick vacation day. I think I need a do-over.
Last night, I finally started to feel better. I'm sure the bottle of (local) La Grange Norton red wine and the outdoor fire (thanks Katherine!) helped a lot.
Oh yeah... In two weeks I'll be in New Orleans at ApacheCon! I hope to see you there!

