Other people I eventually figured out what motivated them. One guy wasreally good at tinkering and getting a prototype working. He wrote sysadminPerl. He could wire stuff together, write shell scripts, and write really badPerl and really bad C, but kind of get it working. Then we would be like,“Holy crap, you researched all this stuff, you got all these componentstalking to each other?”

We were setting up a voice bridge to LiveJournal so you record somethingand post it to LiveJournal. There were just so many moving parts involved. Ithought it was painful as hell. He loved it. He figured it all out and got itworking. Then we just rewrote it all. And we figured out that was the way itworked with him. He figured out the interface and we would fix it all up.Once I figured out that was his role, we got along great.

Seibel: So you’ve hired for your own company, and I assume you’ve beeninvolved in hiring at Google. How do you recognize a great programmer?

Fitzpatrick: I often look for people that have done, like, a lot of stuff ontheir own that wasn’t asked of them. Not just their school project or justwhat their previous employer had them do. Somebody who was passionateabout something and had some side project. How did they maintain it andhow serious did they get with it? Or do they do a lot of quick hacks andabandon them?

Seibel: Do you have favorite interview questions?

Fitzpatrick: One of the ones I’ve given a few times because it was on myAP programming test is given two decimal numbers as strings of arbitrarylength, multiply them. There are a lot of different ways that they could do it.If they’re really good at math—like I’m not—they can find some clever waysto do it really efficiently. Worst case, they can make a class that does justaddition repeatedly.

I tell them from the beginning, “Don’t stress out. You don’t have to do itefficiently. Just get it done somehow.” Some people stress out and have noclue where to begin. That’s kind of a bad sign. The worst case, you justimplement the algorithm you do in grade school.

I actually wrote a program in grade school to do my long division andmultiplication and show the work. Including all the steps and where to crossout. So then we would get these problems, like ten per page or something,and I would type it into the computer and then just reproduce the problemsin scribbles. I did the same thing in chemistry to find the orbitals ofelectrons. But the thing I find is by writing a program to cheat, you learnbecause you have to learn it really in depth to write that program.

Seibel: Do you think that would work for anyone? Instead of teaching kidslong division, should we teach them how to program and then say, “OK,now your task is to write a program to do this long-division procedure”? Bythe time they’ve actually written that program, they’ll understand division.Or does that only work if you have some natural inclination that way?

Fitzpatrick: It worked for me. A lot of times, someone could teach yousomething and you’re like, “Yeah, yeah, sure. I understand.” You deludeyourself but once you actually have to get down and do it at a real level andunderstand all the corner cases, it forces you to actually learn the thing. ButI don’t know if that would work for everyone.

Seibel: Google has a bit of a reputation, as Microsoft also does, ofinterviewers using puzzle questions.

Fitzpatrick: I think those are kind of banned. Or strongly discouraged.Maybe some people still do them, but I think, in general, they’rediscouraged.

Seibel: What did they ask you in your interview?

Fitzpatrick: One question was, imagine you have a bunch of computers ona switch and they turn on the whole rack; come up with an algorithm soevery machine on the rack knows the status of all the other ones aboutwhether they’re on or off. So basically a presence thing. That was prettymuch the constraint. Basically, they described Ethernet: you could send abroadcast to everyone, or you could send it to a specific MAC address. So Ijust kind of walked through all the different strategies to minimizebandwidth and to minimize latency discovering when something’s dead. Thatwas a fun one.

Seibel: What’s the worst bug you ever had to track down?

Fitzpatrick: I try not to remember them. I hate it when it’s somethingwhere your assumptions are so far off. The other day—this is definitely notan example of the worst one ever—I spent 90 minutes debugging somethingbecause I was writing to one output file and reading another file named thesame thing but with one path component missing. I kept rerunning this hugeMapReduce and seeing the output and putting it in GDB and steppingthrough it. “What the fuck? It’s not changing!” Finally I looked at the pathsand I was like, “Holy crap.” I don’t know why I spent 90 minutes on it; I wasso obsessed that I didn’t step back and check, is my command line correct?

There’s a lot of that. We always had some good stuff with Perl like the $_isn’t lexically scoped. So if you fuck with $_ in a sort, you can mess withsomebody else’s far away. So we had this bug that took us forever and wehad a bunch of corruption going on. We finally figured that out. Then Iaudited all our code we had a new policy of “never do this.”

Seibel: What are your debugging tools? Debuggers? Printlns? Somethingelse?

Fitzpatrick: Println if I’m in an environment where I can do that.Debugger, if I’m in an environment that has good debuggers. GDB is reallywell maintained at Google and is kind of irreplaceable when you need it. Itry not to need it too often. I’m not that great at it, but I can look aroundand kind of figure things out generally. If I have to go in there, I generally canfind my way out. I love strace. Strace, I don’t think I could live without. If Idon’t know what some program is doing, or what my program is doing, Irun it under strace and see exactly what’s happening. If I could only haveone tool, it would probably be that. All the Valgrind tools, Callgrind and allthat, those are good.

But a lot of times lately, if there’s something weird going on, I’m like, “OK,that function is too big; let’s break that up into smaller parts and unit-testeach one of them separately to figure out where my assumptions are wrong,rather than just sticking in random printlns.”

Then maybe in the process of refactoring, I have to think about the codemore, and then it becomes obvious. I could, at that point, go back to thebig, ugly state where it was one big function and fix it but I’m alreadyhalfway there; I might as well continue making it simpler for the nextmaintainer.

Seibel: How do you use invariants in your code? Some people throw in adhoc asserts and some people put in invariants at every step so they canprove formal properties of their programs, and there’s a big range in themiddle.

Fitzpatrick: I don’t go all the way to formal. My basic rule is, if it couldpossibly come from the end user, it’s not a run-time crash. But if it is mycode to my code, I crash it as hard as possible—fail as early as possible.

I try to think mostly in terms of preconditions, and checking things in theconstructor and the beginning of a function. Debug checks, if possible, so itcompiles away. There are probably a lot of schools of thought and I’mprobably not educated about what the proper way to do it is. There arelanguages where all this stuff is actually a formal part of the language. Prettymuch all the languages I write in, it’s up to you.

Seibel: You wrote once that optimization is your favorite part ofprogramming. Is that still true?

Fitzpatrick: Optimization is fun because it’s not necessary. If you’redoing that, you’ve got your thing working and nothing else is more importantand you’re either saving money or doing it because it’s like a Perl golfcontest—how short can I make this or how much faster. We would identifyhotspots in LiveJournal, and I would send out some contests. “Here’s some code.Here’s the benchmark. Make it fast.” I sent our load balancer’s header parsing.We were all writing crazy regexps that didn’t backtrack and tried to capturethings with the most efficient capture groups. And we were all competing,getting faster and faster and faster. Then one guy comes over the next day. Hehad written it all in C++ with XS, and so he was like, “I win.”