At Google, I think it’s easy because we have this “launch early and often”philosophy. And because of the way the company is, for a number ofreasons: one, most of our products we don’t charge any money for so it’seasy to say, well, go ahead and ship it; how much could they complain? Theother one is we’re not stamping CDs and putting them in a box so if there’ssomething that’s not complete today or even if it has a bug, it’s not adisaster. Most of the software is on our servers so we can fix it tomorrowand everybody gets the update instantly. We don’t have this nightmare ofinstalling updates. So it makes it easier for us to say, “We’re just going tolaunch things and get some feedback from the users and fix the stuff thatneeds to be fixed and don’t worry about the other stuff.”

Seibel: If you’re working on the design of a big system what are the toolsyou use—do you sit down with a pad of graph paper or a UML drawingtool?

Norvig: I never liked any of these UML-type tools. I always thought, “If youcan’t do it in the language itself that’s a weakness of the language.” I think alot of what you’re doing, you’re dealing at a higher level. At Google a lot ofwhat we do is figuring out how to break things up and parallelize them.We’re going to necessarily run this on multiple machines but we’ve got somany users and, for many applications, so much data; how’s that going towork? So we’re thinking more at the level of machines and racks ofmachines rather than at the level of functions and interactions. Once youget that straightened out, then you can start diving into individual functionsand methods.

Seibel: And so that level of description is just prose?

Norvig: Yeah, mostly. Sometimes people draw pictures. They’ll say, “We’llhave this server here that will be serving these kinds of requests and thenit’s connected to this server and then we’ll use these various tools forstorage and big distributed hash tables and other types of things. We’llchoose these three tools off the shelf and then we’ll argue about whetherwe have to build a new one; which of these existing ones works or do weneed something else?”

Seibel: And how do you evaluate that kind of design?

Norvig: You show it to the people who’ve done it before. They say, “Oh, itlooks like you’ll need a cache here—it’s going to be too slow but thereshould be a lot of repeat requests, so if you install a cache of this size herethat should help a lot.” You have a design review where people look over itand say whether they think it makes sense and then you start building it andtesting it.

Seibel: And you guys have formal design reviews? You worked at NASAwhere they had a very formal design review.

Norvig: Nothing formal like NASA. The stakes are lower for us because, asI say, it’s easy for us to have a failure and recover from that. At NASAusually the first failure is fatal so they were much more careful. We don’tworry about that much. It’s more of a consultation, I think, rather than areview.

There are people who officially read design documents and comment onthem. You go through that and get your design approved. But it’s still much

Norvig: I think it was something that had always been done and so peopleaccept it. Well, I shouldn’t say that completely. Some people it takes a whileto get used to it. One of the typical failure cases is a new hire comes in andthey’re not used to doing this kind of thing so they just start anexperimental branch and they have all their code in there and you keep ontelling them, “Gee, you don’t have any check-ins yet.” And they say, “Yeah,yeah, yeah, I’m just cleaning it up—I’ll check it in tomorrow.” And thenanother week goes by and another week goes by and eventually they havethis one gigantic check-in. And then it’s a problem that too much time hasgone by, it’s hard to evaluate it all at once, and some of the things they’recomparing against have changed out from underneath them. Then they seewhat a headache it is and they learn not to do that.

Seibel: So that’s on the coding side. Are there skills that the reviewersdevelop?

Norvig: There certainly are people who are known for being better reviewersthan others. There’s a trade-off of when you submit a review—do you try to getsomebody who will give you a lot of good feedback or do you try to get somebodywho will just say “OK” as quickly as possible?

Seibel: So what makes the better reviewers better?

Norvig: Well, that they catch more things. Some of it is the trivial stuff ofyou indented the wrong number of spaces or whatever but some of it is, “Ithink this design would be cleaner if you moved this from here over tothere.” So some people will do more of that and others won’t bother.

Seibel: Sort of related to that, does every good programmer turn into agood architect when they grow up? Or are there some people who arebrilliant coders but only at a certain level and they should never be allowedto do bigger designs?

Norvig: I think different people have different skills. One of our bestsearch people is by no means our best programmer, in terms of how the codelooks. But if you say, “Here’s this new factor that we have—you know, how manytimes people click on this page after they’ve done such and such—how do we foldthat into our search results?” He’ll say, “Oh, on line 427 there’s thisvariable alpha and you should take this new factor and raise it to the secondpower and multiply it by 1.5 and add it to alpha.” Then you experiment for acouple months trying different things and you find out he was right except itshould have been 1.3 instead of 1.5.

Seibel: So that suggests he just has this very good mental model of how thesoftware works.

Norvig: He understands the code perfectly. Other people can write codebetter but he understands all the implications of what goes where.

Seibel: Do you think those are related? It often seems that people whowrite the worst spaghetti code are the ones who can hold the most in theirhead—that’s the only way they could possibly write code like that.

Norvig: Yeah, I think that may be.

Seibel: So the reviews here are less formal here than at NASA. What arethe other differences between the “engineering” and “hacker” ethos, in thebest sense of both those words?

Norvig: One big difference is organizational structure and how software isaccepted. Google was founded as a software company, and they went outand hired a CEO who has a PhD in computer science from Berkeley, hired aVP of Sales who has a computer engineering background—it’s throughoutthe whole company. At NASA they’re rocket scientists! They aren’tsoftware guys. They say, “Software is this necessary evil. Straight line code, Ican sort of understand; if it’s got a loop in it, that’s kind of iffy. Then ifthere’s a branch statement inside the loop, ooooh, that’s getting away fromwhat I can solve with a differential equation in control theory.” So they’redistrustful.

Seibel: As well they should be!

Norvig: As well they should be, yeah. And they’re distrustful of innovation.So you can say, “Look at this great new prototype I have,” and they’ll say,“That’s fantastic; I’d love to fly that on my mission—as soon as it’s beenproven on two other missions.” And you go to everybody else and they allsay the same thing.

Don Goldin came in as NASA administrator and he said, “We’ve got to dothis better, faster, cheaper. These space missions cost too much. It’d bebetter to run more missions and some of them would fail but overall we’dstill get more done for the same amount of money.” And that wasundeniably true. Unfortunately it was not politically true. It’s not OK to losea spacecraft. Because the public understands very well, NASA lost aspacecraft. They don’t really know that there’s any difference between a$100 million spacecraft and a billion dollar spacecraft. It’s not like you get tolose ten of the $100 millions instead of one billion. So it was never quitetrue.