Seibel: We talked a bit about some of the really hideous bugs youhad to track down like that thing with GDB. But let’s talk a little bitmore about debugging. For starters, what’s your preferred debuggingtool? Print statements? Symbolic debuggers? Formal proofs ofcorrectness?

Zawinski: That’s changed over the years a lot. When I was using theLisp machines it was all about running the program and stopping it andexploring the data—there was an inspector tool that let you browsethrough memory and I changed it so basically the Lisp listener becamean inspector. So anytime it printed out an object there was acontext menu on it so you could click on this thing here and have thatvalue returned. Just to make it easier to follow around chains ofobjects and that sort of thing. So early on that was how I thoughtabout things. Getting down in the middle of the code and chasing itaround and experimenting.

Then when I started writing C and using GDB inside of Emacs I kind oftried to keep up that same way of doing things. That was the modelwe built Energize around. And that just never seemed like it workedvery well. And as time went by I gradually stopped even trying to usetools like that and just stick in print statements and run the thingagain. Repeat until done. Especially when you get to more and moreprimitive environments like JavaScript and Perl and stuff like that, it’sthe only choice you have—there aren’t any debuggers.

People these days seem confused about the notion of what a debuggeris. “Oh, why would you need that? What does it do—put printstatements in for you? I don’t understand. What are these strangewords you use?” Mostly these days it’s print statements.

Seibel: How much of that was due to the differences between Lispand C, as opposed to the tools—one difference is that in Lisp you cantest small parts—you can call a small function you’re not sure isworking right and then put a break in the middle of it and then inspectwhat’s going on. Whereas C it’s like, run the whole program in all ofits complex glory and put a break point somewhere.

Zawinski: Lisp-like languages lend themselves more to that than C.Perl and Python and languages like that have a little more of the Lispnature in that way but I still don’t see people doing it that way verymuch.

Seibel: But GDB does give you the ability to inspect stuff. Whatabout it makes it not usable for you?

Zawinski: I always found it unpleasant. Part of it is just intrinsic tobeing C. Poking around in an array and now I’m looking at a bunch ofnumbers and now I have to go in there and cast the thing to whateverit really is. It just never managed that right, the way a better languagewould.

Seibel: Whereas in Lisp, if you’re looking at a Lisp array, they’ll justbe printed as those things because it knows what they are.

Zawinski: Exactly. It always just seemed in GDB like bouncing up anddown, the stack things would just get messed up. You’d go up thestack and things have changed out from under you and often that’sbecause GDB is malfunctioning in some way. Or, oh well, it wasexpecting this register to be here and you’re in a different stack frame,so that doesn’t count anymore.

It just always felt like I couldn’t really trust what the debugger wasactually telling me. It would print something and, look, there’s anumber. Is that true or not? I don’t know. And a lot of times you’dend up with no debug info. So you’re in a stack frame and it looks likeit has no arguments and then I’d spend ten minutes trying toremember the register that argument zero goes in is. Then give up,relink and put in a print statement.

It seemed like as time went by the debugging facilities just kept gettingworse and worse. But on the other hand now people are finallyrealizing that manual memory allocation is not the way to go; it kind ofdoesn’t matter as much any more because the sorts of reallycomplicated bugs where you’d have to dig deep into data structuresdon’t really happen as often because those, often, in C anyway, werememory-corruption issues.

Seibel: Do you use assertions or other more or less formal ways ofdocumenting or actually checking invariants?

Zawinski: We went back and forth about what to do aboutassertions in the Netscape code base. Obviously putting in assertstatements is always a good idea for debugging and like you said, fordocumentation purposes. It expressed the intent. We did that a lot.

But then the question is, well, what happens when the assertion failsand you’re in production code; what do you do? And we sort ofdecided that what you do is return zero and hope it keeps going.Because having the browser go down is really bad—it’s worse thanreturning to the idle loop and maybe having leaked a bunch of memoryor something. That’s going to upset people less than the alternative.

A lot of programmers have the instinct of, “You’ve got to present theerror message!” No you don’t. No one cares about that. That sort ofstuff is a lot easier to manage in languages like Java that actually havean exception system. Where, at the top loop of your idle state, youjust catch everything and you’re done. No need to bother the userwith telling them that some value was zero.

Seibel: Did you ever just step through a program—either to debug itor, as some people recommend, to just step through it once you’vewritten it as a way of checking it?

Zawinski: No, not really. I only really do stepping when I’m debuggingsomething. I guess sometimes to make sure I wrote it right. But notthat often.

Seibel: So how do you go about debugging?

Zawinski: I just eyeball the code first. Read through it until I think,well, this can’t happen because that’s going on right there. And then Iput in something to try and resolve that contradiction. Or if I read itand it looks fine then I’ll stop in the middle or something and seewhere it is. It depends. It’s hard to generalize about that.

Seibel: As far as the assertions—how formally do you think? Somepeople just use ad hoc assertions—here’s something that I thinkshould be true here. And some people think very formally—functionshave preconditions and postconditions and there are global invariants.Where are you on that scale?

Zawinski: I definitely don’t think about things in a mathematicallyprovable way. I’m definitely more ad hoc than that. You know, it’salways helpful when you’ve got inputs to a function to at least have anidea in your head what their bounds are. Can this be an empty string?That sort of thing.

Seibel: Related to debugging, there’s testing. At Netscape, did youguys have a separate QA group or did you guys test everythingyourselves?

Zawinski: We did both. We were all running it all the time, which isalways your best front-line QA. Then we had a QA group and theyhad formal tests they went through. And every time there was a newrelease they’d go down the list and try this thing. Go to this page, clickon this. You should see this. Or you shouldn’t see this.

Seibel: What about developer-level tests like unit tests?

Zawinski: Nah. We never did any of that. I did occasionally for somethings. The date parser for mail headers had a gigantic set of testcases. Back then, at least, no one really paid a whole lot of attention tothe standards. So you got all kinds of crap in the headers. Andwhatever you’re throwing at us, people are going to be annoyed iftheir mail sorts wrong. So I collected a whole bunch of examplesonline and just made stuff up and had this giant list of crappilyformatted dates and the number I thought that should turn into. Andevery time I’d change the code I’d run through the tests and some ofthem would flip. Well, do I agree with that or not?