Then I worked at ETI for four years or so until the company startedevaporating. We were using TI Explorer Lisp machines at ETI so Ispent a lot of my time, besides actually working on the expert system,just sort of messing around with user-interface stuff and learning howthose machines worked from the bottom up. I loved them—I loveddigging around in the operating system and just figuring out how it allfit together.

I’d written a bunch of code and there was some newsgroup where Iposted that I was looking for a job and, oh, by the way, here’s a bunchof code. Peter Norvig saw it and scheduled an interview. My girlfriendat the time had moved out here to go to UC Berkeley, so I followedher out.

Seibel: Norvig was at Berkeley then?

Zawinski: Yeah. That was a very strange job. They had a whole bunchof grad students who’d been doing research on natural languageunderstanding; they were basically linguists who did someprogramming. So they wanted someone to take these bits and piecesof code they’d left behind and integrate them into one thing thatactually worked.

That was incredibly difficult because I didn’t have the background tounderstand what in the world they were doing. So this would happena lot: I’d be looking at something; I’d be completely stuck. I have noidea what this means, where do I go from here, what do I have to readto understand this. So I’d ask Peter. He’d be nice about it—he’d say,“It totally makes sense that you don’t understand that yet. I’ll sit downand explain it to you Tuesday.” So now I’ve got nothing to do. So Ispent a lot of time working on windows system stuff and pokingaround with screen savers and just the kind of UI stuff that I’d beendoing for fun before.

After six or eight months of that it just felt like, wow, I’m really justwasting my time. I’m not doing anything for them, and I just felt like Iwas on vacation. There have been times when I was working really alot when I’d look back at that and I’m like, “Why did you quit thevacation job? What is wrong with you? They were paying you to writescreen savers!”

So I ended up going to work for Lucid, which was one of the tworemaining Lisp-environment developers. The thing that really made medecide to leave was just this feeling that I wasn’t accomplishinganything. And I was surrounded by people who weren’t programmers.I’m still friends with some of them; they’re good folks, but they werelinguists. They were much more interested in abstract things thansolving problems. I wanted to be doing something that I could point toand say, “Look, I made this neat thing.”

Seibel: Your work at Lucid eventually gave rise to XEmacs, but whenyou went there originally were you working on Lisp stuff?

Zawinski: Yeah, one of the first projects I worked on was—I can’teven remember what the machine was, but it was this 16-processorparallel computer and we had this variant of Lucid Common Lisp withsome control structures that would let you fork things out to differentprocessors.

I worked a little bit on the back end of that to make the overhead ofspawning a thread lower so you could do something like a parallelimplementation of Fibonacci that wasn’t just completely swamped bythe overhead of creating a new stack group for each thread. I reallyenjoyed that. It was the first time I’d gotten to use a fairly bizarremachine like that.

Before that I was bringing up Lisp on new machines. Which meansbasically someone’s already written the compiler back end for the newarchitecture and then they’ve compiled the bootstrap piece of code.So I’ve got this file full of binary, supposedly executable code for thisother machine and now I’ve got to decipher their loader format sothat I can write a little C program that will load that in, make the pageexecutable, and jump to it. Then, hopefully, you get a Lisp prompt andat that point you can start loading things in by hand.

Which for every architecture was bizarre, because it’s neverdocumented right. So it’s a matter of compiling a C program and thenlooking at it byte by byte—byte-editing it in Emacs. Let’s see whathappens if I change this to a zero; does it stop running?

Seibel: When you say it wasn’t documented right, was it that itwasn’t documented correctly, or it wasn’t documented at all?

Zawinski: It was usually documented and it was usually wrong. Ormaybe it was just three revisions behind—who knows? But at somepoint you tweak a bit and then it would no longer believe this was anexecutable file and you had to figure out what was going on there.

Seibel: So that’s something that comes up all the time, from thelowest-level systems programming to high-level APIs, where things justdon’t work the way you expect or the way they are documented.How do you deal with that?

Zawinski: Well, you just come to expect it. The sooner you realizethat your map is wrong, the sooner you’ll be able to figure out whereit went wrong. In my case, I’m trying to produce an executable file.Well, I know the C compiler will produce one. Take the good one andstart converting it into the bad one until it stops working. That’sprimary tool of reverse engineering.

The hardest bug I’ve ever fixed, I think, was probably during thatperiod at Lucid. I’d gotten to the point where it’s running theexecutable and it’s trying to bootstrap Lisp and it gets 500 instructionsin and crashes. So there I am leaning on the S key, stepping throughtrying to figure out where it crashes. And it seems to be crashing at adifferent place each time. And it doesn’t make any sense. I’m readingthe assembly output of this architecture I only barely understand.Finally I realize, “Oh my god, it’s doing something different when Istep; maybe it’s timing-based.” Eventually I figure out that what’s goingon is this is one of the early machines that did speculative execution. Itwould execute both sides of the branch. And GDB would always takethe branch if you single-stepped past a branch instruction. There was abug in GDB!

Seibel: Nice.

Zawinski: Right. So then that takes me down into, “Oh my god; nowI’m trying to debug GDB, which I’ve never looked at before.” The wayto get around that is you’re coming up to a branch instruction and youstop before the branch, set a break point on both sides, and continue.So that was how I proved that really was what was going on. Spent likea week trying to fix GDB; couldn’t figure it out. I assume a registerwas getting stomped somewhere, so it always thought there was apositive value in the branch check or something like that.

So I changed the step-by-instruction command to recognize when itwas coming up on a branch instruction and just say, “No, don’t dothat.” Then I can just lean on the S key and it would eventually stopand I’d set the break point by hand and continue. When you’redebugging something and then you figure out that not only is the mapwrong but the tools are broken—that’s a good time.

Working on Lisp systems was especially weird because GDB wascompletely nonfunctional on Lisp code because it doesn’t have anydebug info—it’s written by a compiler GDB has never heard of. I thinkon some platforms it laid out the stack frames in a way GDB didn’tunderstand. So GDB was pretty much an assembly stepper at thatpoint. So you wanted to get out of the GDB world just as quickly asyou could.

Seibel: And then you’d have a Lisp debugger and you’d be all set.

Zawinski: Right, yeah.

Seibel: So somewhere in there Lucid switched directions and said,“We’re going to make a C++ IDE”.

Zawinski: That had been begun before I started working there—itwas in progress. And people started shifting over from the Lisp side tothe Energize side, which is what the development environment wascalled. It was a really good product but it was two or three years tooearly. Nobody, at least on the Unix side, had any idea they wanted ityet. Everyone uses them now but we had to spend a lot of timeexplaining to people why this was better than vi and GCC. Anyway, I’ddone a bit of Emacs stuff. I guess by that point I’d already rewritten theEmacs byte compiler because—why did I do that? Right, I’d writtenthis Rolodex phone/address-book thing.