Seibel: What enables that kind of intuition?

Cosell: On the systems I’m very good with like that, like the IMP systemwhen I had it all in my head, or the PDP-1 time-sharing system, even thoughthe system is a multiprogramming, multilayered, interrupt-driven system, Ihave all the dynamics of the system in my head. I know what order thingsare supposed to happen. I know somehow what’s not supposed to happen,when things are supposed to not be happening. That lets me build up amodel for, “How could this thing possibly have happened?”

And at least some of those were two-machine problems, which alsorequired some odd creativity to find. That is, the trouble is something goeswrong on my machine and the evidence of it shows up on yours. I can’tstop—my machine has already processed 6,000 more packets by the timeyours hits the trap that says, “I got a bogus packet.” So now what do youdo? We’d work through, the three of us, finding ways to track those thingsdown and fix them and basically make the system pretty solid.

Seibel: Did you build in debugging code?

Cosell: No.

Seibel: So you had many different tricky bugs, each of which you had totrack down in a unique way?

Cosell: As far as I can remember, we didn’t build in any debugging stuff. Imean, these days, I always point out that you’ve got to make programs sothat they are testable. And the only way to make a program testable is tothink about that before you write the first line of code. You can’t retrofitblock points and assert points and test points that work efficiently and dothe right thing if you wait until the program is working.

But I’m sure that we didn’t think about any of that. We were just trying towrite this incredibly complicated real-time thing that had to be fast. It was ahard enough problem. We didn’t put in any real consistency checks; whowould want to waste time for that? So these things were all ad hoc patches.Jump off into a spare part of memory, run through some hand-coded stuffto check this or that or the other, jump back, and continue.

In fact, it was even formalized. One of the things—I’m pretty sure I wroteit—was a patcher where you could submit a patch to the system and itwould pull one buffer out of circulation and use it to hold the code and linkup to that and then link back. We used to do that kind of stuff but it was allad hoc. We would find some bug and we would crack our heads trying tofigure out what it could be.

A lot of the times, just understanding what the bug is points you at the rightpiece of code. Now you read it more critically and you fix it. Other times,you need to collect more data. Other times, you need to bang your headagainst the wall trying to catch that little bit of evidence that illuminates thething. And we did some of all of that.

Remember, we’re running on a machine that’s got no console, no nothing.In general, the patches would stash away some data and then halt themachine. Then we would probably use the front panel because I don’t thinkthere was a debugger we could run from the terminal that wouldn’t trashthe machine. So we’d look through the appropriate areas of memory fromthe front console, doing examines and deposits to go figure out what wasgoing on.

Seibel: So that’s literally a row of lights?

Cosell: Yeah, a row of lights. Bit per light.

Seibel: And toggle switches to put in the address?

Cosell: Right. Actually, this is better. The PDP-1 had toggle switches. Thisone had, as I recall, push buttons.

Seibel: How did the three of you work together?

Cosell: One of the things that I remember doing shows a little bit of thestyle difference. Will was a brilliant intuitive programmer. All of the hardestproblems that most people couldn’t understand how to do at all, he wouldfind ways to do.

Like the AI engine in Adventure that he did in Fortran of all things. And therouting algorithm and all sorts of stuff in the dynamics of the IMP system,Will had cobbled together. One of the things about a real-time system iseverything has to be timed out. You can’t wait forever for anything becausethere’s no forever in a real-time system.

And a bigger and bigger collection of time-outs were growing up all over theprogram. I tried to understand them and had a hard time doing it. So in oneof my revisions of the source code, I tried to make an algebra for all of thetime-outs. For example, the total time-out to get an acknowledgement for amessage should be eight times the time-out for a single packet to transit thenet plus something. Or, the total time-out for a message to track the net isthe maximum diameter of the net times the maximum time for the packetto make one hop.

I was sort of trying to find out what the basic constants in Will’s mind werewhen he put things together. When two time-outs had the same time werethey supposed to be the same or were they coincidentally the same? Whoknows? How many places do you have to change when you want to changeone of the constants? If you discover dynamically that you’re not waitinglong enough for something to happen and it is timing out when it shouldn’t,you know that you can’t just change that one time-out because these thingsare interrelated.

So I made a whole bunch of sharp sign defines, basically, to try to find thesmallest number of independent constants. I remember doing that becauseit was just really scary. It was one of the places where I was dabbling inthings that really nobody understood because a lot of those constants Willhad put in intuitively and we had tuned to make work, one by one. Thetime-out isn’t big enough and so we would make it bigger, not doing it byfirst principles or algebra, but just tuning it until it works.

Seibel: Did you find bugs that way or did you just put it on a more solidfooting so that as things changed, you could change things in a way thatwouldn’t require endless retuning?

Cosell: I don’t recall finding any bugs. But there were undoubtedly someplaces where there were timers that now had different values than theyused to, but not operationally significant ones, just defensively differentones. It was less so that you could change it if you have to; really it was sothat it made the program easier to understand. I hated having a programthat had 200 randomly chosen independent constants scattered throughoutit and knowing that they have something to do with the heartbeat of thenetwork. I think it simplified some of the code. It made it easier to fathomwhat was going on. It also let us use more symbolic constants. Eight timesdiameter plus pulse time or something like that would be understandable.

Will was sort of the advanced idea man. I remember complaining to FrankHeart about this once, that he got to work on the projects right out of thebox because BBN was doing a lot of very cutting-edge stuff and he wasterrific at finding ways to do things that couldn’t be done before.

He was not as good at getting 100 percent done nailed-down code. He wasreally good at getting 75 or 80 percent pretty good code that worked mostof the time. Will had already gone on to, I think, the TIP, and Dave and Iwere still working on the IMP system and that’s when I redid the routingalgorithm because it had funny constants and I didn’t understand it. So it wasstill Will’s routing algorithm but recoded with my style. And I think it was alittle more solid. At least I understood if it was going to oscillate, why it wasgoing to oscillate, because I made it oscillate.

One of the places where Will Crowther and I absolutely differed—and I hadto put in hours and hours of work and even then he was skeptical—was hebelieved that when you reassemble a program you add more bugs than youremove. So he used to keep notebooks with pages and pages of patches. Hewould go as long as he could patching the existing system before he had toreassemble. Those patches were of patches on top of patches and socomplicated that often his prediction was a self-fulfilling prophecy. It washard, after all of that, to get it just right so that it turned out to be what thepatches were actually saying.