Seibel: That was FreeVote?

Fitzpatrick: Yeah, that turned into FreeVote after it flooded my host.Banner ads were really popular then, or they were just getting reallypopular, and I kept getting more and more money from that, bettercontracts, more cost per click. At the height I was getting 27 cents per clickof banner ads, which I think is pretty ridiculous even by today’s standards.So at the height, I was making like $25, $27 grand per month on fucking clickson banner ads.

This was all through high school—I did this in the background all of highschool. And I worked at Intel two summers, and then started doingLiveJournal my last summer, right before college. So then my first year ofcollege, I was just selling FreeVote, which I basically sold for nothing to afriend, for like $11 grand just because I wanted to get rid of it and get rid oflegal responsibility for it.

Seibel: When you got on your ISP and got to use Unix, did that changeyour programming much?

Fitzpatrick: Yeah. It didn’t drive me crazy. I couldn’t understand what wasgoing on with Windows. You’ve probably seen the Windows API—thereare like twenty parameters to every function and they’re all flags and half ofthem are zero. No clue what’s going on. And you can’t go peek underneaththe covers when something’s magically not working.

Seibel: Are there big differences you can identify between your earlyapproach to programming or programming style to the way you think aboutprogramming now?

Fitzpatrick: I went through lots of styles, object-oriented stuff, and thenfunctional stuff, and then this weird, hybrid mix of object-oriented andfunctional programming. This is why I really love Perl. As ugly as the syntaxis and as much historical baggage and warts as it has, it never fucks with meand tells me what style to write in. Any style you want is fine. You can makeyour code pretty and consistent, but there’s no language-specified style. It’sonly since I’ve been at Google that I’ve stopped writing much Perl.

I’ve also done a lot of testing since LiveJournal. Once I started working withother people especially. And once I realized that code I write never fuckinggoes away and I’m going to be a maintainer for life. I get comments aboutblog posts that are almost 10 years old. “Hey, I found this code. I found abug,” and I’m suddenly maintaining code.

I now maintain so much code, and there’s other people working with it, ifthere’s anything halfway clever at all, I just assume that somebody else isgoing to not understand some invariants I have. So basically anytime I dosomething clever, I make sure I have a test in there to break really loudlyand to tell them that they messed up. I had to force a lot of people to writetests, mostly people who were working for me. I would write tests to guardagainst my own code breaking, and then once they wrote code, I was like,“Are you even sure that works? Write a test. Prove it to me.” At a certainpoint, people realize, “Holy crap, it does pay off,” especially maintenancecosts later.

Seibel: When did you start working with other people?

Fitzpatrick: It was pretty much towards the end of college when I startedhiring other people, and especially once I moved back to Portland aftercollege.

Then the early employees were customer support, so they didn’t write anycode. Then slowly I started hiring programmers. The first person I hiredwas a friend of mine from online. His name is Brad Whitaker and we bothhad websites called BradleyLand or BradleyWorld, so we found each other’swebsites. I was a couple of years ahead of him web programming–wise, ormaybe a year, and he was asking me, “Hey, how do you that,” whether itwas HTML, or frames, or CGI, or Perl stuff. So then I started getting abunch of contract projects and I would give the ones I didn’t want to him.And then we had a project that was too big for either of us so we told theguy, “It’s going to take two people to do this project.” And he flew us outto Pennsylvania. Pittsburgh? I don’t know the East coast at all; I’m a Westcoastguy. Philadelphia? The cheesesteak place.

Seibel: Philadelphia.

Fitzpatrick: Yeah, and we met for the first time at some cheapo hotel andit felt like I knew him already. He was like, “Hey, what up?” He came in andtook a piss in my hotel bathroom without even closing the door as I’mstanding right there. I’m like, “Alright. You’re comfortable.” It was like weknew each other for four or five years, even though we had never met. Westarted working on this stuff together.

He moved up into my spare bedroom and we basically moved all the stuffout of my kitchen, set up a bunch of tables, and worked on computers. Wewould wake up around 10 or 11 and work until noon, and watch someTV—sit around in our boxers and watch TV, and hack, and stay up until 3or 4 in the morning just working nonstop. Then another friend of minemoved down for the summer from UW. This was after my freshman year incollege and then there were three of us working there. The third friend wasliving downtown. He would come on the light rail in the morning andskateboard over to my house. He would sit outside on Wi-Fi, just hackinguntil we woke up, opened the door, and let him in.

Once there were three of us, it was a little crowded in my house, so I waslike, “Oh, OK, let’s get an office.” So we got an office and we were like,“Oh, we have all this space! Let’s hire more people.” We slowly got up to12 over the next couple of years, and LiveJournal got more and morepopular, and then more stressful too, because I was dealing with HR.Or my mom was dealing with HR and my mom was fighting with mebecause she worked for me. I had to make rules for my mom, like, “If youcall me, it has to be personal or business; whatever one you start with,that’s how you end it. You can’t switch from work to personal or personalto work.” I just started hanging up on her if she switched. Then she’d callback and I’m like, “Nope, you lost.” So that was really stressful. She wasreally happy when I sold it, and she could stop working for me and we couldstop fighting.

Seibel: Was your company still doing contract work or was it allLiveJournal at that point?

Fitzpatrick: It was pretty much all LiveJournal. We were also trying tostart a photo hosting-service, which Flickr beat us to. Ours was probablyoverdesigned: beautifully abstract and plugged into everything. But each newinfrastructure thing we did for LiveJournal, we were like, “How is this goingto work on FotoBilder?” so we started building everything abstract.Memcached was abstract because there was no reason to tie it intoLiveJournal. Then we built a file system like GFS, and we built a job queue.So we kept building all these infrastructure components for scaling thatwould work for either of our products, but also because the lessintertwined spaghetti dependency-wise, the easier it is to maintainsomething. Even if it’s a little bit more work, if you can cut somedependencies, it was great, so we started building all that genericinfrastructure.

Seibel: I’m curious about the process that you went through of scalingLiveJournal, in terms of where you started and how you learned the lessonsyou needed to learn along the way.

Fitzpatrick: So it started on one shared Unix box with other customersand pretty much killed that.

Seibel: Running as CGIs?

Fitzpatrick: Yes. Yeah. I think it was probably a literal CGI, fork up thewhole world and die. There was a guy assigned to me at this ISP. I washaving problems with my server dying all the time. I’m like, “I paid my$10.00 a month. Why isn’t it working?” So he would say, “Oh, do this.”Pretty soon I was learning Unix and learning what was actually going on.

Then I converted to FastGCI. Then I tuned Apache and turned off reverseDNS lookups. All these steps you go through. Finally, I was I/O-bound orCPU-bound. Then I got my own dedicated server, but it was still just oneand it was dying and I was out of capacity. I had originally opened it up formy friends and I just left the signup page alive. Then they invited theirfriends who invited their friends—it was never really supposed to be apublic site. It just had an open signup page on accident. So then I putsomething up on the LiveJournal news page and I said, “Help. We need tobuy servers.”