Josh Work Professional Organizations Trip Reports Conference Report: 1999 LISA

The following document is intended as the general trip report for Josh Simon at the USENIX/SAGE LISA Systems Administration Conference in Seattle WA, from November 7-12, 1999.

Friday, November 5, 1999

Today was intended to be a travel day, though I had to bill at least 6 hours. I left my client site at 11:30 to get home, shower, change, finish packing, and catch the 1pm shuttle to the airport. The shuttle arrives on time and gets me to O'Hare on time. Unfortunately, United Airlines couldn't get an aircraft with working engines to the gate on time. Our 3:30pm departure time goes by, with no announcement other than "The aircraft engine idle switch is being replaced and we're running about half an hour late." At 4:00pm, they say they'll reach a decision by 5:30pm as to whether or not the flight will be cancelled. At least they gave us $6 meal vouchers. (Excluding alcohol and gratuity. I opted for McDonalds and still spent over $5 of it.)

At 5:00pm, they decide to tell us more at 6. At 6, they decide to tell us at 7:30. Then, they tell us leave the gate area because they need it for another flight, and after we've left decide to announce a flight change: Instead of 3:30pm flight 943 were transferred to 7:45pm flight 8829—which had a departure of 8:02pm. To their credit, they did apologize regularly, and provided free movies (The Sixth Sense and The Thomas Crown Affair), free alcohol, and a $25 off certificate on a future flight. But they ran out of red wine (and both entrees were beef-based). But the white wine stood up well enough to the pasta dish.

We finally landed at Seattle-Tacoma airport at 10:17pm, taxied to the gate (10:25pm), rescued my luggage from Baggage Claim (11:00pm), caught the last Grey Line shuttle to the Sheraton (11:15pm), checked in (11:45pm), and crashed hard (midnight).

Saturday, November 6, 1999

I slept in, but still woke up before 9:00am local time. I did a bunch of data updates to the Coordination Matrix workbook. I went down to breakfast with the Program Chair and then went to help out with Registration and conference setup. I wound up working the rush hour at Tutorial materials, handing out peoples' tutorial materials and trying to get them all right. (And with only a couple of mistakes we managed to do just that.)

After handing out materials, the Welcome Reception opened. Cash bar, free soda, free fajita fixings, and fun conversations. After the reception, a group of adjourned to the hot tub until it closed at 10:00pm, then to the bar for munchies, drinks, and more conversations. After a little while we looked up and realized it was past midnight, so the party broke up and we all went to bed.

Sunday, November 7, 1999

Sunday I had a full-day tutorial, S2: Linux Systems Administration. Unfortunately, while the instructor knew what he was talking about and was engaging, and the material was correct, the class was not what I expected. The class was indeed on how to administer Linux systems, but was taught assuming no previous knowledge of or experience with any Unix-like systems' administration tips, techniques, or tools. I was expecting a class more on the specific "Here's what's different about Linux administration as opposed to other Unix-like OSes," and was disappointed as a result.

After the tutorial sessions, a small group of 5 of us went to Blowfish, a pan-Asian restaurant in the Paramount hotel. (We got mildly lost and wound up a block or so away at the Paramount theater. Oops.) If you're ever in the Seattle area and want a good dinner, this is a restaurant I'd recommend highly; the menu included a four-onion beef stir-fry, pot-stickers, spring rolls, various sushi and sashimi dishes, shrimp-and-sausage fried rice, some ginger chicken wings, and several other yummy dishes whose details escape me a week later.

After dinner, hanging out in the hotel lobby meeting and greeting folks, I found out that the Sheraton had somehow managed not to get all the CT members' rooms onto the corporate charge account, meaning members would have to pay for their own hotel rooms (and later be reimbursed). Since not every member has a spare $1500 or so of credit card balance, this could have been a serious problem. I fired off an email to Shea Avery (travel coordinator extraordinaire) before bed.

Monday, November 8, 1999

Today I only had a half-day class, M9: Administering Production Linux Systems. This course was definitely more to my liking and my expectations. It assumed previous experience administering Unix systems and concentrated on some of the uniquenesses in Linux. However, it didn't really address production-specific issues, perhaps because the instructor hasn't had experience working in production-critical sites. For example, she didn't understand why someone would want to mirror a striped file system (or stripe a mirrored one, for that matter), to gain additional redundancy in the event of disk failures. She understood it once it was explained to her, but if she had production-world experience she'd've not needed the explanation.

After the tutorial and lunch, I called Shea Avery to confirm that the problems we'd had with the hotel had been fixed. She said that they said it'd all been taken care of. (This theme will resurface. I'm not impressed with the Sheraton yet.)

Spent the rest of the afternoon doing the "hallway track," talking with other senior systems administrators about things going on in their lives and just doing the human-level networking. After that, a small group of 5 or 6 of us went out to Cutter's on the Bay for a delicious seafood dinner. I completely blew my per diem — a glass of wine, an iced tea, a steamed Dungeness crab (yum!), a chocolate volcano cake (pyramid-shaped chocolate cake, filled with a warm chocolate ganache, served hot, with vanilla ice cream and chocolate sauce), and half of a half bottle of muscat (dessert) wine.

Tuesday, November 9, 1999

Tuesday was the Advanced Topics Workshop, hosted and moderated once again by Adam Moskowitz. The 30 or so of us went around the room doing introductions (our names, companies, numbers of users servers and gigabytes of storage), discussed our environments, and mentioned some of the problems we were seeing. We then looked at some of the common themes, such as hiring and growth (virtually everybody present had open positions to fill), scaling (especially at the enterprise level), some tools, and areas where we felt there had to be improvement (such as systems administrators being able to speak the language of business to justify expenses).

The afternoon session of the Workshop included some predictions for what we think will be coming in the next year (wireless LANs, load-balancing hardware, LDAP, the lack of adoption on a widespread basis of Windows 2000, the lack of adoption on a widespread basis of IPv6, an increased demand for H.323 proxies for video conferencing, at least one major DNS outage lasting 24 hours, no new top-level DNS domains (like .web and .biz), and no major problems when the century rolls over. Lest you think that we're omniscient — or that we even consider that as a possibility — we also looked at our success rate from the previous 4 workshops. We were right about some things, dead wrong on others, and 1-4 years ahead of our time on still others. So take these predictions with a grain (or bushel) of salt.

Finally, we wrapped up the workshop with a discussion of some problems we're facing (a VMS to Unix transition in one place, the administration of customers' router passwords in another, and so on), with possible solutions bandied about. We also mentioned some interesting or cool stuff we'd done in the past year. Other than Y2K Remediation and documentation of policies and procedures, I hadn't done a lot.

Tuesday evening there was an impromptu "students BOF," a Birds of a Feather session for students and those interested in bringing more and younger people into the USENIX Association and the SAGE Special Technical Group. The meeting ran longer than I had, though I was there to back up my Program Chair, David Parter (who is also the USENIX University Liaison to the University of Wisconsin). At 7 I switched to the "Workplace Issues for Gay Men, Bisexuals, Lesbians, Transgendered, and Friends" (the "queer BOF" for short), as I co-host it regularly. We did the introductions bit and talked about how we've seen improvements — benefits like health insurance for our partners, inclusion as a protected class in the anti-discrimination clauses — in the past several years, as well as how we've still got a long way to go before sharing equality. One of our subcommittees reminded the group that the USENIX Association has money in their budget set aside for special groups if the Association can help — for example, publishing leaflets or booklets or participating in other tasks. (USENIX President Andrew Hume tries to come to the BOF at each conference, but he hadn't yet arrived as of Tuesday evening.)

After the BOF, 21 of us wandered out in the rain to Isabella's, a small Italian place (Stewart at Third Avenue). Very good food; yet another excellent dining choice — and yet another shot to the per diem. (How can people afford to live on $35/day for 3 meals, even without alcohol? Fast food is not a valid answer.)

Wednesday, November 10, 1999

Wednesday marked the start of the actual technical conference. The day began with the all-CT breakfast at the Pike Street Cafe in the Sheraton hotel. (Actually, I lied. The day began with me waking up at 7:10 for a 7:30 breakfast because some sneaky gnome had changed my alarm to go off at 6:30pm instead of 6:30am.) Virtually everybody who was expected showed up (on time!), and the various members of the coordination team — me (Conference), Jacqui Galow (Marketing), Shea Avery (Travel), Sharon Fox (Recruiting), Matt Bezinski (Sales), and Carolyn Hennings (On-Site) — briefed the rest of the folks present on expectations, schedule changes, and so on. (Shea and I talked briefly about the hotel. It seems that a couple of members' rooms were STILL not on the Corporate direct-bill list. Shea promised to look into it again. I'm even less impressed with the Sheraton. This theme will resurface.)

Announcements and Keynote

The first session started with the traditional announcements from the Program Chair, David Parter:

Andrew Hume, President of the USENIX Association, requested feedback on the direction of USENIX and SAGE. If you have any comments, please feel free to forward them to him, to a member of the USENIX Board, or to a member of the SAGE Executive Committee. [Editorial note: You can send them to me and I'll forward them along if you wish.]

Hal Miller spoke next. Due to a variety of personal issues, he has retired as President of the SAGE Executive Committee (though he remains on the Committee). Barb Dijker is the new President of SAGE, effective November 9, 1999. She then stepped up and presented the 1999 SAGE Outstanding Achievement Award to Wietse Venema for his "continual work to improve the security of systems," including such tools as tcp_wrapper, satan, and postfix, as well as the coroner's toolkit.

David Parter then presented the best paper awards:

The LISA 2000 Program Chairs were announced: Remy Evard and Phil Scarr.

David next introduced our keynote speaker, Joe Ruga from NASA (now with IBM Global Services). Joe is a systems administrator in the aerospace industry and he discussed his experiences with changes in the technology industry, with owning versus leasing, centralized computing versus distributed computing, and so on. He touched briefly (too briefly, in most attendees' estimations) on managing the shuttle launches. The two most important tips he mentioned were to befriend your users (to keep them from yelling at your boss) and to document your work (to be able to understand it yourself in a month or six). In summary, things will change over time — organizations, structure, what you do, how you do it, to whom you do it — so be prepared to manage expectations.

Session 2: Electronic Mail

The next session, the first technical session of the conference, was the one I was Session Chair for. The session, on electronic mail, included three papers.

"ssmail: Opportunistic Encryption in sendmail"

Damian Bentley, Australian National University; Greg Rose, QUALCOMM Australia; Tara Whalen, Communications Research Centre Canada

Greg Rose began by noting that most electronic mail sent across the Internet is sent in the clear, unencrypted. While users can exchange public keys on an individual trusted basis, there is no good, scalable, public key infrastructure today. Until one exists, he and his co-authors created ssmail. ssmail prevents against eavesdropping, which is a passive attack. It does not guard against active attacks, such as penetration (reading the unencrypted mail file in a spool), sending mail to the wrong host, or man-in-the-middle attacks. It provides privacy but not authentication or authorization.

The solution ssmail uses is to add the CRYPT routine to the extended protocol (ESMTP). When speaking to another sendmail daemon, the CRYPT command is sent. Assuming both hosts agree that they can encrypt the session, they exchange nonces and confirm the encryption (250 XCRYPT OK). Everything after that, including MAIL FROM and RCPT TO and the data itself, is encrypted. They use a 768-bit prime number from SKIP (used in IPSec, the security extensions to IP) and SHA-1 hashing when needed, and supports the RC4 and 80-bit SOBER-t32 encryption algorithms today. (You need to get an export license from the Australian government, but it's usually a rubber stamp work and takes about 2 weeks.)

The software is available from and is in beta test now. Future work is to add more algorithms, authentication, and possibly encrypting the mail spool.

"MJDLM: Majordomo based Distribution List Management"

Vincent D. Skahan, Jr., and Robert Katz, The Boeing Company

Robert Katz spoke about the Boeing environment, where the senior-most management wanted to be able to send targeted email to company employees (thereby giving rise to the subtitle of the paper, "It's not spam when WE send it"). They use MJDLM to send out alert mail that points to the intranet for detailed information. This has allowed Boeing to save hundreds of thousands of dollars in paper memorandum distribution. Basically, using Majordomo, they create lists on a nightly basis (using the centralized database from Human Resources), allowing only authorized senders to send the messages and redirecting bounces and replies appropriately. Design constraints required that it be cheap (free) and available immediately.

The current scale is with 2-3 mail hubs (where expansion of aliases takes place), taking between 12 seconds and 6 hours to deliver (based on the size of the list). There are over 1,000 end hosts (including 17 Exchange servers, which receive 90% of the mail) and over 2,100 messages sent since inception (July 1998), ranging from 18 recipients to over 140,000 recipients.

Future enhancements include handling bounces better, using newer versions of underlying software (currently Majordomo 1.94.4, Perl 5.004, procmail 3.11 pre 7, and sendmail 8.8.8), and provide a web-based front end to the command-line tools for list administration and maintenance.

"RedAlert: A Scalable System for Application Monitoring"

Eric Sorenson, Explosive Networking; Strata Rose Chalup, VirtualNet

Eric Sorenson spoke about RedAlert, which has a command line interface, application programming interface, and a server. It was deisgned to be extensible, easy to integrate, and catch failures over time. It uses intelligence and history to determine if something is a real problem or just a false positive. Written in object-oriented perl 5, it uses thresholds and watchers to determine if some number of events exceeds thresholds and takes corrective action as necessary. It can be obtained from under the GNU Public License (GPL).

Vendor Floor

After the session ended, I went to the vendor floor and did some walking around. The floor had opened at noon, and was still fairly mobbed. The CT booth was packed to overflowing, with attendees filling out lead and contact sheets to get the give-aways (t-shirt, SPECTRA poster, and recruiting CD) and enter the drawing for the Palm Pilot and the Sega DreamCast. The web camera was up and running — though I still think we should've put a hand-lettered "Gone Drinkin'" sign in front for an hour or so. Got some nice gimmes — stress balls, keychains, Sendmail screwdriver kit (Cat Allman has great toys at the booth every time!), mints, t-shirts, and so on.

Session 3: Referreed Papers, The Way We Work

The next session I attended was the Referreed Papers track called "The Way We work." Though I was late into the first paper, I did watch it on the monitor while eating lunch.

"Deconstructing User Requests and the Nine Step Model"

Thomas A. Limoncelli, Lucent Technologies/Bell Labs

Tom Limoncelli provided insights on how he developed the 9-step model of systems administration. to help reduce user complaints and to more often solve the problem reported correctly the first time. The steps can be broken into 4 phases:

  1. The Greeting ("Hello!")
    1. The Greeting
  2. Problem Identification ("What's wrong?")
    1. Problem Classification
    2. Problem Statement
    3. Problem Verification
  3. Planning and Execution ("Fix it")
    1. Solution Proposals
    2. Solution Selection
    3. Execution
  4. Verification ("Did it work?")
    1. Craft verification
    2. User verification/Closure

While this may seem straight-forward to those of us who have administered systems for longer periods of time, some people either don't know or don't follow this process. The perils involved in skipping steps can lead to solving the wrong problem (steps 2-5), choosing a solution that doesn't solve the problem (step 6), making a mistake executing the solution (step 7), not checking our own work (step 8), or having the user call back with the same problem (step 9). Worst yet, some administrators don't even bother with step 1 and don't answer the phone or their email (or whatever the contact method is) politely.

"Adverse Termination Procedures -or- `How to Fire A System Administrator'"

Matthew F. Ringel and Thomas A. Limoncelli, Lucent Technologies Bell Labs

Matthew Ringel discussed how to fire a systems administrator, as well as what to do when you're the one being fired. The paper itself contains several case studies from which the authors created a 3-tier model:

In summary, if you're in the unenviable position of firing someone, you need to ensure that all three tiers of access are covered, because leaving one or more undone can result in a disgruntled person (with super-user privileges in the case of system administrators) having access to your systems, networks, and data. If you're the one doing the firing or the one being let go, be professional. You may have to work with these people or companies again, and while expletives may be satisfying they're also counterproductive.

"Organizing the Chaos: Managing Request Tickets in a Large Environment"

Steve Willoughby, Intel Corporation

Steve Willoughby discussed that more than simply software but also the infrastructure to support customers' needs. Status reports are good (if not essential); having data that proves what you do and how much you do is absolutely required for management to increase (or sometimes even maintain) head count and budget. Electronic mail and simple scripts are okay for managing problem reports but tend not to scale well in an enterprise environment. Steve's group designed and implemented a new system to meet the needs of both customers and system administrators.

Having service level agreements (SLA) and senior management of both the customer and support sides is required. Intel also rotates its senior people onto the help desk, automates processes, and allows the user to control the closure of a ticket. They've found that this system scales better, requires fewer administrators per user, and results in users having more control over their problem reports and feeling happier about the process.

Future plans include more work on root cause analysis to help resolve problems before they become disasters.

Session 4: Practicum, MIT Web Servers

The final session of the day was the Practicum session on how MIT handles publishing documents on the web. Since MIT is a major Andrew File System (AFS) user, everything is stored in AFS space. They provide access to AFS to the web server machines and let the content owner manage the content. The environment itself lends itself well; they use AFS and Kerberos and well-understood backups and have a scalable environment.

The web support team provides recommended tools and services (such as mail forms, image maps, restricted access, a search engine, a WYSIWYG editor, interactive campus maps, certificate-based authentication, and web publishing training). Future plans are upgrades to these (such as discussion groups, a better search engine, usage statistics, SSI, individual-level restrictions, secure FTP, a new WYSIWYG editor). Unfortunately, I had to leave this session early to handle a few minor conference-related problems.

Evening Activities

After resolving the issues and catching up on electronic mail, I met up with some friends and we headed off by bus to the conference reception at the Museum of Flight. The museum itself was grand, and we had full access to the static and interactive exhibits (including the flight simulator). The meal — mostly finger-food and desserts but also some roast beef carved on the spot — was excellently catered. (Later I was discussing this with Judy DesHarnais, the conference organizer for the USENIX Association, and she said that the Museum only allows that one firm — McCormick and Schmidt — to cater events there. We could see why; the food and service were both excellent — which cannot be said for the Washington State Convention Center or the Seattle Sheraton and Towers.)

A group of regulars adjourned from the reception a little early and went to the hot tub to soak for a while. After that — and they closed at 10:00pm all week, drat — we wandered down to the lobby bar for drinks, more munchies, and more conversation.

Thursday, November 11, 1999

Morning Activities

I attended the first paper in the Referreed Papers track since it won the Best Student Paper award.

"A Retrospective on Twelve Years of LISA Proceedings"

Eric Anderson and Dave Patterson, University of California at Berkeley

System administration research is practical but repetitive. Academic research is detailed but irrelevant to systems administration in the real world. The idea behind the paper was to take the best from both the systems administration and academic disciplines.

Looking at the subjects of the papers at LISA over the past 12 years, Eric noted that one third of the time is spent dealing with requirement changes (74% of the cases), one third of the time is spent handling internal anomalies (24% of the cases), and one third of the time is spent dealing with users' confusion (4% of the cases). Looking at the subjects by task time, we see that 47 topics have more than one paper in them, indicating that systems administrators do lots of different things. Some are slow and steady (indicating good practices), some are bursty over time (indicating possible duplication of effort), and some pause then resume (indicating where technology changes have increased a solved problem into an area where the solution no longer scales).

Eric looked for patterns. External changes can indicate areas where new research is needed. Repetition of tasks can be automated. Some topic areas started with papers discussing the specific case ("Here's how we do this") and later became the general case ("Here's how to do this anywhere").

More information can be obtained on his web site at

After this paper I segued over to the Budgeting talk for a while, and then went to the Membership Services desk to get my program Committee t-shirt. I wound up in a discussion there with the SAGE Executive Committee's sitting duck member and a couple of other people about SAGE and local groups.

Session 2: System Administrators' Body of Knowledge

I had good intentions. I was going to attend Geoff Halprin's talk on the System Administrators' Body of Knowledge. He started by going over some recent legislative changes in Australia (where he's from) on a par with the US' Communications Decency Act.

Geoff began the Body of Knowledge portion of the talk by briefly discussing the roles that systems administrators typically play: troubleshooter, walking encyclopoedia, toolsmith, researcher and student, technical writer, strategist and tactician, and doctor and counsellor. He went on to discuss some of the problems we face, including lack of management understanding what we do, lack of reporting standards (for example, what does "available" mean: Service up? System up? On what time basis, all the time in the period, or only during scheduled uptime?), lack of workplace standards (every job is different), no time to research or prevent problems, and so on. Systems administration is the integration point where the application, user, and command all meet.

Unfortunately, as Geoff was beginning to get into the meat of the talk, the alarms went off. The voice-over announced that "an officer has been dispatched," and a few minutes later confirmed it as a false alarm, and muted the noise. (The alarms continued to sound, but quietly. Geoff could talk over it with the microphone.) No sooner did the voice-over announce the confirmation of the false alarm than the alarms went off full-force from the beginning again. I stepped out of the session to investigate and determined several things:

  1. The Washington State Convention Center alarm system can only be deactivated by a member of the Seattle Fire Department.
  2. The alarm was triggered EITHER by the vendor show floor's pizza oven being too close to the heat sensor for the fire alarm OR by a blown fuse or circuit breaker caused by the vendor show floor's pizza oven.

The alarms were eventually silenced for good about 25 minutes after they initially sounded. But I missed the rest of the talk.

Afternoon Activities

For the first half of the afternoon I was a floater. I was in and out of the vendor show floor (where I found out that our company President would be flying up for the day to see some sales leads and to attend our hospitality suite in the evening), and spent a brief amount of time in the Practicum talk on Ethics (unofficially titled the Rob and Lee Show). During the afternoon break I ran copies of the Program Committee announcements to the various Session Chairs (Thursday evening had over 25 BOFs scheduled and 9 hospitality suites).

Session 4: Invited Talk, Approaching a Petabyte

Finally, a talk I could sit through in its entirety. Hal Miller, the immediate past President of SAGE, gave a talk on what it's like to approach a petabyte of storage.

A petabyte is 1024 terabytes, or approximately 1.1x1015 bytes. (For the curious, the next orders of magnitude are exabyte (EB) and zetabyte (ZB).) The trends are towards explosive growth but with bandwidth bottlenecks. The desire seems to be the equivalent of "dial tone" for IP networking, computing, and storage. This is all well and good, but how do we get there and make it work?

The problems with a petabyte are many. Hal touched on some of them: 1 PB is approximately 100,000 spindles on 18 GB disks. Mirrored 5-way that's 500,000 spindles (and 2 copies offsite). Mirrors take 70,000 spindles, plus RAID drives, spares, and boot blocks, so we're talking around 1,000,000 total spindles. At $1,000 per that's $1 billion. Just for the disk — this excludes the costs of servers, towers, networking, and so on. Where do you put these disks? What are the power and cooling requirements for them? How do you perform the backups? How accessible are the backups? Where do you store the backups? How can you afford the storage, the facilities, the power, the cooling, the maintenance, the replacement of disk? As you can see, there are many questions but few answers.

Why is this relevant? Who faces this problem? Oil companies (geophysical research), medical research (including genetic research), and movie companies (special effects) face it now. Atmospheric sciences, oceanographic sciences, manufacturing, and audio delivery will face it soon. And academic institutions will face it as well, since they do research as much as (if not more than) commercial institutions.

More information (and the slides from the talk) are available at

Evening Activities

Tonight's the hospitality suite at GameWorks! I get there a bit early (and have to show my CT LISA'99 Oxford shirt for them to let me in) to help with setup. There are 40 or so CT people present and we make sure folks are available to work the door (handing out the $17.50 game cards, entries for the give-away drawings, and so on), work the table (handling the entries for the drawing, giving folks the Auspex bouncing blue ball with red light inside, and keeping people from taking the drawable give-aways early; unfortunately, the CT baseball caps walked off before we could stop people), work the web camera (Bill Huff found that the T1 line wasn't there so he improvised by running phone cable and using the dial-up modem in the laptop), and work the crowd.

Over the course of the 3 hours we were open, we had over 1,100 people come through. Several people stopped me to say this was the best hospitality suite they'd ever been to at LISA (and one of those was someone who'd been coming to LISA since the beginning). I don't know how much we spent (nor what percentage was CT's and what percentage was Auspex's), but it has to be close to $100,000. But given the amount of positive buzz the next day and the satisfied expressions on people's faces, I definitely think it was worth it.

Friday, November 12, 1999

The last day of the conference always becomes odd, if not downright surreal. This year was no exception.

Session 1: Referreed Papers, Network Security

I started by attending the Referreed Papers track, though I only got to see the first paper.

"Snort: Lightweight Intrusion Detection for Networks"

Martin Roesch, Stanford Telecommunications, Inc.

Snort, based on pcap, is a lightweight (small), free, packet sniffing agent. It has a simple configuration language with flexible rules and a detection engine. It is very powerful and is designed to be a "poor man's honeypot." It can sniff for macroviruses and can be used to replace Shadow and tcpdump.

The source and documentation are available at and sample rules are available at

Game Show: Win, Place, or Show

Around 9:45am I was tapped on the shoulder by the program chair and I left to discuss the game show segment with him. This was more fitting than usual as I was one of the people responsible for the format change. At the Program Committee meeting back in June there had been some discussion that the LISA Quiz Show ("It's NOT Jeopardy!") was getting a bit long in the tooth — it's done with great frequency at both LISA and USENIX, as well as other venues. So we thought of other possible game shows to borrow from and decided on Family Feud as a jumping-off point.

Since we didn't have time to get legal approvals from the Mark Goodsen Company, and the thought of Rob Kolstad kissing every contestant was amusing but probably unwise, we changed the format substantially. We assembled 55 questions and put together a web survey with many unique single-use URLs to prevent ballot-box stuffing. We handed out cards with the unique URLs on them to all conference participants on Tuesday, Wednesday, and Thursday.

The Rules

Each game would have 2 teams of 4 participating in trying to determine the answers. However, it wasn't always the top ten answers; we'd eliminated obvious responses. For example, the question "What's your favorite soft drink" had Coke, Pepsi, and Mountain Dew removed as possible answers. Each game consisted of 4 rounds:

The final game would omit round 3 and add a different final round, where each team has to match the answers to the same question. Since it's possible though unlikely for a team to remember that (1) matches (c) and (2) matches (e) and so on, the team that goes second is blindfolded during the first team's attempt.


I collected the applicant entires from the registration area and the Terminal Room and met with Rob Kolstad (host) and Dan Klein (announcer, foil, and game show operator). We went through the results from the online survey (330-some conference attendees responded to the 55-question survey on the web), collated the answers (how many different spellings of "Eisenhower" are there?), and ran through a few sample rounds of the 5 different games to work the final bugs out of the game show system.

We selected 32 of the applicants at random, for 6 teams of 4 (the first 24) and 8 alternates (in case any contestants didn't show up). The first two teams were to play each other in one game of four rounds, then the next two teams, and the final two teams. We'd have a brief intermission and then the two highest-scoring teams would face off in the finals.

Teams and Games

Game 1
Team 1: CAPS LOCK (300 points)
David Grieg (winner, Monterey USENIX 1999)
Esther Filderman
Trey Harris
Tom Reingold
Team 2: The Winners (300 points)
Mark McCullough
Phyllis Smith
Nick Stoughton
Wolfgang Friebel
Game 2
Team 3: Team Dot Dot (150 points)
Frederick Palmer
Joseph Pingenot
Clipper Dolan
Andy Mondore
Team 4: Core Dump (320 points)
Johan van Zanten
Mark Allen
Aaron Mandel
Rami Dass
Game 3
Team 5: End of File (260 points)
Jedrzej ("Yen") Nasiadek
David Thompson
Brian Kirouac
Ken Barnhouse
Team 6: Moo Shu Weasels (350 points)
Joe Ruga (Keynote speaker)
Torkel Thune
Dan Pritts
Jon Kuroda

In the Finals, Team 6 (Moo Shu Weasels) beat Team 4 (Core Dump) with a score of 500 to 380.

Since we had a tie for third place, the 2 300-point teams independently got to answer a final question, and the winning team would take home the third-place prizes. Team 2 (The Winners) correctly answered which car make most respondents said they drove (Ford).


Before the Finals, a group of people (the musical group sing; sing; halt) came up to sing a song they'd written the night before in the elevator: "The C Days of Y2K." Its lyrics are too long to reprint in full here, but the final verse was:

On the C-th day of Y2K my server said to me:
Enter system password,
Out of ttys,
Can't fork process,
panic: double panic,
can't open socket,
bad magic number,
no route to host,
hme is down,
stale file handle,
/var is full, and
it's January 1970!

It received a standing ovation from the audience (along with occasional harmony from the singers there).


Each contestant received a copy of Essential Python from New Riders' Press and a gift certificate good for a free book from Addison-Wesley. The third-place team got SAGE t-shirts for each participant. The second-place team received $50 each; the first place team received $100 each.

Evening Activities

While the conference officially ended at the end of the Game Show (which ran long by about 40 minutes, although it started late because the Convention Center staff didn't show up to remove the wall between rooms and set up the combined stage — yet another nail in their coffin, as far as having another conference there goes), the post conference events began. First, a group of 8 of us went to Fullers, the upscale restaurant in the hotel, for a very nice (if overpriced) dinner. Good service and excellent presentation, but hotel prices. After dinner I went to the Dead Dog party for a while, where I got to see the wonderful "Favorite Things" video that kc claffy had brought. Much conversation, relaxing, and plotting for the 2000 and 2001 Program Committees.

Saturday, November 13, 1999

Saturday was my birthday. I had brunch with Tom Limoncelli, Tom Reingold, and Strata Rose Chalup (two members of the Program Committee and a coworker of one of them), then did some touristy Seattle stuff (Pike St. Market, some used bookstores, an old fashioned candy store, and so on) and we discussed the conference (what did and didn't work) and plans for the next conference (in New Orleans LA in December 2000). The concensus seemed to be to make hotel reservations early (as in, the day of the announcement) and restaurant reservations now. After brunch and touring around, I went back to the hotel and spent some time relaxing in the hot tub (with friends and some members of the '99 and '00 Program Committees).

Then we met up with the other still-in-Seattle folks for dinner. Well, it was more complicated. Adam called me, I called him (but he was on the phone with David), he called me back, and we realized quickly that an in-person discussion would be better. So Adam, David, Pat Wilson, and I met in my room to go over dinner options. To make a long story short, we somehow managed to get not-too-distant reservations at Isabella's (see Tuesday, above) for the 13 of us. Dinner was delicious again, and Pat told the waiter it was my birthday so I got a piece of birthday tiramisu with a candle in it.

Sunday, November 14, 1999

Today was the travel day. Luckily, I woke up early — since there were a few difficulties in checking out. Seems that the Sheraton had still not put my room and tax onto the Corporate account and expected me to pay for it. (They let me put a hold on it and leave the balance on the folio until Austin could fix it. Of course, they then decided to bill my card in full anyway. It's since been resolved successfully, but I'm not at all impressed with the efficiency of the Seattle Sheraton Hotel and Towers.)

Got to the airport in plenty of time. The aircraft, however, didn't. It was about 50 minutes late due to problems in Denver (where it was coming from). So we took off about 50 minutes late, but somehow managed to make up 20 minutes in the air and only got in half an hour late. The trip itself was uneventful.

Back to my conference reports page
Back to my professional organizations page
Back to my work page
Back to my home page

Last update Jul16/05 by Josh Simon (<>).