Josh Work Professional Organizations Trip Reports Conference Report: 2006 LISA

The following document is intended as the general trip report for me at the 20th Systems Administration Conference (LISA 2006) in Washington, DC from December 3-8, 2006. It is going to a variety of audiences, so feel free to skip the parts that don't concern or interest you.

Sunday, December 3

I managed to get out the door on schedule to head off to MSP. I did manage to take the recyclables out for recycling and to mail the letter (proxy statement) off. However, I managed to screw up two things while driving: First, I was on autopilot ("east of I-394 means I'm heading either to the doctor's office or to Minneapolis proper) so I didn't get onto US 169. Whoops. So instead I went via MN 100 which wasn't a huge deal; at 8am-and-change on a Sunday morning the route itself wasn't going to be a problem. Heck, even I-494 was pretty empty. Second, when I got to the airport, I went up one level too many on the parking ramp, so I'm parked in the wrong-for-me blue ramp instead of in green or gold. Not a huge deal, just means more walking and a tram ride when I get back to MSP on Saturday.

The flight east was uneventful. We were delayed 18 minutes on departure ("maintenance is signing off on the log book now," though they never said what part was broken and needed replacing). My reading light kept switching itself off and on again with nobody at the control, so I reported it to the flight attendant to get it fixed the next time the plane's in for in-cabin service and switched to using the iPod instead of the magazines for my in-flight entertainment. Got to DC uneventfully, found my way to the Metro stop, thence to the hotel, and finally got checked in and upstairs by about 3:15pm. Unpacked, tried to cool off (exercise, like climbing a hill, in sweatshirt, heavy winter coat, and carrying a laptop, a CPAP machine, and a week's worth of clothing including 2 special-function outfits, leads to overheating), then wandered down to the conference registration area to check in. Got my badges (seems that their new registration system is stupid and prints receipts even for cancelled people, so since I'd registered-canceled-and-reregistered I had two badges — which technically entitles me to two t-shirts, two Proceedings CDs, and so on). Reported that to the onsite conference office folks so they can fix it eventually, got my T-shirt, paid for the workshop I'm scribing, and then hung out in the lobby chatting with folks until dinner.

Dinner this evening — and incidentally the only food I had to eat today — was at a lovely bistro and bar called Medaterra just a couple of blocks from the hotel. The hotel concierge gave us a 20%-off coupon for the table, so the eight of us — Brian, Amy, Aaron Bob G., Joe, Ted, Tom,¹ and I — got some really good food for really reasonable prices. I had the mussels appetizer (mussels in a chardonnay and garlic sauce with fresh tomatoes and some fresh parsley) and the slow-roasted lamb shank (served with a tomato sauce over rice and mixed vegetables). They ran out of the double chocolate overload cake (I nibbled some of Brian's and it was yummy) and were already out of the caramel pecan chocolate thingy, so I wound up with the citrus baklava which was quite tasty.

After dinner, hung out in the hotel lobby bar with all sorts of people, catching up on Stuff and Life and Things, until my brain started shutting down around 11pm, so I headed upstairs and crashed.

Monday, December 4

Today was my free day, with nothing on the schedule. I was going to join folks for breakfast at 7:30 if I woke up in time, but when I rolled over and saw 7:08 I decided I wasn't going to rush and rolled over and went back to sleep. Finally woke up around 9am, did the morning ablutions, then hung out in the registration lobby, catching up with people and killing time until lunch.

David, Rowan, and I went across the street to the Open City Café, where I got a burger and fries for a more-reasonable price than at the Marriott. Spent the afternoon hallway-tracking (and capturing quotes, including the commentary about Mr. Salami from the book Aaron was reading Little Opal which brought the house down), then off to dinner at Tono Sushi where 6 of us got green tea, miso soup, and enough sushi so we were all stuffed (75? pieces or so) for about $20 a head.

After dinner, did some more hallway tracks until folks started to congregate in the bar. Had some interesting conversations (some of them even related to work) before heading off to bed early, for Tuesday I get to work.

Tuesday, December 5

Tuesday started when I met Aaron, Bob G., and one of Bob's cow-orkers for breakfast in the hotel's Perle's Café. $19 for the breakfast buffet after tax and tip, which is something like half the per diem. Oi. Luckily lunch was paid for.

On the technical side, Tuesdays are the Advanced Topics Workshops. Once again, Adam Moskowitz was our host, moderator, and referree.

[... the rest of the ATW writeup redacted; check my LJ and my web site for details if you care ...]

After the ATW broke up, I helped with the Unloading of the Car and helped lug supplies from the garage in Park Tower over to the LOPSA After Dark suite in the Central tower. Did some setup and then several of us — Aaron, Cat (who arrived late), David, Michael (who arrived later), Rowan, Strata, and Travis — adjourned to Harry's Pub in the hotel for dinner.

We rushed through the end of dinner to make it to the GBLT[UVWXYZ] BOF, also known as the motss.bof, the Alphabet Soup BOF, and "Faggots and Dykes and Queers, Oh My!" Michael ran it and used one of David's questions to break the ice: We went around the room and introduced ourselves by name, company, and what happened when we came out as sysadmins. Several folks treated it as a euphamism, and others treated it as a straight (you'll pardon the expression) setup. We had the usual benefits discussions (which confused our sole non-US-based participant, who is from a country with state-sponsored socialized medicine and the equivalent (in all but name) to gay marriage), some commentary about restaurants, some bemoaning of the lack of the hot tub at the hotel, and then broke up. Some folks went to dinner; I instead headed off to the Cambridge Computer BOF to grab the free ice cream. They were out at the time I was there, so I had raspberry sorbet-wrapped frozen vanilla yogurt which wasn't bad (and for the price was fantastic). I then adjourned to the LOPSA After Dark suite for conversation, bartending (no, not in my bartender drag), and more conversation until LOPSA closed down the suite at midnight to be nice to the hotel.

Wednesday, December 6

The conference technical (as opposed to tutorial) sessions began this morning. My day began with breakfast with Bob G., Frank, and Moose at the hotel buffet. Had a made-to-order omelet (ham, cheese, and scallion), bacon, pork sausage, chicken and apple sausage, breakfast potatoes (with onions, no peppers), a banana, and a strawberry croissant. Surprisingly enough, we're too far north for the buffet to have (southern) biscuits and gravy. Ah well.

The keynote session started with the usual statistics and announcements. This was the 20th annual Large Installed System Administration (LISA) conference, dating back to the first workshop in 1987 (75 people). Just think: Next year, LISA will be old enough to drink!

This year's conference statistics:

This was followed by thanks to the usual suspects: program committee members, external readers, chairs for IT and Guru tracks, USENIX staff and board, speakers, and attendees. Program Chair Bill LeFebvre noted some special presentations to try to drum up interest in them, pointed out the new "Lunch and Learn" with Dr. Alva Couch on Friday from 1-2pm, and our new closing session of improvisational comedy. He also reminded us of the social and evening activities, including BOFs and the various receptions. There were also two programming changes noted: One guru couldn't make his session but had someone there to cover; another speaker couldn't make it so his 45-minute talk was cancelled.

This year's award for Best Paper was for a student paper, "A Platform for RFID Security and Privacy Administration" by Melanie Rieback, Georgi N. Gaydadjiev, Bruno Crispo, Rutger F.H. Hofman, and Andrew S. Tannenbaum. The committee also awarded an Honorable Mention award for "A Forensic Analysis of a Distruibiuted Two-Stage Web-Based Spam Attack" by Daniel V. Klein. The SAGE Outstanding Achievement award was given to Tobias Oetiker(?), Dave Rand, et al. for their work on MRTG and RRDTool, which combined helped move system adminsitration from a purely reactive to a more proactive model. Finally, the annual Chuck Yerkes Award for Mentoring, Participation, & Professionalism went again to Doug Hughes for his continued support.

Cory Doctorow was our keynote speaker. Finally, an engaging speaker who was passionate about his topic and while it wasn't particularly new information, it was information that we needed to hear. And it's the first keynote we've had in several years that was sufficiently interesting that folks didn't feel the need to walk out en masse.

In the second session, I went to Elizabeth Zwicky's invited talk, "Teaching Problem Solving: You Can and You Should." She provided some useful tips and techniques for helping folks who aren't necessarily computer experts or wannabes learn how not to feel stupid when it comes to computers.

For lunch, I went back to Medaterra with Aaron, Frank, Jonathan, Leigh, Mark R., and Philip. I had a falafel which was a bit heavy on the parsley but very delicious nevertheless.

The first afternoon session was "Site Reliability at Google/My First Year at Google" by Tom Limoncelli It was more or less an overview of Google's site reliability tools that they use to maintain and manage the incredibly large number of machines involved. In the break I did a quick swing through the vendor exhibition and nothing jumped out at me. The final session for the day was "Open Source Software and its Role in Space Exploration," by DJ Byrne of NASA's Jet Propulsion Laboratory. An interesting talk about past and future missions and some of the interesting things the Mars rovers can do.

For dinner tonight, several of us — Aaron, Bob G., Frank, Ted, and Tom — hiked to Dino's. I had the roasted garlic with gorgonzola and a tomato sauce with golden raisins, then a lasagna with pork and veal with crispy pancetta. Interesting take on a classic dish. The texture was a little weird, but it was still very tasty. Dessert was a warm chocolate cake with vanilla gelato and an espresso sauce.

We got back to the hotel too late to make the SAGE BOF and decided I didn't want to go to the LOPSA BOF, so I got the scoop from folks later on at the LOPSA After Dark Suite, where I chatted and tended bar (no, not in my usual Dead Dog drag: I didn't bring it this trip) until we shut down at midnight (and Security came by at 12:05am to shut us down anyhow).

Thursday, December 7

Today's sessions were heavily data- and storage-oriented. My first session was Lee Damon and Evan Marcus' "Drowning in the Data Tsumnami," where they tag-teamed the history of how we stored data and how we've now reached the point where it's trivially easy to store but harder and harder to discard. Disk is cheap, and we tend to buy more rather than clean up after ourselves. Information, or finding the right data, especially in the long term, is hard, and we're drowning in the quantity of data. Threats to data are age (where the media wears out), hardware (readers go away, OS and applications to read them go away), losing the decryption keys, and indexing (finding which piece of archival media the information we want is actually on). They went on to define an archive explicitly as a place to store data that (a) takes data in, (b) preserves and protects it, and (c) allows appropriate access to the data. As IT people we need to work more closely with librarians and archivists to manage the metadata (the data about the information).

The second session was Dan Klein's invited talk, "Perfect Data in an Imperfect World." Basically, data is colllected everywhere on everything forever, and it never expires. If you want to prevent your past (think "youthful indiscretions") from catching up to you (think "run for president"), you've got a few options, but none of them are realistic: Never do anything wrong, don't do dumb things, don't publish anything you'll later regret (including your browser cookies), and don't care what people think. He went on to discuss some of the ways people are already tracked today: digital cameras in public spaces, GPS log data from cell phone call records, GPS units in phones and gars, GSM triangulation, and so on. There's no real good answer to this that technology can solve: Technology is neutral, it's the uses of it that can be for good or for ill.

I had lunch (again) at the Open City Café since I knew dinner would be on the pricey side. Mark R., Mike, and I all had burgers.

After lunch, I went to Cat Okita's talk about Identity 2.0. In general, an identity management (IM) system should be:

Some of the problems with IM systems are:

Now that we know what an IM is, what's Identity 2.0? It's basically IM centered on the user, and is intended to solve problems like too many identities, too many passwords, and the all-or-nothing distribution of information: If I tell the computer I'm jss1113 and use my password to prove it, there's all the associated data (my LJ interests list, my web site, my location, my age, my high school and college, and so on). There are protocols, frameworks, and standards for all this, with major companies involved in the space and cooperating. While we're not there yet, there's a lot of promise.

My final session today was Andy Seely, a civilian consultant, talking about the challenges involved with computing in a command-and-control environment in a war. Sure, you need an enemy to fight, but you also need troops to fight the enemy, weapons for them to fight with, all the logistics (bullets for the guns, food for the troops, support staff, and the transportation for everyone and everything), and the information (who to shoot, where, when, and so on). The information was the focus: many data sources come in and they synthesize and summarize it as needed. The OS and hardware tend to lag well behind leading edge, since unlike most environments most of us deal with, it's not just product or money involved, but life. Server crashes or other unplanned outages means people could die, so situational awareness is essential. Other concerns are field conditions (power, cooling, age of and concomitant repairability or replacability) and environmental factors (such as dust, moisture, scorpions, mortar shells, non-citizen local staff, time zone differences between the field and the back office, rapid personnel rotation including leadership, and a lack of resources like books, phones, and the Internet).

In lieu of the conference reception (because "Carnival of Fun" as a concept just didn't work for me), several of us — Geoff, Greg, Mike, Pat, and I — headed out to the churrascaria Fogo de Chão for "meat on swords". Despite incorrect directions from the concierge at the hotel (who directed us to Union Station instead of Metro Center three stops earlier), we managed to get there only 30 minutes late for our reservation. We had filet mignon, ribeye, top sirloin, bottom sirloin, house sirloin, pork sausage, bacon-wrapped chicken, lamb loin, and the salad bar, and we declined the lamb chop, chicken leg, and beef ribs. We also had 2 bottles of wine and were wonderfully replete by the time we called it quits.

Headed back to the hotel and up to the LOPSA After Dark suite to converse, schmooze, and tend bar. I forgot all about the combined scotch BOF and birthday party for Cat and Tom until after it'd ended. Whoops.

Friday, December 8

The first session I went to was Dan Kaminsky's "Black Ops 2006: Pattern Recognition." Unlike last year, when Dan's talk was rescheduled because he had the week wrong and had to rebook his flights, he was in town in time for this morning's talk. He wasn't in the auditorium on time, since he misremembered it as starting at 9:30am instead of 9:00am, but we were up and running before 9:10am. I guess that's improvement. There were three major bits in the potpourri of his talk:

He also noted that if you wanted to be depressed, look at online banking on a technical basis: 26% of the top 50 online banks are insecure and provide either your ID, your password, or both over the net in the clear or in such a way as to make breaking the encryption trivial. Google for "Why is this secure" and you'll get the login panels from several of them, and they don't always use https, they provide a picture of a lock but no actual security, and so on. (My bank does it right and encrypts the entire session: it's all https.)

During the morning break I helped Pat drag the LOPSA shirts down from the LOPSA suite to the lobby so they could get loaded into the car.

My second session today was the first that had nothing I really wanted to see. I did attend Adam Moskowitz's Hit the Ground Running session on bad interview questions, mainly to heckle. He summarized his 3-hour half-day class in about 10 minutes: You can't ask certain questions (age, marital status, religion, politics, medical history, personal habits, and so on) because they're illegal and you could get sued. You shouldn't ask certain other commands because they're not going to give you what you need to know. "What does ${command} do?" or "What does the -x option do?" are poor, because they test trivia; instead, since you want to know what they know, what they understand, and how they troubleshoot things, ask leading questions and make them walk through the process. I then bailed from the session and hallway tracked until lunch.

I grabbed a quick and cheap McLunch before the WIPs session. We had nine speakers: Each speaker gets time to set up and tear down their laptop-to-AV system connections and 5 minutes to speak. The audience voted at the end for the best WIP. The WIPs presented were:

The winner, judging by audience applause, was the A/V representative from MSI, Anthony, for his help. He won a bag of licorice whips.

The conference's closing session was Improv for Sysadmins by Bob Apthorpe and Dan Klein. Before they took the stage, two announcements were made: has stepped down as SAGE Programs Manager and Alva Couch is now in that role, and Paul Anderson is the LISA 2007 program chair.

Bob and Dan demonstrated several improvizational comedy techniques and applied each of them to how sysadmins (and IT folk in general) tend to send mixed messages. They provided some techniques for listening to the other person and for showing high or low status by both body language and spoken language. Being hunched over and avoiding eye contact is low-status, but using language like "Be right with you" is high-status; putting these in the same context can confuse people. In short, listen to the other party, observe status, and make the other person right: If these all happen, your job gets easier and you become a better communicator.

I went out with the 0xdeadbeef crowd to The Prime Rib for dinner. This is an old style steakhouse (with a cigar area in the bar we had to walk through to get to our table) which does stuff right. As a group we got aperatifs, appetizers and salads, the entrées, the vegetables, the desserts, and the post-prandial drinks over a leisurely 3.5 hours. Personally, I went with the small prime rib (since I didn't need the 20-25 ounces of the large which was bone-in), a baked potato with butter and bacon (they didn't have chives and I neglected to ask for cheddar), three glasses of wine with the food, a banana split, and a few sips of the Warre's 1977 vintage port (very nice, perhaps a bit young still).

We got back to the hotel around 10:30pm and headed up to the Dead Dog party in the old tower. Stayed there catching up with folks I'd missed for most of the conference until about 1am, then headed back to my own room to do the initial packing and crash.

Saturday, December 9

Today was the travel day to return home. Woke up before the alarm, finished packing, checked out, and headed off to the Metro (red line to blue line to airport shuttle to terminal), cleared security (despite the swab from the CPAP machine tripping the explosives alert on the scanner: the machines are set to be very sensitive), and hung out, first with Aaron and then Rowan before the former headed off to Madison.

The flight itself was uneventful, though there was a lot of hiking (from the gate to the terminal tram to the baggage claim area to get the checked baggage, to the escalator to the parking level, to the parking tram, to the blue ramp, to the car). I'd remembered that I was parked on the blue ramp and either the 4th or 5th level and within 3 rows of the elevator bay door. Luckily I guessed right and was on blue 4, row Q. Got home (despite some idiot rubberneckers watching cops at a no-fatalities accident scene on I-494) reasonably quickly, and unpacked before a quick late lunch of a pizza pocket and eventual dinner of bake and serve chicken cordon bleu and potatoes au gratin.

¹   Who told a truly horrible joke. Ask him about golden delicious apples.
²   "Registered attendees" is misleading, as it includes people who are registered for one or more tutorials, one or more days of technical sessions, or an exhibition floor booth, and also includes comped registrations in addition to the paid registrations.


Back to my conference reports page
Back to my professional organizations page
Back to my work page
Back to my home page
Last update Feb01/20 by Josh Simon (<jss@clock.org>).