Josh Work Professional Organizations Trip Reports Conference Report: 2009 LISA

The following document is intended as the general trip report for me at the 23rd Systems Administration Conference (LISA 2009) in Baltimore, MD from November 1-6, 2009. It is going to a variety of audiences, so feel free to skip the parts that don't concern or interest you.

Friday, October 30

This was my between-conferences travel day. Unfortunately, to make it in time for the 6:35am flight out of Orlando, the shuttle wanted to pick me up at 3am. (4:30am would have been more than enough time.) It turns out that the Northwest kiosks at MCO don't accept input until 3:45am, and the first agents arriving were supposed to be there then but were running late. Checked my bag and made it to security (the one furthest away was the only station open, of course), then caught the tram and hiked to my gate.

While the first flight departed on time, I had two screaming children in the row behind me and one occasionally-crying baby in my own row. It was a bumpy ride when we approached Memphis due to storms in the area — bad enough we actually aborted the initial landing attempt and went into a 10+minute holding pattern before the not-as-bumpy landing. Made it to the gate for my next flight in time to have boarded it... if the inbound aircraft had landed. It was about 55 minutes behind schedule thanks to the weather, so my 8:31am CDT departure turned into a 9:14am departure (43 minutes late). Despite the extra time on the ground, the luggage handlers at MEM didn't manage to get my bag onto the BWI-bound plane; when they announced "all bags delivered" and mine wasn't there, the lost-luggage folks checked and said it was showing as still in Memphis. As a "We're sorry," they gave me a $25-off coupon for my next flight on Northwest or Delta and 1000 bonus miles. (When I called later to see if it had gotten onto the 6pm flight from Memphis they told me it wasn't in Memphis but had somehow — they didn't say if it was at Orlando (MCO) or Memphis (MEM) — gotten forwarded on to Greenville (GSO). It's supposedly coming in on a Delta flight around 8pm so it might get to the hotel by midnight.)

Anyhow, managed to get to the hotel and they gave me a lovely room with a harbor view and a king size bed... and a front door that sticks badly enough to require me to kick it in to get it to open. I didn't want to deal with this on an empty stomach — other than the pseudo-cookies served with the beverage service on both flights, I'd not had any food all day and it was 1:40pm when I got to the room — so I grabbed a cuban sandwich and fries from the hotel restaurant (and while there was no napkin or silverware when I sat down, the waitress did get me them (a) without my having to ask and (b) before the food arrived). Unfortunately the computer went down and they couldn't get me my check for over 10 minutes. The waitress was apologetic, the manager was nowhere to be found, and my attitude was that they could either give me my bill or comp the meal. She was writing out the bill longhand when the computers came back up.

Back to the room and sure enough, the door's stuck. I call to complain and they say they'll move me... but there are no king-bed/harbor view rooms available and cleaned. I went down to chat with the front desk (planning on escalating to the manager on duty), but the clerk was able to move me (from 2811 down to 2411) more or less immediately, and she also clarified the Internet billing ("Pretend you're paying, but the actual fees won't hit your folio") and agreed to follow up with the restaurant management about the lack of availability of the restaurant manager when the computers were down.

Spent the rest of the afternoon catching up on the usual timewasters (mostly personal and work email, Facebook, IRC, LiveJournal, Project Runway, Top Chef, and Twitter) and writing up more of both of the OTCW and LISA trip reports.

For dinner, the concierge recommended a seafood-and-pasta joint a couple of blocks away. I was tempted by one of the house specials — a 12-ounce lobster tail stuffed with shrimp and scallops and topped with crab imperial. Very large. Very tasty. And with a salad and soft drink, very much an entire day's per diem.

Saturday, October 31

When I woke up, the message-waiting light was flashing. Turns out the airline did drop off my bag and it was waiting for me at the front desk. For the record, yes it was tagged correctly (with the MCO-MEM-BWI barcoded sticker), and it had the baggage-expedite tag from Delta (with the routing to BWI/DL 1886 via ATL/DL 5149, so it did get here via Atlanta from Greensboro; the GSO-ATL flight was 86 minutes late, landing at 6:04pm instead of 4:38pm, and the ATL-BWI flight was 20 minutes late, arriving at 7:53pm). It apparently got to the hotel around midnight. Once I got it to my room and unpacked, it even appeared as if everything was present and intact.

I did a quick read-through of the conference sessions online, and a quick walk-through of the conference space (still mainly a medical conference of some sort with most folks in business formal drag). It looks like at least 83% of the technical program that interests me all week is in the same ballroom salon, and the two workshops (Sunday and Monday) are across the hall at the far end of the fourth floor from the elevators and escalators. I escaped before Anne and Devon could put me to work helping set up Registration.

I swung back by registration shortly after they opened at 4pm, got my badge and shirt and paid for my workshop. Realized the registration desk had neglected to give me my bag and proceedings USB drive, so went back to get that. I then participated in the scavenger hunt. This year's card moved us away from the 3x3 bingo grid of pictures and required some actual conversation. There were 15 lines, each with a statement (like "...is on the USENIX Board" or "...is teaching a tutorial" or "...works at a college or university") and participants needed to get 10 of the 15 lines signed. I had ten done by 6pm, but by the time I'd gotten back to the member services desk to claim my prize they'd closed. So I went over to the Newbie BOF, er, Welcome Get-Together. Had munchies, heckled the presenter (it's okay, he heckled back), and hung out and caught up with friends. Headed out with six others — Marybeth, Bryan, Dave, Tom, Maurita, and Dan — to Mo's Crab and Pasta Factory. I had a house salad and a jumbo lump crab cake appetizer and that was enough. We got back to the hotel with enough time for a quick jump in the (warm) pool — the hotel doesn't have a hot tub, though the nearby Marriott Courtyard does and supposedly we have access — before heading back to Tom's suite with Matt and Jesse to talk LOPSA, politics, publishing, USENIX conference program chairing, and other topics while drinking some really choice aged rum Matt brought.

Sunday, November 1

Today was the University Issues Workshop. To my surprise when I paid for it yesterday it's only a half-day workshop (last year it was a whole day), so it cost the University half what I thought it would. (It was always set up as a half-day this year; I just never looked at that fact in the brochure.)

In the afternoon, I hung out in the 3rd floor (Grand) lobby and caught up on email and web while hallway tracking.

For dinner, six of us — Kyrre, Lee, Matt, Æleen, Michael, and I — went across the street to Fleming's Steakhouse. They had a Sunday night prime rib special: Choice of salad (I went with Caesar), 12-oz prime rib, choice of side (loaded baked potato), and dessert (chocolate lava cake) for $40, plus a wine flight of three shirazes (2006 Peter Jehmann, 2007 Ballast Stone, and 2005 Moyer; the first and last were really good, the middle was okay but nothing to write home about).

Monday, November 2

Today was my first and only free day, with no travel or meetings scheduled, since Sunday, October 25th. Did some minor email catch-up, wound up having some surprise LOPSA meetings, had an Irish pub lunch with some of the LOPSA tech team, and did some more hallway tracking.

For dinner, Carson and I went to Charleston, Cindy Wolf's restaurant. It's traditional French-inspired with tasting portions; you build your own experience from the 15 to 20 courses on the menu. I started with the rich lobster bisque with curry (cream-based lobter bisque with sherry and curry, served hot, poured (tableside) over chilled lobster meat and an herb oil), went on to the veal sweetbreads (fantastic; crispy outside, moist inside, with a properly subtle flavor) and the rabbit confit (over sausage risotto) before finishing with the duck breast (over white beans; basically a deconstructed cassoulet). For dessert, I went with the caramel mousse, served between layers of dehydrated whole-apple slices, with a small scoop of apple cider sorbet and some chopped roasted pecans.

After dinner, we went back to the hotel where I did some more hallway tracking (much of it raving about the dinner) until it was crash time.

Tuesday, November 3

Tuesday's sessions began with the Advanced Topics Workshop; once again, Adam Moskowitz was our host, moderator, and referee. [... The rest of the ATW writeup has been redacted; please check my LJ and my web site for details if you care ...]

After the workshop, I grabbed a quick dinner at Roy's Hawaiian Fusion with Carson and Brian; split an appetizer sampler and had the seafood plate (fried shrimp and grilled scallops over Thai-spiced black eyed peas). A quick run to the GLBTF BOF (where I wound up co-hosting since our scheduled host got stuck at a confused dinner), with three first-time attendees, then off to the LOPSA After Dark suite where I did some setup and prep (cutting the smoked gouda, cheddar, and swiss cheeses and a lemon and two limes for the drinks, and arranging the bar area) before heading out to crash.

Wednesday, November 4

The conference technical (as opposed to tutorial) sessions began this morning. My day began with the keynote session, which started with the usual statistics and announcements. This was the 23rd annual Large Installed System Administration (LISA) conference. Since the recipient had a class to teach at 9am, we broke with tradition and started with the SAGE Outstanding Achievement award. David N. Blank-Edelman — author, speaker, instructor, and conference organizer, received it for his 25 years of experience, expertise, and entertainment, and was rendered speechless.

With David safely off to teach his 9am class, we resumed with the numbers. We had only 815 registered attendees (which got up to around 860 thanks to additional walk-ins by Friday, but that's still down about 18% from last year). This was followed by thanks to the usual suspects: program committee members, external readers, chairs for IT and Guru tracks, USENIX staff and board, speakers, attendees, sponsors, exhibitors, and vendors. Program Chair Adam Moskowitz reminded us of the Birds of a Feather (BOF) sessions in the evenings, and the poster sessions on Wednesday and Thursday evenings. This year we received 38 refereed paper submissions (of which 4 were withdrawn) and accepted and published 12 papers.

We then went on to present this year's awards for Best Paper:

Finally, the annual Chuck Yerkes Award for Mentoring, Participation, & Professionalism went to Luke S. Crawford for his consistent, helpful, and astute assistance on the member forums.

Next we had our keynote speaker, Werner Vogels, CTO of Amazon.com, giving his talk "Ahead in the Cloud: The Power of Infrastructure as a Service." The marketing blurb said he was going to talk about how Amazon built "a reliable, flexible infrastructure that can scale up or down at a moment's notice" and how he would review the lessons they learned focusing on state management which is a key factor for scalability, reliability, performance, and cost-effectiveness. However, based on a bar conversaton the night before, he changed it to be more of an infrastructure-as-a-service talk about cloud computing. At a high-level overview it was a (not very pushy) sales pitch; while I'm glad to have a better understanding of the Amazon service offerings, I'm in no position to take advantage of any of them.

In the second session, Raymond L. Paden of IBM spoke on "How to Build a PB Sized Disk Storage System." He gave some architectural examples about what types of environments work at what scales. Some of the issues you need to consider include your I/O profile (where the cache should be, if the file system cache uses the system's virtual memory or its own, what the access types are like (large vs small files, read-only or read-write, single or multiple file systems, and so on). Develop your use cases and design the storage to handle 3-4 standard deviations above the mean data rate; benchmark on your real-life applications and usage if possible, and on synthetic benchmarks that represent your workload if not. Cost, capacity, performance, and reliability are all aspects of the eventual storage solution you need to consider. For those of us who remember when TB was large, this is very much more of the same: Many of the same issues are still with us.

For lunch, I went with Mark and David and Carson to Bagby Pizza where we pigged out on meat pizzas.

In the third session, I went to Raytheon's Michael K. Daly's invited talk, "The Advanced Persistent Threat." Critical infrastructures and the governments, corporations, and individuals supporting them are under attack by increasingly sophisticated cyber threats from hostile entities. Their goal is to gain access to intellectual property, personally identifiable information, financial data, and targeted strategic information. This is not simple fraud or hacking. It is intellectual property theft and infrastructure corruption on a grand scale. This talk discussed, mostly by analogy, the ways operators of national critical infrastructure are working to combat these threats and the collaborative partnerships that have been formed to strengthen our collective defenses. Some of the recommendations provided were to block uncategorized web sites at proxies, use split-dns, use split-routing, block common bad attachment types (and maybe ZIPs), block basic malware methods (SNR), and use collaboration groups.

In the fourth session I attended the Google Wave plenary session. The first speaker arranged for a backup and then didn't show up; the second speaker just plain didn't show up; the first's backup speaker was not entirely prepared. There was no good definition of what it is and why we should care, though he did differentiate between Google Wave the product and the wave technology behind it. He went into some of the technology details, but the demonstration bombed. As has been written elsewhere repeatedly, Google screwed up. Somehow they made the entire technology (not just the product) seem uninteresting. Worse, it's so preliminary there's absolutely no security (which is an afterthought at most). (Google has since apologized, apparently provided a better demo at their BOF Thursday evening, and is allegedly taking steps to keep this disaster from happening again.)

After the plenary session I did a quick run through the vendor floor. Small again this year, some products that might be of interest if we ever have money again, but no really cool gimmes that I saw.

For dinner, the 0xdeadbeef crowd — Brent, Janet, Dan, Bill, Adam, Steve, and I — took the hotel shuttle across town to the Prime Rib for dinner. I started with a Lagavulin 16 before dinner, then the 18- to 21-oz bone-in prime rib and a baked potato, with nibbles of the green beans that went by, and a perfectly serviceable merlot that Bill chose. Had an old fashioned banana split for dessert (and yes, it came with chocolate, strawberry, and vanilla ice creams).

Got back to the hotel in time to run to the BigFix hospitality suite where I got to see (and taste!) the cake made for us by Duff Goldman of (or more likely, one of the staff at) Charm City Cakes (as seen on Ace of Cakes on Food Network). Duff was there with his band, soihadto, and while the music was a bit loud for my tastes they did a good set.

While there I ran into Carolyn Hennings, who's one of the IT speakers tomorrow (on ITIL) and a former coworker back from Collective in Chicago. I introduced her to her IT coordinator (Doug Hughes) and brought her by the LOPSA After Dark suite for a free drink (as opposed to the cash bar that BigFix was running). I tried to ease her nerves about speaking tomorrow morning.

Thursday, November 5

Today started with another plenary session where Shane Canon of the Data System Group in NERSC at Lawrence Berkeley National Laboratory gave his invited talk, "Cosmic Computing: Supporting the Science of the Planck Space Based Telescope." Basically, his group supports the Planck space-based telescope. The large science projects are pushing the envelope of computing and storage and management. He is not an astrophysiscist, cosmologist, rocket scientist, or even computer scientist; he's a member of the Planck Consortium. He talked about the science, the Planck Mission, the data pipeline, NERSC, big data, and its challenges.

The science is looking at the cosmic microwave background (CMB), where about 400,000 years after the Big Bang, the expanding universe cools through the ionization temperature of hydrogen (where a proton and electron become a hydrogen atom). Cosmic meaning filling all of space; microwave redshifted by the expansion of the universe from 3000K to 3K, and background is the primordial photons coming from "behind" (relative to us) all astrophysical sources.

Why do we care about the CMB? It's the earliest possible photon image of the universe, and its existence supports a Big Bang over a steady state cosmology. It lets us learn details abut cosmology and ultra high energy physics. It was initially discovered by Penzias and Wilson in 1965 while they were improving transmitters for Bell Labs (which won them the 1978 Nobel Prize for Physics). There are other efforts to measure CMB, such as ground-based (AMiBA, CBI), balloon-based (Boomerang, MAXIMA), and space-based (COBE, WMAP). (COBE got Mather and Smoot the Nobel for Physics in 2006.)

The Planck mission is to measure the CMB fluctuations to provide the sharpest picturever of the young universe (at 380000 years old). Objectives include understanding more about the Big Bang, the structures of the Universe, dark matter, inflation theories, and so on. The satellite (microwave telescope) is a joint ESA/NASA mission launched this year, a 2+year all-sky survey from L2 at 9 microwave frequencies from 30 to 857 GHz, launched May 2009. The data set is huge: O(10^12) observations, O(10^8) sky pixels, O(10^15) GFlops/map, and so on. The detectors have to be very cold, so there's cryogenic cooling involved (which is why we're limited to 2 years or so). After 6 months it'll have a full 360-degree sweep view to map the universe. The resolution and accuracy of Planck is tremendous compared to WMAP.

Formally, the data is beamed from Planck to various ground stations (run by ESA). Some minor analysis is made before LFI is sent to one data processing center and HFI to another. They clean it up, filter out the known noise, and send it on to other places like NASA/JPL's Infrared Processing and Analysis Center (IPAC). NERSC has computational resources that NASA doesn't have, so while they're not formally part of the chain they have both HFI and LFI data sets.

The CMB is a time series of data and signal strength, used to build a map. There's an iterative process to remove known bad data and get a Gaussian best fit. The maps are used to build power spectra. The data looks to be increasing by 1000x over the next 15 years. Most recently, we had the first on-the-fly siimulation capability in 2008 and the first on-the-fly Monte Carlo sim/map (100x FFP) in 2009.

NERSC is the flagship computing center for DOE's Office of Science, operated by LBNL and is focused on delivering productiong HPC to nonclassified research. They began in 1974 to support fusion science and moved to LBNL in 1996. They have about 3000 users, 400 projects, 500 code instances. They ocus on unique resources, with high end computing and storage systems and interfaces to high speed networking, allocated across the multiple projects (and the mix of projects changes a lot over time).

System-wise, their big system is a Cray XT4 ("Franklin"): 9740 nodes, 38288 Opteron cores, 8GB of memory per node, 26 flops sustained SSP (355 Tflops/s peak). The new system ("Hopper") planned for 2010 production is a Cray XT5 at 3-4x Frankin's performance. Storage wise they have 450TB storage on Franklin with 25GB/sec bandwidth; Hopper will have 2PB+ storage and 70GB/s bandwidth. (The speaker runs the NERSC Global FIle Syste (NGF), 400TB/5.5 GB/s, based on GPFS.) Archival storage has 60PB capacity with 7PB in use, 10 Sun robots and a 130TB disk cache.

So what about the big data itself? Many of the programs at NERSC generate a lot of data. PB data sets will soon be common for climate (generating 10s of PBs), genome (0.5PB this year, doubling thereafter), particle physics (16 PB/yr once LHC is back online), and astrophysics (5PB/yr) projects. Because the data is shared between institutions we're seeing community growth; often the folks collecting the data aren't the only folks analyzing the data. In 2003 NERSC changed from being a data source to being a data sink.

There are keys to success for working with big data, including:

The second session was Carolyn Hennings speaking on "Is ITIL All Theory and No Practice?" Her hypothesis is that everything we do in IT can be mapped to something in the Information Technology Information Library (ITIL); her talk was a very high level analogy-based set of evidence to support that. As IT professionals we provide services. A service delivers value to the customer, creates a desired outcome, and without their ownership of cost and task. A customer pays money; a user is concerned with whether it works for them and whether it helps them get something done. A business service gives a direct value to customers; an IT service gives indirect value. For example, at a gas station you pay for gas and fill your tank and drive away. You get IT services at the pump which are indirect.

She talked about the ITIL version 3 live cycle of service management. First, there's the strategy (what and why); second, the design (who and how); third, the transition (make it so); and finally operations (keep it going). Surrounding it all is continual Service Improvement (make the process better).

One problem with the talk is that the speaker assumed a level of knowledge in the audience as to what ITIL is, and I don't believe that everyone did. Once the question-and-answer period began it actually got more in-depth, such as recommendations to become compliant. Adopting ITIL is a huge organizational change. Bottom-up and top-down both give resistance (people hate change). For steps, where are you today, what does the customer need, where do you want to go, where are the gaps or problem points? Adapt the relevant portions of ITIL to those. You need to change attitudes, behaviors, and culture. Documenting policies and procedures is all well and good, but the organization has to embrace the QoS aspect or they're just words on the page (web or paper).

For lunch, I went out with Carolyn, Chris, and Mike to the nearby sushi place, where we talked more about ITIL before segueing into general conversation. We had to rush back to the hotel in time for the third session, where Elizabeth Zwicky gave her invited talk "Searching for Truth, or at Least Data: How to Be an Empiricist Skeptic." Always an entertaining speaker, she talked about empiricism and skepticism, mainly using examples from her experience. To oversimplify, being a skeptic is a way of looking at the world, where science, sysadmin, and security interact. "That's interesting. I wonder what I could find out about it." Numbers are your friend: They help with planning, troubleshooting, security, and not falling for pseudoscience.

When it comes to recognizing data there are 3 questions: Is this data, what is it data about, and what conclusions can we draw about it? Hearsay, numbers without context, and conclusions are not data. Data is observations, self-reports, and numbers in context.

Statistical skepticism means asking questions. For example:

If it is data, what is it about? "47 users complained about slowness yesterday" is real data but more about the users than about the network. Most data is about lots of things; user complaints could be it's really slower, or there's a new app, or they're just not happy.

What conclusions can you draw from the data shown? It's easier to find data than truth, so be very cautions in the conclusions you draw.

Finding data needs basic tools, including a programming language, programs to look at the guts of things (like dtrace, truss, wireshark, tcpdump, profiler, etc.), and programs to make graphs/pictures (spreadsheet (excel, graphviz, keynote, gnuplot). Basic knowledge? Regexes, some SQL, some XML, and basic statistics.

When finding data, do you (a) mine existing sources, (b) cause new data to exist, (c) simulate or extrapolate, (d) share with others, or (e) make stuff up [like this talk].

Once you have data, what do you do with it? Maybe fascinating things will just jump out at you. Maybe just ask why. Maybe analyze it. Sanity checking is important.

After the break (pretzels but only diet caffeine, which annoyed some people), I went to Keith Scott's invited talk, "Delay/Disruption Tolerant Networking." At a high level this discussed the possibilities of interplanetary Internet. To overssimplify it's UUCP or Bitnet all over again, thanks to the time and distances involved. An interplanetary Internet would have large distance, intermittent (but generally scheduled) and expensive connectivity, and no end-to-end data path.

Delay causes disruption: Stock TCP implementations fall off quickly with distance, and application-level timers expire. Also, there are only 3 antennas that can talk to Mars, and only one at a time, and you don't have time on it. Why tolerate delay/disruption? IP architecture assumes end-to-end path, cheap round trips, etransmissions from the source are a good way to provide reliability, end-to-end loss is small, etc.

DTN uses a store and forward approach. They refer to Bundle Protocol endpoints or applications by name... "I don't know where www.example.com is, but it's on Earth so go that way." Example URIs include:

IP routing builds a current picture; DTN can route based on time and schedules since you know what things will be like later. Protocol mechanics can include status reporting flags to report on receipt, custody, transmit, and use spearate report-to addresses. Routers should only trust their immediate neighbors.

After the sessions ended I did a quick swing through the poster sessions (where the presenters are available for one-on-one discussions on their work) before the reception. They actually had good food (especially for hotel catering), with antipasti of artichoke hearts, salami, pepperoni, seared ahi, and prosciutto, tortellini carbonara with peas and prosciutto, pumpkin ravioli, and pepperoni pizza. No dessert though (which was a theme of the tutorial lunches too, now that I think of it).

After the reception was wrapping up (and I didn't win anything in the raffle), I headed up to the Roosevelt Suite for the Scotch BOF. It was small this year; we only had two scotches and the last of the good rum plus a lot of good chocolate and conversation.

Friday, November 6

This morning started with a plenary session by Bruno Michel of IBM entitled "Towards Zero-Emission Data Centers through Direct Reuse of Waste Heat" or more colloquially, "Green Data Centers." In our data centers, we're effectively turning electricity into heat and getting hot air back. High-performance liquid cooling allows datacenters to operate with coolant temperatures above the free cooling limit in all climates, eliminating the need for chillers and allowing the thermal energy to be reused in cold climates. They have demonstrated removal of 85% of the heat load from high-performance compute nodes at a temperature of 60 degrees Celsius and compared their energy and emission balance with a classical air-cooled datacenter, a datacenter with free cooling in a cold climate zone, and a datacenter with chiller mediated energy reuse. The talk explained how their method reduces energy consumption by almost a factor of two compared to a current data center and reduces energy cost and carbon footprint by an even larger factor.

The Q-and-A session asked about what administrative changes would be required (ideally minimal changes), how long the liquid cooling could last (today's technology gives a 10-year lifespan for water cooling), and if any of this were in production yet (no). Perhaps the best question was, "This solves global warming, photovoltaic cells, and desalination. How do you feel about it?" His answer? "Great."

One of the problems with this year's conference — and if this is the worst thing about it, that's fantastic — is that virtually every time slot had something I was at least somewhat interested in based on the abstract in the conference program. Yes, some of the talks didn't live up to my expectations (as noted previously), but even so, 11 out of 12 is a pretty darned good ratio. As you might have guessed, this was the one block where I was at best indifferent as to what to attend, so I took the chance to treat it as a Hallway Track block.

For lunch, a couple of small groups joined to form a group of 10 in the hotel restaurant. I had a burger (which was decent enough). I just didn't feel like going out.

After lunch I decided to hang out in the "Interviewing and Job Hunting Skills" guru session, mainly to be able to heckle Adam and Tom. In addition to the standard good advice for both candidates and interviewers (be honest, be timely, remember it's a two way street, know when to be quiet, salary negotiating tactics, ask the interviewer to specify the level of detail desired, do yur research, don't feel shy about asking questions, target your resume and cover letter for the specific job), they had some other advice. It's probably worthwhile to have a public Internet presence (but remember that anything you do is there forever). If you move to a new city, contact your local user group for face-to-face networking. Certification depends on the certificate (some are meaningful and some are not; some are relevant to the job and some are not). Quality of life is important; having some things on your resume may cost you opportunities. One example is if you list your open source projects will the prospective employer think you'd rather work on open source stuff than your assigned tasks? And if they do, would you really want to work for them?

In the final block of time, the always entertaining Dan Klein spoke. CERT/CC was founded over two decades ago, SANS was created 20 years ago, and USENIX has been providing a forum for world-class security experts for even longer than that. Yet in spite of these prominent centers of excellence (and their sage advice), we keep seeing new attacks, new exploits, and new vulnerabilities-in simpler terms, "same stuff, different day." It's not just because there are more bad guys out there (although there are), and it's not just because the bad guys are smarter (but they are). In Dan's opinion it is because we are working with tools and systems that are fundamentally flawed. Our house of bricks is built on a sandy foundation, and we now find ourselves at a crossroads-the same crossroads that every technology has faced in human history: start over again and do it right from the start, or keep doing it wrong until it all falls over in a heap.

Dan's talk took a lighthearted look at some really bad news: Either we will have to spend a lot of money redeveloping our basic tools, infrastructure, and operating systems properly, or we will have to spend a lot of money patching bugs and regularly recovering from security disasters (and continually be faced with the same basic problems). With one way we have a lot of unhappy people now, while with the other we will have a lot of unhappy people later.

In the 1950s, the architect Frank Lloyd Wright was given a tour of Pittsburgh which ended atop Mount Washington. He was asked, "What should we do?" In his inimitable style, he looked around and said, "Raze it and start over." Having lived in Pittsburgh for 35 years, Dan agrees that he was right. He has worked with computers for as long as he has been in Pittsburgh. Frank's advice is strangely apropos to our chosen profession, too....

For dinner this evening, I took a small group (Dan, Philip, Trey, and Walter) out to Fogo de Chao for meat on swords. Delicious as usual, though the gauchos seemed to be off-by-one on the doneness scale (what they thought was rare was really medium rare). I had cow (top sirloin, bottom sirloin, bacon-wrapped filet, regular filet), pork (baby back ribs, sausage), chicken (bacon-wrapped breast), and lamb (chops and loin), as well as the salad bar (from which I had more meat) and the veggies. I chose not to do dessert (I had a bite of Philip's, with his permission) as I was too stuffed.

I got back to the hotel around the same time as the LOPSA After Dark folks were moving the supplies up to the Roosevelt Suite for the Dead Dog party. I set up the kitchen, stocked the fridge with the gallons of beer and white wine and the extra mixers, moved a table out of the bedroom into the hallway to act as barstand for the cups, the booze, the red wine, and the mixers, trying to keep traffic flow manageable. I also set out some of the food and helped control the chaos as other folks started to arrive before we were ready for opening. During the party itself I mostly hung out in the bar area as a bartender/facilitator (not in my "bartender drag" though). Eventually I went out to nibble some of the munchies, and chatted with folks throughout the evening (both at the bar and in the suite). We finally closed down around 1am.

Saturday, November 7

The final leg of the almost two weeks away. Managed to check out of the hotel without difficulty (and they'd correctly zeroed out the daily Internet charges), say my Goodbyes to those waiting to head out to the Cryptography Museum, get to the shuttle, get to the airport, check my bag (overweight but I can't get it below 50 pounds so oh well), clear Security, grab a late breakfast/early lunch at the food court, and eventually board the aircraft. We depart and take off on time, the three babies within earshot don't cry or scream much, we land and arrive on time, and they didn't lose my bag this time. I find my driver and we head off to the not-really-a-limo and head out of the airport.

And in the ramp curving between the airport road and the access to get to I-275, the right front tire blows out.

I am not making this up.

They call a second driver and car to transfer (whee, transfers on the side of the road next to 55 MPH traffic... haven't had to do that since, um, 1985?), and make it the rest of the way home without further incident. Whew. Process most of the postal mail, pay some bills, unpack, nuke up some taco pockets for dinner, and crash.



Back to my conference reports page
Back to my professional organizations page
Back to my work page
Back to my home page
Last update Feb01/20 by Josh Simon (<jss@clock.org>).