The following document is intended as the general trip report for me at the 29th Systems Administration Conference (LISA 2015) in Washington, DC, from November 8-13, 2015. It is going to a variety of audiences, so feel free to skip the parts that don't concern or interest you.
Today was my travel day. Headed off to the airport a bit early to avoid stress. Parked near my usual spot but on the down-ramp heading out to save a little time and mileage when I get home next week. Got to the terminal shuttle as a new one was arriving, was first in line in the Priority lane, and cleared Security without incident. Got to the gate with plenty of time, boarded without incident, and had my free drinks (screwdrivers since it's morning and I don't care for tomato juice). Also managed to get a banana and a brakfast cookie to offset the booze, which was good.
Took the subway to the hotel, got my room key, and headed upstairs. Unpacked and got settled in before heading down to see who was around for lunch. Ran into Derek (who'd already eaten) so headed out in the light rain to Hot N Juicy Crawfish for a crawfish po'boy and cajun fries. Got back to the hotel and napped a bit before heading down to the lobby spaces to hang out with the other early arrivals until Registration opened.
Registration opened a bit before 5pm, and I got my badge... though it took the gentleman three boxes to find my name. One would think that "know the alphabet" would be a requirement for the person handing out badges; apparently, one would be mistaken. Hung out for the next couple of hours at the Welcome Reception, catching up with friends old and new. At 7pm I met up with John Arrasjid, Mike Ciavarella, Rudy Gevaert, David Nolan, and Matt Simmons, and the 6 of us headed out to dinner. We went down the block and around the corner to Afghan Grill. The table split a couple of sampler appetizers and I had the Qabili Palao with lamb, served with saffron rice and covered with sauteed sweet carrots and raisins. Delicious.
When we got back to the hotel I joined an ever-increasing circle that included most of my dinner companions, the 2 current program chairs, 2 other past program chairs, and some folks I haven't seen in 1–5 years. I passed on ordering a $14 caipirinha and headed up to bed around 10pm since I'd been awake since 4am.
Ah, first-night-in-new-bed syndrome. I was up on-and-off all night, and finally gave up around 6:30am. Caught up on email, showered, and headed down to the conference space; swapped out my badge holder for a Organizing Committee one (because Casey didn't want me to walk around with the old black LOPSA one that people confuse for the USENIX Staff one). Said Hello to a few more people as they arrived then did Hallway Track for the bulk of the morning. I managed to get my badge reprinted (they'd used the long form of my name instead of my preferred "Josh") and worked with the registration folks to Macguyver my laptop bag strap.
After lunch on what will become the exhibition floor, I headed to my afternoon workshop: "Plays Well With Others" or Improv for Sysadmins. It was an overview of improv exercises with discussion as to what we might learn from them, why we might use them in our work or even home lives, and any additional comments.
I found some of the listening and ideation exercises useful, and might use some of them as ice-breakers at VOICES team meetings.
After the workshop ended, I dumped my laptop in the room and went out to Fogo de Chao with Nomad and Rowan. I skipped the chicken and the pork sausage when they came around, but still had bacon-wrapped filet mignon; flank steak; lamb, both chops and leg; parmesan-crusted pork tenderloin; and sirloin 3 ways (bottom, top, and house special). Plus the sides (mashed potatoes, crispy polenta, and fried plantains) and the salad bar (where I stuck mostly to meats and cheeses plus some sesame-dressed green beans). And a caipirinha, because as long as I'm gonna blow my per diem out of the water I may as well enjoy myself.
Because my ankles and hips were irritated with me for standing up for the duration of the workshop, I merely said some Hellos at the game night space before heading upstairs to medicate and lie down briefly.... I napped until 10:30pm or so, when I decided I didn't want more drinks or dessert so I'd write up some more trip report and go to bed.
Today was my mostly-free day. I slept in until nearly 9 — after not getting up at 6 or 7 — and had a leisurely morning. I did the Hallway Track both before and after lunch, and crashed the tutorial/workshop lunch (which was really tasty today: A good variety of both leaf and orzo salads, white bean soup, ratatouille, olive oil-poached cod, and chicken breast, with tiramisu, cannoli, and lemon-rosemary crumb cake as dessert).
For dinner, I went across the street with Travis Campbell, Ben Cotton, David Nolan, and Dan Rich to Tono Sushi. The drizzle was mildly annoying but not a huge deal. They seated us upstairs; the rest of the upper level was a group of 20 in the side room and a group of 30 behind us in the big room. We got the complementary bean sprout salad for starters, then miso soup, and then sushi. We'd asked for a dragon roll and a rainbow roll and said to let the chef decide what else to give us. We wound up with smoked sea bream nigiri, mackerel nigiri, an eel roll, a salmon roll, the monthly special (calamari and jalapeno) roll, and 5 hand rolls (3 tuna and 2 yellowtail). Excellent food.
Got back to the hotel, chatted with new arrivals in the lobby, dropped the umbrella and jacket off in my room, then went downstairs to the lobby bar to hang out with folks. Despite the carbs in the barsnax I was starting to run down so I headed upstairs a bit before 10pm, wrote more of this trip report, and crashed.
Despite sleeping in yesterday, my body was up and awake by 7am. I checked my work email (and answered a time-sensitive missive from my grandbosss but ignored emails from the ticketing system). Got downstairs before the coffee service vanished and saw a couple of newly-arrived folks.
For the first time in 18 years, I can't truthfully start my Tuesday writeup by talking about the Advanced Topics Workshop; ATW ended in 2014. (I made a point of wearing my ATW commemorative t-shirt from last year's final workshop.) My morning was unscheduled so I did the hallway track again. Lunch was Taco Tuesday and decent enough, though the desserts (cold churros, flavor-free tres leches cake, and lime tarts that had very little lime flavor) left much to be desired.
After lunch I went to the System Administration in Higher Education workshop. It was effectively a forum for IT staff from higher education institutions to share ideas, concerns and experiences with peers from other institutions. Our topics included practical ideas for improving support, how to set priorities, security in a higher education context, budget concerns, and participation on campus and getting involved outside the day-to-day job.
I was part of the "dress up nice" group for the 0xdeadbeef dinner Tuesday night. We wound up going to Charlie Palmer Steak. I wound up getting the French onion soup en croute (instead of bread or croutons under the gruyere cheese there was a large dome of puff pastry on top of the bowl; the broth included chunks of brisket and calvados), a 12-oz bone-in ribeye, and we shared asparagus, hand-cut fries with chipotle mayo for dipping, and twice-baked truffled potatoes with bacon and chives. We had a malbec with the meal (2 bottles for the 4 of us having wine), and then I had a glass of orange muscat with my dessert (chocolate pecan tart, carmelized banana, poppyseed ice cream, with chocolate mousse and a tuille). And then finished Adam's muscat because you can't let booze go to waste.
Adam, Branson, and I held a mini Rocky Horror Picture Show BOF outside of the NOC when we got back, and that kept me out until past midnight. I finally got to sleep around 1am.
Despite not getting to bed until 1am, my brain insisted it was absolutely sure someone else was in my room around 3:20am. That wasn't that much fun (especailly since my brain was wrong). I tossed and turned much of the rest of the night before giving up a bit past 6:30am. Caught up on work email, decided to declare bankruptcy ("I'll never catch up") on both Facebook and Twitter, showered, dressed, and headed downstairs for the sessions.
I got to the announcements and keynote session a bit early to be able to get a seat with a table and electricity. Program chairs Cory Lueninghoener and Amy Rich began with the usual announcements and thanks. This is the 29th LISA. The program committee had 225 submissions, and accepted 44 talks, 23 tutorials, 18 mini-tutorials, 12 workshops, and 5 papers and posters, and as of shortly before the keynote we had about 1,104 attendees. Thank you to the staff, committees, liaisons, coordinators, sponsors, and expo vendors.
LISA 2016 (the 30th) will be December 3–8 in Boston and John Arrasjid and Matt Simmons will be the program chairs.
This year's awards:
- Best Paper — A"Hyperprobe: Towards VM Extrospection," Jidong Xiao, Lei Lu, Hai Huang, and Haining Wang
- LOPSA Chuck Yerkes Award — Jaimie Riedesel, aka sysadmin1138
Today's keynote speaker was Mikey Dickerson. Each year, the Federal Government spends more than $80 billion on information technology. Too often, these projects are over budget, behind schedule, or on-track to fail. But the stakes couldn't be higher. Millions rely on the basic services these systems support-like care to disabled veterans, health insurance to those who have never had it, and loans to first-generation college students.
The U.S. Digital Service partners the best talent from the private sector with dedicated professionals already in public service to transform the way government works for the people it serves. Mikey talked about how this work is going and how we can help. (This was a follow-up from his talk last year.)
- Be careful what you post online.
- Dress nicely 'cause the press will pillory you.
- Media is weird.
Observation: The US is a 226 year-old open source project with 538 committers in constant flame wars in the media.
Some specific lessons:
- The system does not care about your feelings.
- Cruft you don't use can still hurt you. Keep it simple if reliability matters.
- The older a system is, the harder it is to change.
- We have to get past "root cause analysis." With sufficient complexity there is more than one root cause.
"When you debug a distributed sytem or an OS kernel, you do it Texas-style: You gather some mean, stoic people, people who have seen things die, and you get some primitive tools, like a compass and a rucksack and a stick that's pointed on one end, and you walk into the wilderness and you look for trouble, possibly while using chewing tobacco." -- James Mickens.
A lot of times, "not news" is what you're trying to accomplish."
I was planning on attending Doug's mini-tutorial "Solving Problems and Identifying Bottlenecks with truss and strace," but the room was already overfull with all standing-room spaces filled before I could get in the door. Luckily I know the speaker so I can ask him things after the conference. I wound up doing the hallway track instead, since none of the other sessions interested me.
After a quick tour of the vendor floor (no really good swag, though I grabbed a couple of t-shirts, some Disney mouse pens, and a pair of Google socks), I grabbed some lunch (mediterranean theme: hummus, shrimp-n-cilantro salad, olives and artichokes and other crudites, chicken and lamb kabobs (I skipped the veggie kabobs), and spicy meatballs), I headed over to Derek's mini-tutorial "Contract Negiotiation for System Administrators," which was geared towards SA/vendor negotiations and not, say, the hiring process. (The current slides are at http://megacity.org/slides/.
Some things to seriously consider:
- Want FOB Destination, or as close to destination as possible, since that's when it becomes yours. Any problems in shipping? It's their problem until you own the gear, then it's your problem.
- Contract should be last resort; it's for handling things when they go wrong.
- Read the EULAs and SLAs. There's often stuff hidden in them (both good and bad).
- Force majeure is the bane of your existence but usually present in most contracts. An example is "Loss of utility power" in force majeure — it invalidates the "provide my data center power" clauses.
- "Reasonable" and "best effort" are not the same. "Good faith effort" is much closer to "reasonable."
- Most things are negotiable, but not everything (e.g., MSFT EULA).
- Documents incorporated by reference and are subject to change should have a notification clause AND an opt-out clause. Consider also an opt-out clause for when a badcompany buys your wonderfulcompany.
- Include ending conditions:: What do we want to happen when the contract is up for renewal or change? Should it auto-renew (and at what rate, or month-to-month), or terminate completely?
- Consider branding/marketing: Can they use your name/reference? Can we use theirs? What about our competitors?
- Redline review is essential. (See the slides.) And have your management and attorneys sign off on it. Make sure your attorneys understand why you're making wacky changes. Your lawyer is on your side and you need to present a united front to the vendor.
After the break I went to the IT track for Tim Hockin's "The Latest from Kubernetes" and Nicole Forsgren's "My First Year at Chef: Measuring All the Things."
Tim was up first. This was not an introduction but aimed at the newer cooler features. Containainers are different for deployment, management, monitoring, updates, scaling, replication, and so on. It's a different way of managing systems. Kuberneetes is open source driven by Google and written in Go. Kubernetes can be a pain to set up, but there are some ways around it (cf. Google Kubernetes Engine).
New features in v1.1 include:
- iptables kube-proxy
- L7 http load balancer
- quotas by CPU or memory
- network plugins
- cluster auto-scaling (grow/shrink as needed)
- persistent volumes
Nicole was up next. She rubs science on things to make DevOps, ITOps, and so on better. Where to start? Check that is is happening and start at the highest level. Metrics provide opportinity for communication and alignment. IT and Dev can be doing the same things under different terms, or different things with one word, and communications can clear that up. If you're still not sure, ask people what's important to know about (for us, our customers, our immediate management, etc.)?
Remember that they don't need to be perfect, and if they don't work you can toss them. You want to use these to increase communications across the organization. Start with the important things to the organization or team. Benchmark (within the team, with other tames, throughouth the company or industry). Things should be communciated and visible throughout the environment — to improve communications and start discussions.
Having seen the vendor expo floor at lunch I decided to skip out on the happy hour (in retrospect that was wise; I heard nothing positive about the pseudo-dim sum food served there — the best comment was that it tasted like dim sum made by someone who'd only seen a picture) and instead went to dinner with Travis Campbell and John Sellens at Open City for burgers. We got back to the hotel and Nicole Forsgren was sort of stuck in a decision loop (an "to eat or not to eat, and if so where" kind of thing), so Travis, Rowan Littell, Mark Llamourine, and I dragged her out to Medaterra for dinner (for her) and dessert (for me). I wound up getting the house-made baklava and since it was still Happy Hour there I added on a $1 margarita.
As we got back to the hotel I overheard someone mention that the scotch BOF was that evening, so I dropped in on it. Had some decent Macallan 16 and a Talisker Storm Somethingorother and wound up schmoozing and chatting until about midnight, when I turned into a pumpkin and went back to my room.
My body is at least consisitent; it woke me up for good by 6:30am. Caught up on the work email (nothing urgent to deal with) and TwitBookPlus, wrote more trip report, and headed down to the conference level.
Today's sessions started with our second keynote, Christopher Soghoian's "Sysadmins and Their Role in Cyberwar: Why Several Governments Want to Spy on and Hack You, Even If You Have Nothing to Hide." He talked about the latest developments in government surveillance, their implications, and the special relevance these developments have for our sysadmin community.
His job is to explain technology to the lawyers. He gave a lot of anecdotes, referencing news stories from the past 6 or 7 years.
Some of the take-aways include: Everyone does unsafe things (opening PDFs from http or email, curl|sudo bash, reusing passwords, etc.), and we really should stop. We all know better. Encrypt your disks, encrypt your traffic, don't trust random URLs (including PDFs), and so on. And because we're sysadmins with theoretical access to Things, we are all targets of every government's surveillance and they'll go to work and personal addresses.
What can we do about this, especially now? It'll probably lead to the CS and engineering communities increasing encryption and security by default. So: Use modern technology (such as Signal for your smartphone; encrypts voice calls and text messaging). Encrypt your disks. Go to https everywhere on the web; kill plaintext on the Internet. Look at qubes-os.org (which effectively sandboxes everything).
After the morning break I went to Nicole Forsgren's last-minute replacement talk, "How Continuous Delivery and Lean Management Make Your DevOps Amazeballs." She talked about the latest research (which she did with Jez Humble and Gene Kim) that uncovers what really drives business outcomes of market share, profitability, and productivity as well as DevOps transformation awesomeness. These include continuous delivery (and what is most important when you do CD) and lean management (and what that means for us).
DevOps — technology, practice, and culture together — help turn IT from a cost center to a competitive advantage. The 2015 data was analyzed similarly to the 2014 data. IT performance is still predictive of organization performance. In high-performing IT organizations, you're 2x more likely to exceed profitability, market share, and productivity goals, and have a 50% higher market cap grown over 3 years (if you told them your stock ticker symbol). IT performance is measured by both agility (deploy frequency and lead time) and reliability (MTTR and change fail rate). High performers are more agile (30x more frequent deployment) and 200x faster lead times than their peers (that is, the time from code commit to code deploy), and more reliable (168x faster MTTR and 60x change success rate (up from 48x and 3x in 2014 respectively)).
Version control, monitoring, automated testing, and continuous delivery all correlate. Culture is also important.
They wanted to know what drives IT performance for predictions. Based on the correlations, they looked at Continuous delivery (test deployment and automation, continuous integration, and all production artifacts (including CM) in version control):+--> predictive of lower change fail rates / [ Continuous Delivery ] --> [ IT Performance ] --> [Org. Performance ] \ +--> predictive of lower deployment pain (esp. for Ops)
They also looked at Lean Management (WIP limits: drive improvement, visualizations to monitor work, and monitoring to make business):+--> Improved org culture (Westrum) -------\ / vvv v [ Lean Management ] --> [ IT Performance ] --> [Org. Performance ] \ +--> Decreased burnout
What about non-profits or organizations without local IT staff or use contractors who don't follow these practices? How do you bridge that gap to make them do this stuff? Every business is a software business, so delivering any value or efficiency requires doing best practices to get there. If your contractors don't care about that you may need new contractors.
How do you sell DevOps (which may be an anti-ITIL) to an ITIL-focused organization? ITIL can be very regimental with a lot of approvals and handoffs and so on. Try inviting those people to write up things for the Kanban board. CABs in the traditional sense don't work because they're too far removed from the work and thus not a good judge of the work. CABs are great at notification, not necessarily at approving things.
How can us invisible sysadmins take these numbers to our managers and not fall out of the conversation? By casting it as an investment and that IT Ops is part of the research and development process.
If you can't get management support you can backdoor by having ops and dev talk to each other without management approval. Get it working and then management won't shut it down.
That was followed by Branson Matheson giving Chris McInery's talk "Practical Advice for Small and Medium Environment DDoS Survival." Anyone running an Internet presence today is susceptible to denial of service attacks. The scale of these attacks has ballooned to over 400Gbps in the past year, and the barrier to producing these has dropped tremendously. While there are several proposed solutions at an Internet Engineering level, none is uniformly adopted to head off the issue, so it's left to individual site owners to make practical trade offs to protect themselves. In this talk, he discussed some of the trade offs and advice for what a small or medium site owner can do to survive in the current climate.
DDoSes are getting bigger and bigger, botnets are very easy to get to.
Know yourself: Have a core understanding of your environment, including how things can be impacted. What does your traffic look like? What protocols and ports do you need, and how open do they need to be? And what does recovery look like for you? Does it impact your business? How much downtime can you afford? Can you shut down some of your services to keep other services viable? Remember that it's a back-and-forth: Bad guys do something, we block, they find another way in, we block again, and so on and so forth.
Know thy adversry: Knowing what they're thinking can be irrelevant, but it might help inform their methodologies which can inform our defense. There may be multiple attackers with different motivations... and some may be smarter than you. The mitigations are the same and determined by what you know about your own enviornment.
Attacks can be at the network layer (vertical: any of the 7-layer model) or point (horizontal: border? servers? isp's infrastructure?) as well as the source (direct or indirect, my own botnet, coercing others via reflection attack).
Defense: Reduce the bad traffic. What effort at each level are you going to use? Are you going to do it yourself or have someone else do it? You can packet-filter at the network level (though it can become a bottlencek; cf. overrunning state tables). You and do ACLs and firewalls (especially at your upstream ISP). IPS can help and tends to be smarter (doing deep inspection) than a firewall (yes/no). Rate limiting helps, even at the application level (cf. application amplification).
Lunch was again on the expo floor. I had a lobster roll, a couple of barbecue brisket sliders, a mini ham sandwich, some of the appple-cranberry-pecan salad, potato salad, and house-made potato chips. I also did a quick run-through to see if I could pick up some swag for the rest of my team as a sort-of "sorry I've been out most of the past two weeks, have some crap" gift of sorts. I did get some Puppet stuff and more Google socks.
After the lunch break I stayed downstairs for Connie-Lynne Villani's mini-tutorial, "Interfacing with Humans: How to Manage in Prod Ops." Whether you call it Prod Ops, System Engineering, or simply "keeping it all working," ops managers face some particular challenges. How do you build new projects and services while solving all the production emergencies caused by the old, broken infrastructure? How do you juggle the demands of other teams in the company while keeping the site running? Above all, how do you give your team agency and keep them happy in a high-pressure, distraction-driven, 24x7 environment?
Humans as humans are very hard to systematize; they're very unpredictable. That's the hardest part about management. Also, production operations roles are hard; it's very thankless because nobody notices when things go well. As a manager, you (or someone you delegate who's better at it) should rave about the successes. It may feel like bragging — and it is — but everyone else does it so you should too.
Some takeaways include:
- Collaborative versus command-and-control styles
- Celebrate successes; treat mistakes as nothing special
- Nothing is more important than keeping your people healthy and happy (modulo things like ATC and hospitals and so on).
- After about 7 people you're at -1 capacity because of family life, vacations, sick leave, and so on. Plan for that.
- Just because you can do all the things doesn't mean you should.
For the final scheduled session of the day I thought about going to the Git mini-tutorial but rereading the description I was not the target audience, so I did the hallway track instead... then took a power-nap before the conference reception.
The reception itself was pretty decent. They did an Americana theme with a couple of diner booths and a soda fountain-like bar (no shakes or malts though). Foodstuffs included asparagus wrapped in either asiago puff pastry (hot) or prosciutto (cold); chicken or chickpea sliders; crab dip, spinach dip, and breads and chips for dipping; mini-cottage pies (mislabeled as "shepherd's pies" even though they're beef); potato parfait bar (mashed or sweet potatoes and a variety of toppings); and an ice cream sundae station. For what it was, it was a decent enough dinner.
After the reception we had another Rocky Horror Picture Show BOF with a bigger audience, which covered the time until the Organizers' Party at 10pm. I went upstairs intending to make an appearance, schmooze a little, and then go to bed around midnight.... and wound up closing out the party at 1:30am. Got back to my room... and I'd managed to either demagnetize or corrupt the room key, since the lock just flashed amber-and-red at me. Headed down to the front desk to get new keys, went back upstairs, and was in bed a bit after 2am.
And despite the 2am bedtime my body woke up for good by 6:30am. Showered, dressed, rebooted both the computer and phone which were misbehaving enough to warrant it, and headed downstairs to the conference to start my day.
For the first session I attended Jez Humble's plenary talk, "Lean Configuration Management." Configuration management is an essential ingredient in creating high performance IT. But how it's implemented matters. Jez presented the principles that enable high throughput and stability and the configuration management practices behind them, using models drawn from the Lean movement.
He had four take-aways:
- Throughput and stability are not a zero-sum game.
- Culture predicts information flow which predicts high performance.
- Build quality in at the source.
- Systems engineers are doing product development.
He referred back to Nicole Forsgren's talk yesterday: IT is important, and its performance is still predictive of organization performance. Refer to the 2014 and 2015 "State of the DevOps Report" documents for more details.
Nothing in the antepenultimate session blocks jumped out at me so I did the hallway track until lunchtime. Lunch today for me was some couscous salad, a tomato-cucumber with red onion (needed feta), some tilapia, and some chicken breast (and the proteins weren't overcooked; yay!). After lunch for the penultimate session block, I went to the Invited Talks track. First up was kc claffy's "A Brief History of a Future Internet: The Named Data Networking Architecture."
NDN is part of ICN, rethinking networking and the underlying protocol, moving chunks of content around instead of necessarily point-to-point protocols. NDN is 7 years old.
Evolution of communications (esp. over the past century): Phone network was focused on building the wires, and IP focuses on getting packets to the destination. The Internet, like telcos, is about getting something (be it voice or data or something else) from one node to another. NDN focuse on getting data from the cloud, but we don't care about source just that it's fast and cheap to get it. The fundamental idea is that most of what the Internet is used for nowadays is the latter.
Remember that the Internet is successful but the protocols are very old and not designed to support what it's doing. We want to improve trust and security, reduce complexity and cost, and enhance "fit" with applications. (Oh, and make it backward compatible, too.)
So if we don't want to continue the status quo of incremental patches and updates to TCP/IP, we can consider a new architecture based on lessons learned on how we use the Intenert todau: Data distribution communications model, secure the data not the channel for the security model. In the new model, the network ships data (bits it knows are needed), and it considers multicast better. New application development model; where developers can select or crate a security model up front.
There are two packet types today: An interest packet ("Here's what I want") and a data packet ("the content I wanted"); data packets are cryptographically signed (encryption is optional). Packets say what not who, so there's no source or destination. It's much bigger than content distribution and more than just a network of caches a la Akamai or the old MBone. Endpoints must and routers may check the signatures. Signatures themselves are themselves data packets. The namespace design conveys capabilities and structures trust.
Some big data applications like high energy physics are starting to use NDN. It's very bleeding edge right now; there's a lot of pain right now. There's a huge research agenda, broad wide deep and exciting. 10 campuses are working on it, the software is all open-source and online for free at named-data.net and GitHub.
Second in the block was CJ Estel's "Managing and Tracking Database Deployments." We've all been there, managing multiple environments that all have different SLA's and release schedules. With systems, managing configuration files can be automated with something like Chef or Puppet. With code deployment, tools can easily push out the latest release. Databases present a unique challenge where you can't just push out a current state. You have to identify and walk through every change that has been applied and likely have to install the changes in the same order throughout environments. In this presentation, CJ went over the tool that was developed in-house to address the complexity of database deployments. CoverMyMeds plans to open source this tool. The tool is extensible and at the time of release will support PostgreSQL, MSSQL, and MySQL.
DBdeployer is basically a way to abstract out database creation, reading, updating, and deleting regardless of the actual database type. (He ran through a mildly confusing demo.) MySQL isn't quite ready for prime time yet; we hope to have it done in the next 2–3 weeks.
In the final block I went to the closing sessions. Amy Rich thanked Xirrus and reminded us to fill out the surveys. LISA Build came out to tell us about the network setup and operations.
The closing keynote was James Mickens' "It Was Never Going to Work, So Let's Have Some Tea." It's difficult to administer large systems; Mickens argued that we should just give up. People today have too much self-confidence and every idea is a terrible idea from the get-go. He gave a very sarcastic rant about.... Well, about an hour or so.
A bunch of us took a group photo for Mark Roth, who was unable to attend this year due to Reasons, after the session broke up. I then dropped my stuff in my room before meeting with Peter Eriksson, Trey Harris, Philip Kizer, Dan Rich, Brian Sebby, and Dennis whose last name I didn't catch, and heading to Hot N Juicy Crawfish for dinner. Six of us more or less split 2 lb shrimp, 1.5 lb lobster, 1 lb crawfish, 1 lb king crab legs, 0.5 lb snow crab legs, 24 andouille sausage slices, 1 basket of soft-shell crab, 2 baskets of calamari, corn, and redskin potatoes (the seventh just had some hot wings). I escaped at $40 including tax and tip since I avoided the booze.
Got back to the hotel, dropped the wallet off, updated the budget sheets, did the bulk of my packing in advance of tomorrow's departure (remaining: CPAP, laptop, today's clothing, and toiletries), and headed up to the Dead Dog party. It started a bit slowly, since nobody was sure when the host would get back from dinner (my guess was 9:30pm, and when I got there at 9:15pm they were mostly finished with setup); I stayed for a little over 3 hours, having lots of good conversations, munchies (mostly high-end chocolates and some TIm-Tams), and drinks (Macallan, Talisker, and a 23-year aged rum whose name I forget). Alas around midnight or so I started running (further) down so I made my round of goodbye hugs and left around 12:30am. My key worked in my room door lock tonight so I went to bed reasonably quickly.
Alas, it's time to return home. I woke up twice (at 3am and 7:30am) due to biological necessities; I couldn't fall asleep after the second. I caught up on work and personal email and TwitBookPlus (more or less), showered, shaved, and finished packing the toiletries and CPAP. Had morning plans with a local friend so I didn't manage to do breakfast with anyone from the conference.
I did manage to do a late brunch or early lunch with Jimmy at District Kitchen. We were going to try Open City but the line was out the door. I wound up getting 2 eggs (scrambled) with bacon, toast, and hash browns, and it was fine. We finished up a bit after noon so we went back to the hotel where I finished packing, checked out, and headed out to the airport.
People had said online that there were long waits for the trains so I gve myself plenty of extra time. And then I had a short (less than 10 minute) wait for a Red Line train, followed by a shorter (less than 2 minute) wait for a Yellow Line train, no wait to check my bag and hand it off to TSA, no wait (beyond getting my boarding pass to display on my phone) at Security — and somehow I got TSA Pre for this flight so I didn't have to strip off my shoes and belt, or take my 311 bag CPAP or laptop out for the X-ray thingy — and I got to my gate with something like 3 hours before boarding. Whoops. Luckily, unlike backwaters like Detroit, National Airport has free WiFi, so I was able to get both electricity and Internet while I waited.
Boarding eventually happened. Got aboard, got my drink (kalimotxo), got to watch everyone else stroll past. On-time departure (with 3A and 3B being the last to arrive), taxi to the runway, wait for our position, and when on-deck with only one ahead of us an altercation breaks out between 2B (whose cellphone made the "marimba" sound as if receiving a phone call or text or having an alarm go off) and 3A. 2B claims she never heard the "turn off your electronics" announcement on push-back; 3B jumps in and starts shouting. Flight attendant asks if there's a problem, and 3A and 3B both say Yes and start berating 2B. Tensions diffused: 3B can continue to use his iPad during takeoff because his is in airplane (no cell) mode, and 2B must put her phone in airplane mode. It got resolved before we took off, but yeesh. (I was in 1A, and 1B was the father of 2B and husband of 2A. I was staying out of it. I did note to the flight attendant later one and away from the other passengers that I could provide the facts of the situation and endeavored not to speculate. He noted that 3A had a friend in the Pentagon plane on 9/11 so she's self-admittedly sensitive and he suspects that she assumed that, based on skin color and clothing and yesterday's disasters in Paris, that the family might've been Islamic terrorists. (Oh puhleeze.) We'll see if they need me to say anything.)
Other than that the flight was uneventful. Had some pretzels to buffer the kalimotxo, landed and arrived early, got my bag from the conveyor (signage said carousel 6, phone app said carousel 6, but the delivery status showed it as carousel 10, so... it was at 10), headed up over down and out to get the shuttle to the other terminal's parking deck, headed up over down and across to get to my car, loaded up, and headed out of the airport. Traffic was moving at or above posted speeds until Michigan Avenue, but I was able to leapfrog past the slowpokes. Stopped by the diner for dinner before going home, unpacking, writing more of this trip report, and going to bed.