Conference Report: 2015 LISA: SysAdmin in Higher Education Workshop

The System Administration in Higher Education Workshop asked what's different and what's the same for system administration in the higher education realm versus industry, including the challenges faced by practicioners. About half of the attendees worked in central IT at their institution as opposed to a distributed IT shop such as college (within a university) or department level. Most of us were at bigger institutions... or ones that felt bigger.

Improving Support

Our first topic of discussion was practical ideas for improving support. One manager has a team of 5 engineers and 12 students (the outsourcing of a lot of work to students works really well except during finals week). They have about 40 to 50 products in their service portfolio, most of which are commercial off-the-shelf products, and half his staff spend most of their time maintaining them. He wants to move towards a service center model so his engineers can be freed up to work on more interesting problems than provisioning and day-to-day operational tasks. Suggestions included providing self-service access for faculty and students to do certain provisioning tasks themselves, abstracting the work into smaller chunks, generalizing (for example, "students" as opposed to "art students" and "engineering students"), and retiring obsolete or combining common services in the portfolio.

Keeping staff engaged by handing off the routine stuff to the help desk helps. How else do you keep staff engaged? Automate (or "provide consistent service delivery for") the daily-operations tasks so people can work on projects instead. Over time people find their interests and that's okay. Another's organiztion is stable: Employees have been there for 35 years, so there aren't a lot of new people. Some want to be more engaged but the institutionalists don't want people to be engaged because that means the oldtimers would have to change.

Another is like that; someone is unwilling to disengage when workload says he should. Someone wants to be the de facto SPOF. He's holding knowledge and won't document or disengage or relinquish tasks. This needs management buy-in and culture change (no one product owner). There may still be specialization but information needs to be shared (e.g., SMEs are okay).

Consider switching jobs and not ask each other for help, doing the routine tasks, and just using the documentation. Mentoring is another possibility.

How do we measssure success and translate that into the right operational changes? Ticketing systems can provide some metrics. Surveys to faculty ("how're we doing") can be useful, especially if repeated so you can measure change over time. Justify new projects or products by (faculty) demand. What if your goals ("faculty: keep this guy here") conflict with the dean's ("move this FTE to Central")? If you do surveys, who gets to see the results (raw data, analysis, future actions or priorities)? Some people wish they got more negative feedback than they do, even if it's "I like this but..." — we're not perfect.

Setting Priorities

Our next discussion was on priorities. Other than incident handling, how do you prioritize what's important? It's not unique to higher ed but we put a twist on it: There's no obvious budget bottom-line to point to. A lot of institutions of higher learning care about teaching and research; how do you measure that?

In an ideal world priorities would be obvious and management would help with guidance. Our priorities should align with those of the college or university, which is usually about teaching and learning, research, and service, depending on your environment. Those areas are inherently messy and can't be planned the way "build a building" will be (which is messy but in a constrained way).

Can you abstract priorities to "my faculty, students and staff"? Not entirely. You still need to plan for end-of-life and capacity changes. Ask faculty if there'll be other changes (for example, Java to C or if Eclipse will go away). Remember that priorities may be different for the group (maintain stable network) and for yourself (continually learn, teach, and research in your own field). Regardless of that, you need to make sure things keep working. Build things to stay stable 24x7 in a 1-person shop yet move technology ahead.

One team has a goal of stability, so changing hardware or software is declined, and they do trouble tickets for issues and weekly meetings with the researchers for possible future planning. Another team is cleaning up after years of non-management.

In a department of 21 (plus students) on a 4-person infrastructure team, someone went from taking direction from their boss into a feedback loop — providing ideas for improvement, simplifying workflows, provide new ways of contributing (including beyond their own group).

As an ITIL teacher, the business drives the priorities, and it's based on urgency and impact in the operational work. Trouble ticketing systems are a good start for incident handling.

For another, it varies at the university (research, teaching, and patient care), college (research and teaching), department (projects based on survey results as defined by our director), and team (such as infrastructure) levels. Having regular one-on-one meetings is essential.

Individual priorities are yours regardless. On a professional basis, what you're prioritizing needs to align with the rest of your institution. We need to provide clear advice and recommendations to senior management for them to decide; we shouldn't be making decisions at our levels.

You need to be sensitive to the unwrittten rules: What about those with bad histories (faculty person A has more problems than another faculty person, or hr issues behind the scenes)? Can VIPs be flagged in the system? If your manager is not setting your priorities, let them know what you are prioritizing.

We need to set priorities because we don't have enough resources. Kanban is a way to organize and prioritize work and can help with communications (in all directions).

Security

Next we talked about security. Universities aren't really that much like businesses. What are the unique aspects of higher ed? Some can't say "No" (like "no porn" or "no Netflix"), but some rate-limiting may be useful.

Someone thought they had a security problem and hired a CISO. Their only directive is "Security." Issues of privacy are being disregarded in the name of security.

Some of the challenges: Research institutions have short-timers — but IT isn't told when and where they went. How do we ensure accounts are closed when they should be? What about when credentials or machines are compromised (and three-letter agencies come for it)? How do you get fellows with prestigious awards to choose longer passwords without writing both ID and password on a sticky note?

Other challenges include personally-owned devices ("BYOD"), application hosting (where the institution provides containers and infrastructure but the customers build their own inseucre front-ends with SQL injection possibilities), worldwide collaborators (so the institution can't block countries known as threats — which won't work long-term anyhow because the threats move), and senior faculty who don't want to change what they've been doing.

A policy or advisory group that meets regularly can help write the policies and make them sane and applicable with buy-in from the relevant sources. Have the CISO keep the chancellor or executives quiet until they're ready to act. Remember that "declare by edict" often doesn't work in academia; there's no boss-employee relationship here. We don't have the ability to tell faculty how to do things; we can make recommendations but they are responsible for their data.

Some places are trying top-down edicts and IT is having to dance around the push-to-centralize. "Academic freedom" is a red herring: We're not trying to prevent faculty or researchers from doing their work. In reality, a "grant" is a contract between the granting agency and the university and has requirements that may include security. Some grants have specific security requirements (including FISMO). One is "You might have to monitor logs" — but they could use that requirement to justifying it for a splunk license... to montior those logs as well as everything else.

What's at risk? Intellectual property of research and reputation of the researcher and institution. It seems like the lawyers and executives are finally catching up to what we understood 15 years ago about how dangerous technology can be. They seem to be much more interested in a vendor-provided solution or service, shifting the responsibility and blame (and liability) to someone else. Is that good or bad?

Some places use a combination of vendor and internal tools, VLAN segmentation, SQL injection review. Even with all those you have to use them correctly. Remember though that regardless of whether it's internal or vendor, it's still your institution's name above the fold of the New York Times.

There are some times of liability you can't get away from. Some have to store SSN, PCI, or PPI somehow. Some business processes need to be fixed (for example, an SSN is needed in one place but stored in multiple places).

Budget

Our penultimate discussion was about budget. Some places do have an adequate budget, some do for equipment but not people, and some just don't.

One place is moving towards cloud-based services like AWS. They also had their Symantec contract expire and move to Sophos. They could move from hosting their own to AWS which is PCI-compliant. It shifted the expense from capital expenditure (CapEx) to operational (OpEx) and freed up FTE resources internally. CapEx is almost always easier to justify than OpEx.

Do an honest analysis: Are you the service provider or a service broker? Can you manage external services? Remember you may be a customer not a provider. Recommending others' stuff instead of your own may be hard. Remember that doing customer support is hard when you're at the mercy of the third party provider.

Monitoring and alerting (your internal people) is not necessarily possible when you're not the provider. How do you monitor cloud-based services? (You don't.) You may or may not have lowered your users' service level.

Handing off the "fun" stuff to the cloud and being a service broker can lead to disengagement of the IT staff. It might save time and money (at least CapEx), but it loses the staff engagement. Handing off some stuff to the cloud lets you focus on the stuff that you're keeping.

Campus Participation

Our last topic was participation on campus. We generally advised everyone to get involved on campus-level committees, both technical and non-. Faculty and staff boundaries may be problematic but making the connections is very valuable. There's also off-campus activities like ACM, EduCause, IEEE, LISA, LOPSA, USENIX, and so on. Find those formal or informal networking groups that work for you and participate.

Back to my conference reports page

Back to my professional organizations page

Back to my work page

Back to my home page