So, You Wanna Be A Sysadmin?

September 13, 2011

This blog post is *ancient*, and preserved only for historical record.

So you wanna be a good sysadmin? I don’t blame you. It’s fun, and it’s lucrative. Especially if you do it right.

The difference between a good admin and a bad one are many and varied.

Most importantly, it boils down to a level of devotion to the company. And a knowledge that is expansive.

Not that it has to be all-encompassing. One of the important things to know is the limit of your knowledge. Know your comfort zones, and know your limits.

There is nothing wrong with not having an instant answer, but there is something with not being able to find an answer. Powers of inference and deduction are key to your success, and you should exercise them frequently.

I’d much rather work with someone who knows what they do not know, than someone who will blag it.

You’ve also got to recognise your weaknesses, and work hard to ensure that they don’t impact your work.

Among sysadmins, the biggest weakness I see is ego.

Ego, especially in an enterprise or commercial environment is a killer. Egotistical sysadmins will act as if the system is their Property. Often failing to document important infrastructure, or trying to make sure that “the company can never fire them” as they’ve engineered themselves into a critical position.

This is a massive, yet all too common anti-pattern among admins.

As I mention earlier on, a good admin needs to be devoted to the success of the company. With or without them.

One of the biggest problems with Sysadmins with an ego the size of a planet, is the inability to accept personal responsibility. This is a real double-edged sword. There’s some things which, at some point in time, will be, or will have been your fault. It’s up to you to do two things. One, you’ve gotta take the rap for it. Admit you were wrong, and apologise. Secondly, and more importantly, it’s down to you to debrief the rest of the company about what happened, why it happened, and what you and your team will do to ensure that it doesn’t happen again. You’re allowed to make some mistakes, everyone does, but it’s a fool who doesn’t learn from past mistakes.

You also need to consider what happens if you’re unfortunate enough to get fired/made redundant?

Bad times, but you’ll only make them worse for yourself if you’re petty enough to have installed logic bombs or backdoors.

Let me tell you this. There are some very skilled sysadmins out there. Some better at computer forensics than me. Many with better toolkits and detection algorithms and hardware to recover deleted files.

We will catch you. You will get found out. You will get into trouble, and you will never work in IT again.

You could spend 10 years building systems and making a perfect CV, but if you let your ego get the better of you, you might as well have not bothered.

I work hard to ensure that the changes I make are made for a reason; and they’re well documented. The thing I fear most in any system, is a Single Point Of Failure. Engineers have worked for years to eliminate SPOFs from a number of systems. The important SPOF that must not be forgotten is the human factor.

If you were hit by a bus tomorrow, what would happen? Are there passwords that only you know? Scripts that are only on your Mac?

Citing ‘security’ as a reason not to disclose information is stupid and childish. Sure, don’t put passwords on a public wiki, but do put them in a password management utility, and do share access to that to your team. It’s the same as that old adage, “There’s no I in TEAM”, but without a solid team, your skills are of no use to the company.

Empire building is a common trait among sysadmins. Often in a new job, an admin will seek and destroy existing systems because they’re not perfect in their eyes. Only in their eyes, mind you.

An existing system could have been running non-stop for 20 years on AS/400, but that doesn’t warrant a costly move to x86 blades, based on a pitch from an amply bosomed Booth Babe.

No, again, you need to think in terms of what is best for the company. “The Greater Good”, not flashy new toys.

The flashy new toys thing is massively dangerous. There are companies out there who don’t shop around, who aren’t interested in actually researching what they’re being sold, and where mis-selling of infrastructure happens a lot. It’s down to the sysadmin (or engineer) to actually test out stuff, instead of going with whichever has the best looking hardware, or the most promised (but invariably, individually licensed) feature sets. If that means going with slightly less bleeding edge technology, because it’s more tried and tested, then so be it.

There is absolutely nothing wrong with being a lazy sysadmin. Being irresponsible, on the other hand, is a massively concerning trait, and should be addressed as soon as possible.

I frequently find myself doing things to make my life easier. I’m a big fan of automated installations, Puppet, Kickstart/debian-installer, and centralised logging and monitoring.

The reason for this, is that I don’t want to repeat myself. I find building systems very enjoyable, but the minutia are very tiresome. I’d rather do something once, and repeat the action, rather than do something a dozen times on different servers.

Being a lazy sysadmin effectively means that you can concentrate on the important and interesting stuff, rather than spending all day working on something boring and trivial.

Being irresponsible would be ignoring logs, and being obstructive with documentation, and going out of your way to piss people off. This is the “media” view of many sysadmins, not helped by the wicked and evil character Denis Nedry in Jurassic Park. You remember, he took power systems and security offline so that he could steal stuff un-noticed, then initiated logic bombs and so on to cover his tracks and prevent his detection.

Don’t be like that. You might end up like him, getting eaten by a Dilophosaurus.

(You’ll probably just get fired, and never work in IT again)

Profile picture

Written by Tom O'Connor, an AWS Technical Specialist, with background in DevOps and scalability. You should follow them on Twitter