Sunday, 25 August 2013

If it ain't broke, don't forget to upgrade it

Stable systems are great. Day to day, they require little management and they keep doing their job. It makes sense to leave well enough alone and work on other stuff, right? It's a common approach and it kind of makes sense. Except, if you ever need to update software on that system. And that's when the pain begins.

Here's a few examples of straightforward tasks that have ended up being unreasonably time consuming, infeasible due to the number changes required on production boxes, or near impossible due to the level risk introduced when changing software the was depended upon by other components.

  • Installing Dell OpenManage utilities 
  • Updating Subversion
  • Updating Net-SNMP
  • Installing Puppet 
  • Administering EMC CX3 NAS
I've found that the requirement to update software often comes when improving management of the server. The software packages that provide the key services like databases, application servers or web servers are not upgraded that often and the update process is fairly straightforward. Although updating between major versions tend to require a whole new set of dependencies that may be more challenging to satisfy.

Now even though you're improving the system to make it more manageable or reliable it doesn't mean that there won't be objections to upgrades from customers or within your own team. Some typical objections:
  • It's been working well enough up until now, there's no need to change it.
  • An application developed by in house development team will have to be re-tested or changed. They don't have time for that.
  • We've got fires to put out, that's not even burning!
None of these objections are valid reason not to upgrade something, but they are things you take into account when managing your approach to upgrades. IT Operations should be adept at keeping things running optimally and being proactive at doing so. Waiting for something to break is not a great strategy. The time spent investigating incidents that are actually caused by known bugs will easily outweigh the time taken to make small upgrades. If another application needs to be re-tested or changed then you'll have to work with the development team to do so. This may be difficult because they won't have the same priorities that Operations have but as long as the problem with not upgrading can be expressed in ways that affect the running of that application, you should be able to get some traction.

It won't always be easy to make upgrades, and you will have to compromise sometimes. But it's important to have a strategy of regular updates in order to maintain a stable environment.

No comments:

Post a Comment