Monthly Archives: January 2012

Change Management Workflows

In a previous post I covered the basic fields you need in a CR (Change Request) and promised to look at optional info that more heavyweight CM processes might use.  In this post I’m going to focus on workflows: approvals, reviews, and validations.  While there are other fields that you can find on a CR, it’s really the workflows that are a critical part of CM.  The basic CR we looked at last time could be used even if only one person is involved in the entire process – but typically multiple people are involved.  One person might request a change, someone else might execute it, someone else might validate that it worked, etc.  If you have workflows like these that involve multiple people, you really want to have those workflows recorded in your ticketing system – so that you can see when one step is completed (and by whom) and the next has begun, and so that you can tell from your ticketing system what stage a ticket is in and who currently owns it.  Please note that I’m not saying you HAVE to incorporate these flows into your CM process – they are appropriate in some places but not in all.  As I said in earlier posts, you should have the right amount of process and paperwork that’s appropriate for your environment – no more, no less.

Here in fantastic ASCII format is a basic CM process, showing how review/approval/validation workflows might be linked:

Create  –> Review –> Approve –> Schedule –> Execute –> Validate

Each of these steps could potentially be performed by a different person or role.  If this seems crazy to you, note that in some cases it may be required for compliance or governance reasons that different people be involved in the various stages of the process.

Create

A request for a change often comes from a non-technical resource.  It might be a customer, a project manager, a product manager, etc.  The request will not have any execution details in it, it will just say what is being requested and why.  This will then be picked up by a technical resource that will create the actual procedure to be executed, including rollback and validation steps – hopefully with an eye towards automating the change in the future.  🙂

Reviews

A technical review step is generally performed by a peer of the person who created the execution procedure and/or will be executing it.  The purpose of the review is to ensure that the procedure is complete and correct and doesn’t require any special or unique knowledge on the part of the executor.

There may also be a business review step, where a peer of the original requestor evaluates the “why” behind the change request to ensure that it is appropriate to execute.

The output of a review step may involve changes to the procedure itself or sending the CR back to the original author for modification, in which case it will come back through the review step after those modifications are made.

Approval

The approval step is a gate prior to execution.  It is often combined with the review step above, but may be separated – usually this is done when technical reviews are done by SMEs in the relevant systems or technologies, but those SMEs may not have larger/wider knowledge about all the systems that could be impacted.  In such cases there will typically be a separate approval (which may be done by a committee like a CAB) signifying that the change has been reviewed in a larger context and is deemed safe and appropriate for the system as a whole.

Schedule

Changes may be executed at the discretion of the executor, or there may be more central scheduling for some or all changes  One typical case is when there is a defined maintenance window where changes that impact customer-facing services are performed.  These windows are often planned and scheduled by a specific person or role to ensure that the limited time of the window is used most efficiently while guarding against change collisions.  (Executing multiple changes simultaneously can lead to great difficulties with diagnosis if something goes wrong – since you’re not sure which of the changes is the cause of the problem).  Even outside of maintenance windows, changes may be centrally scheduled for other reasons.

Execution

Someone actually performs the change.  Nothing special to see here.

Validation

After the change procedure has been performed, there may be a handover to another individual (typically a peer of the executor) to perform technical validation of the change.  A second pair of eyes can sometimes see unintended effects that otherwise would have been missed, and also eliminates any temptation to “paper over” what appears to be an inconsequential deviation from the CR (“it was just a one character typo!”).  This validation should be tightly coupled in time to the change execution as until this validation is complete the system is potentially in a bad state.

Separately, there may be a business validation step to ensure that the intended effect has occurred.  This could potentially be done at some remove in time from the execution and technical validation.

Record your workflows!

Arguably the most valuable part of using a ticketing system to track CRs is how easy it makes it to handle workflows.  You can see at a glance where something is in the process and who owns it.  You can record the results of each step in a workflow, including who performed it and timestamps.  And you can use the ticketing system to actively manage the workflow, including auto-assignment and sending reminders or SLA warnings if appropriate.


Getting Ubuntu to send through Gmail for Nagios

Recently I set up a Nagios installation running in a VM on my laptop (Virtualbox on a MacBook Pro).  This was a quick and easy way for me to get some monitoring going without needing a real server somewhere.  (I know, I know – “Use the cloud dumbass!”  It’s complicated).

Anyway, the trickiest part was getting the notifications to actually go to my gmail.com account. Gmail doesn’t accept regular old SMTP connections.  You need to use authentication over TLS (or SSL).  I didn’t know how to set this up, but I found this great page that explains most of it:

Mark Sanborn’s guide to getting postfix to send through Gmail

This guide was written several years ago and while it almost worked for Ubuntu 10.10, two changes were required.

The first change was to use the new Equifax CA cert because the one that Gmail is currently using isn’t part of the postfix config in 10.10, although the right cert is already on the system. Fixing this is very straightforward:

cat /etc/ssl/certs/Equifax_Secure_CA.pem >> /etc/postfix/cacert.pem

The second change is to tell the mail server to actually use the transports file that the guide has you set up. This must have already been there in 2007 but wasn’t there for me. This was a one-line change to /etc/postfix/main.cf:


root@gunther-VirtualBox:/etc/postfix# rcsdiff main.cf
===================================================================
RCS file: RCS/main.cf,v
retrieving revision 1.1
diff -r1.1 main.cf
19a20,22
> # use a transport file to send stuff to google for gmail.com
> transport_maps = hash:/etc/postfix/transport
>

(Yeah I still use RCS when I’m modifying config files. Try it!
apt-get install rcs ; mkdir RCS ; ci -l file
You’ll love it).

By following the instructions at the link and adding these two steps, my VM was sending email to my gmail.com account and everybody was happy.