If organizations are serious about ensuring their IT services truly meet user needs, a whole new approach to measuring IT performance is required. Research suggests that, even where service-level agreements (SLAs) are green, it is common for users to be dissatisfied with the service they are receiving.

SLAs are almost always based around technology measurements — availability, response times and so on — and since what users actually feel is rarely taken into consideration, they are often still left seeing red.

Organizations must therefore find a way to understand user experience, measure it with statistical rigor and apply that understanding to improving how IT is deployed and used.

But how do you turn “user experience” from something that’s woolly and hard to define into something you can measure and improve? One approach, known as Net Promoter, measures customer sentiment based on recommendation. In recent years, this has been adopted by many customer-focused organizations as a better measure than customer satisfaction; it can also measure user experience. It seeks to understand — and score — when a service gets so good (or so bad) that people will tell others about it, unprompted.

Although Net Promoter is a useful way to measure user experience, it needs to be complemented with a clear understanding of what contributes to the ratings achieved. To do this, a user experience model of 30 to 50 different elements needs to be created. These elements can then be analyzed to understand exactly what is critical in driving Net Promoter scores, and what really doesn’t matter.

Boxing clever

The next challenge is moving from analysis to action. One very powerful way to visualize the data is to use a Boston Matrix where service teams can plot the level of importance of a particular element against its level of performance. The matrix is divided into four boxes — Focus, Maintenance, Watch and Opportunity — that show clearly where a user experience element stands at any one time.

The Focus box shows the key user experience elements that currently aren’t performing well — these are the areas you should concentrate on improving. The Maintenance box also contains key elements, but ones that are already performing well. These are elements to keep an eye on, but which don’t need pressing attention.

The Opportunity box displays elements currently performing well that are nevertheless slightly less important; this may be a place to make savings and refocus resources elsewhere. The Watch box contains elements performing badly which aren’t currently important (although they may be in the future.)

The reason this method is so powerful is that it allows you to track how the importance of particular user experience elements changes month to month, and how your actions can alter their position in the matrix. You can see elements moving from one box to another, giving you the opportunity to fix problems before they become serious. Indeed, it’s possible to observe complete lifecycles for different user experience elements.

At the moment, this approach is new to the IT services industry, but forward-thinking customers are already demanding it because the service improvements, productivity gains and ease of management it offers are so compelling. And companies who stick to the old-style SLAs may well find it’s not just their users who are seeing red when the SLAs are green — that color may also become a prominent feature of their CIO’s performance review.

• Mark Nicholls is head of customer experience at Fujitsu UK & Ireland.

See also a longer version of this article, on measuring IT user satisfaction in the workplace.