By Jedd Fowers, Senior Benchmarking Consultant, EDS and By Scott Feuless, Senior Consultant, Compass

Benchmark analyses of outsourced IT services are a commonly accepted method to assess the value delivered by a provider.  Typically mandated by contractual clauses in the agreement, and executed periodically throughout the contract term, benchmark reviews measure and evaluate whether services are being delivered for a fair market price and at the appropriate level of quality. Conducted in the context of prevailing industry standards, benchmarks can help manage the complexities and challenges of outsourcing contracts by providing a much-needed reality check in a rapidly changing environment. 

One longstanding point of contention in the benchmarking process is how best to define the measurement standard for the analysis. Specifically, should the contract in question be compared against a blended average of similar agreements? Or should a high-performing group of peer organizations constitute the point of reference in assessing the value of services delivered?

Some recent reports argue that, for a variety of reasons, average levels of performance are the most appropriate measure to apply to a benchmarking exercise. One recent article in Global Services states that it's "mathematically impossible" for all outsourced customers to receive above average rates.  But should customer organizations really be expected to invest significant resources in an outsourcing relationship just to achieve "average" performance?  And how many providers would cite as their value proposition the ability to deliver only average pricing and service quality over the long term? 

A more specific problem with using averages as a comparative standard is that the deals that are benchmarked by third parties are often problematic to begin with. As such, using the average of a pool of benchmarked contracts is like saying that the average behavior of all the kids sent to the principal's
 office should be used to define the standard for behavior of students at an elementary school. 
Some have also argued that benchmarkers use only a handful of peer contracts (four to six in most cases) to define a comparative reference group, and that such a small sample is a statistically inaccurate way to define top performers – specifically, "top quartile" or "top decile" peers.  

That's a bit misleading.  When "top quartile" is a contractual requirement, a benchmark applies "all" available and appropriate data for the particular service or operation being analyzed to define the category of top performance.  So, depending on the specific operation, this might be, say, between 20 and 100 data points.  All of those points are used to determine what qualifies as "top quartile" (or whatever criterion is specified in the contract).  Once that is determined, four to six points are selected within the top performing range so that statistically meaningful average values can be extracted.

An effective benchmarking analysis recognizes that it is unrealistic and counterproductive to expect every benchmarked function to fall within the "top quartile" or "top decile" category. In a real-world environment, high performance in one area is often offset by higher costs in another.  For this reason, a big picture perspective is essential — a benchmark process that focuses too much on narrowly defined functional areas risks missing the forest for the proverbial trees.

Ultimately, the debate over how to define categories of performance in quartiles, deciles, or other measures is of secondary importance.  If the purpose of the benchmark exercise is to assess the value of existing services, then comparison against above average performers and leading practitioners — however defined — provides a clearer picture of performance, and, more importantly, of how to establish goals and priorities for improvement. 

The fact is, unit costs for technology products and services are continually changing. Top performers use benchmark results to adjust deals accordingly over time, while below average performers fail to keep pace with the rapid change of industry standards.

Elements of an Effective Benchmark

It's true that outsourcers, customers, and third-party benchmarkers are evolving in their approach to managing the measurement and evaluation process, with the overriding objective of using the initiative to enhance strategic value. Compass has defined the following criteria as essential to an effective benchmark exercise. 

Database or reference group:  Establishing an apples-to-apples context for the comparison is critical.  Specifically, the environments the customer is compared against should be reasonably comparable in terms of size, scope, complexity, and other factors.

Adjustments: Even when the organizations comprising the comparative reference group are very similar to the customer organization, some normalization of scope, quality, and pricing of services will be required.  These adjustment calculations can be based either on prices extracted from existing outsourcing contracts, or on costs incurred by internally managed organizations, but a database of costs that is granular to the functional level is an absolute requirement for use as a basis for accurate adjustments.

Transparency: Both the provider and customer organization should have access to information regarding the reference group and the nature of any adjustments. Specifically, all parties must have a clear understanding of how pricing targets are defined, what goes into the calculation of those prices, and if and how adjustments are made.  However, as both parties must be willing to accept the benchmarker's customer confidentiality requirements, transparency has limits.

Profit motive: Customers must recognize that providers must make a profit for the relationship to be successful. This means that a provider who can deliver services for significantly less than prevailing market rates should reap the benefits. For example, if $10 per call constitutes a fair market price for help-desk services, then a customer should agree to pay that rate — even if a provider is able to deliver services for $5 per call with competitive quality. This type of stipulation is essential to motivate providers to support benchmarking initiatives and to invest in innovation.

The ultimate objective of a benchmark initiative should be strategic and constructive, and not merely an opportunity to knock 10 percent off prevailing prices. The focus should be on how to deliver best value, maintain competitiveness, and identify new opportunities to innovate. With these objectives in mind, customers should never settle for "average," because "average" can always be improved.    

Scott is a senior consultant with Compass, a global management consulting firm that provides third-party benchmark analyses of outsourcing agreements.