KM healthcheck: How to effectively measure KM performance

Three things you will learn from this Masterclass:

1. Why you need to measure the performance of your firm’s KM systems

2. Which metrics to combine to obtain a balanced measurement of performance

3. How to manage the potential ‘gaming’ of results by fee earners and staff

 

“Not everything that can be counted counts, and not everything that counts can be counted” – Albert Einstein.

How is a managing partner, head of KM or indeed anyone involved in a knowledge project or system to know whether it is value for money?

This is an enormously difficult question. It is probably tempting to institute a simple time-related measurement system with time-based targets, similar to those used to measure fee earners.

Unfortunately, while this may confirm the activity levels of your knowledge professionals, it may not tell you anything about the value of their actions. There is no point in a professional support lawyer working diligently for long hours on a particular method of sharing, say, current awareness, when that method itself is ineffective for fee earners.

Measuring efficacy

Can you measure the efficacy of knowledge systems? I use knowledge systems in a broad sense to include all kinds of knowledge-focused work, not just IT systems. This includes formal and informal teaching, networking events, efforts to encourage collaboration and knowledge-based marketing, such as newsletters and seminars.

Some would say that it is impossible to measure knowledge systems, as knowledge is what happens inside a person’s head as he synthesises new information with existing experiences.

The effects of knowledge systems are also difficult to measure, because they tend to be intangible and intricately entwined with other systems, and affected by serendipitous events.

For example, a fee earner is mulling over a complex issue and is planning to look at the firm’s knowledge bank after making a coffee. At the coffee machine, he bumps into another fee earner, chats to her for a while and discovers that she has recently dealt with the same issue and found a neat solution. Is this a success for the knowledge-sharing culture, a failure of the knowledge bank, neither or both?

Similarly, if you are trying to understand whether it is more cost effective for lawyers to work in X way than in Y way, this may have a more complicated result than a change in chargeable hours. There could be lower chargeable hours per matter and lower billing levels in the short term, but higher quality advice, higher customer satisfaction, lower professional negligence claims, more profitable fixed fees, lower attrition rates and/or more matters undertaken.

That isn’t to say that firms shouldn’t try to get some understanding of the value of their knowledge systems. They need to know the return on investment offered by different knowledge projects and whether to expand a pilot or ditch something unproductive. What this means is that measuring the success or otherwise of knowledge projects is complex and needs a multifaceted approach.

Importantly, before embarking on designing measurement systems for knowledge assets, you must ask yourself why you need to measure them. What is the business benefit of knowing whether a knowledge system (in the wider sense) is working well or not?

In an ideal world, you would have multiple measurement systems for all of your knowledge systems and then act on those results, adjusting and replacing systems continuously. But, many firms cannot afford to work this way. It is therefore helpful to have an inkling of what you might do with the results before you start measuring a system.

For example, you may hear on the grapevine that your competitors are replacing their old off-the-shelf client relationship management (CRM) systems. You wonder if you ought to have a better understanding of how user-friendly your CRM system is and how valuable staff are finding it compared with a newer version, which is supposed to be better adapted to how lawyers work.

This is an excellent idea but, if you simply do not have the budget to replace your CRM system in the near future but do have the budget this year to tackle your training systems, you should leave the measurement of your CRM system until it becomes more of a priority. Concentrate on those aspects that you both need and can afford to improve.

Balancing measurements

There is a saying, often attributed to Peter Drucker, that what gets measured gets done. This means different things to different people, but I take it to mean that:

1. once measurement systems are in place, staff will take a project more seriously; and

2. people consciously or subconsciously try to ‘game’ systems to improve their score, whatever the effect on the system that is being measured, causing unexpected distortions.

For example, if you try to ascertain the value of a webpage containing current awareness that is managed by a knowledge lawyer by measuring the number of views or page-loads of that webpage, you may find that the lawyer adapts the information put on the webpage to make it more interesting to readers.

This is obviously not a problem if a fascinating hook encourages the team to read useful current awareness that makes them better informed about client needs, helps them to produce higher-quality legal advice and/or reduces professional negligence claims.

However, it is a problem if:

• staff read the funny story that has been posted and not the current awareness information;

• the quality of the current awareness that has been posted is poor; or

• the returns from the information for the business don’t justify the amount of time spent collating it.

Gaming is difficult to avoid because people are almost always smarter than systems. The best way to smooth out any effects is to measure results from a number of different angles: qualitative and quantitative; leading and lagging; and from balanced perspectives.

1. Qualitative and quantitative

Quantitative measurements are often simple and cheap to collect. They provide numeric results that are naturally easier to compare and contrast. For these reasons, they are often the first thing that managers tend to measure. Unfortunately, they are particularly susceptible to ‘gaming’ by staff.

Qualitative measurements are more difficult and costly to collect and give less clear results, but are far more difficult to ‘game’. For example, a fee earner whose current awareness work is measured by numbers of page loads may easily subconsciously game the results. But, he would find it far harder to affect the results of: a focus group of fee earners who are asked about the value of the current awareness; and a client survey (with open questions) about the quality of the fee earner’s advice and awareness of current industry events.

The difficulty with solely using qualitative measurements is that, as well as the cost and time to the business of collecting them, those offering their opinions can quickly get tired of giving feedback, so they need to be used judiciously.

 

---

Qualitative and quantitative measurements

Qualitative

Meaning: Relating to/based on the quality or character of something.

Examples: Focus groups, questionnaires with open questions, observations.

Quantitative

Meaning: Relating to/based on the amount or number of something.

Examples: Page loads, usage statistics, numbers of complaints, number of knowledge stories, outage levels, new sales following a seminar.

---

 

2. Leading and lagging

Most businesses tend to measure things that have happened over a set period in the past by financial results, time recorded, profits per equity partner or impact on the share price.

The difficulty with only taking this approach is that, just as with the stock market, past performance isn’t always a good indicator of future performance. Just because your precedent database has served you well enough in the past does not mean that it will suit your business in future or that it is the most effective means of managing precedents.

Leading measurements concentrate on the components that make up the lagging indicator and can be used as early warning systems. Unfortunately, they can be difficult to identify and capture and are often more susceptible to gaming. As with qualitative and quantitative metrics, the best solution is to have a mixture of leading and lagging indicators.

 

---

Leading and lagging measurements

Leading measurements

Meaning: Indicators that drive the performance of lagging measurements and tend to be forward looking.

Examples: Surveys, usage, web stats, numbers of documents available, focus groups.

Lagging measurements

Meaning: Measurements of what has already happened at the end of a set time period.

Examples: Profits per equity partner, chargeable and non-chargeable time recorded.

---

 

3. Balanced perspectives

Robert Kaplan and David Norton first proposed the balanced scorecard in 1992. They understood the misleading signals that traditional financial accounting measures gave and so proposed that businesses ask themselves four basic questions:

1. How do customers see us? (Customer perspective)

2. What must we excel at? (Internal perspective)

3. Can we continue to improve and create value? (Innovation and learning perspective)

4. How do we appear to shareholders? (Financial perspective)

As well as being a useful means of developing a strategy and keeping a whole business on course, these perspectives are useful for those measuring the benefits of different projects.

For example, there is no point in continuing with the pilot of a new knowledge system which provides a great customer experience but doesn’t translate into increased profitability in some way – whether that is through increasing new work from existing customers, widening the customer base or making existing work levels more profitable.

Similarly, there is no point in a knowledge system which satisfies financial requirements, but negatively affects attrition rates and the firm’s ability to attract talented staff, as the firm may soon have no one to fulfil client needs.

Tailoring results

When designing a measurement system, the other key aspect you should bear in mind is the audience for the results. If you are the managing partner who wants to understand a knowledge system’s efficacy for your own benefit, this isn’t a problem, but if you are the head of knowledge who has to report on the efficacy of systems to a number of different audiences, you will probably need different measurements or descriptions of measurements for different audiences.

If you need to report to the finance board to explain the return on investment of a new system, you will use different measurement systems to persuade them. You will explain the results in a different language to that you would use to explain to junior members of staff why changes have to be made and perhaps encourage some creative thinking about solutions to knowledge problems.

Understanding limitations

Measuring the business benefits of knowledge systems is far from straightforward, but there are strategies you can adopt to ameliorate the difficulties, focus your efforts and make the most of your results. The key to successful measurement is to understand its limitations and to take a multifaceted approach.

 

---

10 ways to make your KM performance measurement a success

1. Consider why you need the measurement. What is the business benefit of knowing the result?

2. Consider what can you do with the results. Can you afford to change things if the results are negative?

3. Check what you can afford to spend on measuring.

4. Be clear who will collect the data and how they will do it. Who will analyse the data and who will present it to the audience?

5. Use existing measurements where they already exist and give insight/added value.

6. Have a mix of quantitative and qualitative measurements, as well as leading and lagging measurements, but remember that not all metrics are equal – some results are more important than others.

7. Don’t measure everything or get distracted by what other businesses are measuring. Focus on what is important to your business.

8. Look at primarily long-term trends, but keep an eye on the present.

9. Speak in the language of your audience when presenting findings.

10. Keep measurement methods and results under review and act on the results.

---

 

Hélène Russell advises and trains law firms on knowledge sharing for improved effectiveness, efficiency and profitability (www.theknowledgebusiness.co.uk)

Endnote

1. For further information, see Knowledge Management Handbook, Hélène Russell, Law Society Publishing, 2012