Above and Beyond KM

I made a mistake the other day.  As I was leaving for work, I checked the weather report to see how warmly I needed to dress.  The forecast said 40 Fahrenheit.  So, my brain went through the following fairly logical steps:

1. On the Fahrenheit scale, freezing occurs at 32 degrees.
2. Today’s temperature is only 8 measly degrees above freezing.
3. Therefore, it is practically freezing and I should dress warmly to avoid practically freezing myself.

So I put on my winter coat and walked out the door.  Moments later, it was clear that I had misunderstood the data.  I saw people walking in light jackets and, in a couple of slightly crazy cases, in shirtsleeves.  Where did I go wrong?  While 40 is fairly close to freezing, in New York City in January it can feel balmy — especially if it comes on the heels of a cold snap.  If you doubt this, think about how you  dress in the autumn in New York City as the temperature is plummeting towards winter.  Warmly, right?  To be specific, would you wear a sweater if it were 50 degrees Fahrenheit in September?  Yes, most probably.  Now think about a 50 degree day in March.  In New York City, you’re likely to see folks wearing shorts and T-shirts.

What is critical to this analysis is knowing that we’re talking about New York City rather than Miami AND we’re talking about specific times of year.  Both elements of context have a huge impact on how we interpret the bald data of temperature.  It is no different when thinking about the metrics you’ve so carefully collected (I hope!) to help understand the efficacy of your Enterprise 2.0 or knowledge management project.   Knowing that activity levels have risen may be interesting, but knowing that happened against a backdrop of falling business levels makes for interesting analysis.  What’s going on?  Why?  The metrics by themselves don’t tell the complete story.  They need faithful, honest interpreters who can place them in their correct context and draw appropriate conclusions.  We need to be those faithful, honest interpreters.

By the way, it’s snowing heavily in New York City as I write. I’ll be dressing warmly.

[Photo Credit: Qiao-Da-Ye]

Recently I had the interesting experience of reading survey results relating to a subject I actually knew something about. At first blush, the numbers were quite impressive. And then I read a little more closely and discovered that the presentation gave the impression of results that were better than warranted by reality. Since just the “bare numbers” had been reported, important context and nuance were lost. As a result, the story the numbers told was a little misleading.

So how do we restore context, nuance and meaning? And, more importantly, how do we help initiate needed change within our organizations? According to the folks at Anecdote, the answer lies in telling good stories and then listening properly to those stories:

Surveys and metrics can uncover trouble in an organisation, but they usually don’t help you identify the reasons for dysfunctions, let alone generate the resolve to springboard people into action. Instead, learn to use stories as listening posts and tap into the emotion to spark action. From time immemorial, stories have contained collective lessons in condensed form. When gathered and examined, stories that are told in your organisation reveal important themes and patterns that in turn indicate effective solutions.

To be clear, I’m not trying to trash quantitative analysis. However, I do believe there are some things that can be communicated best by numbers and other things that can be communicated accurately only through narrative. Be very sure that when you make your choices about what to measure, how to measure and how to report the results, you choose the right tools and methods. If you cut corners here you will compromise your project and, possibly, your credibility. Why risk it?

[Thanks to Stan Garfield for pointing out the Anecdote post.]

You get what you measure. This isn’t news — first you decide what you want to achieve and then you design your metrics to let you know when you’ve arrived. That’s good practice and it’s the message of my earlier post, The Metrics Mess. Simple stuff, right? Wrong. You’d be amazed how often folks misunderstand where true success lies and, therefore, collect metrics that drive them in the wrong direction.

Let’s take the example of the typical law firm. How does it define success? Profits per partner? Long-term client relationships? Employee attrition? Recruiting rates? The reality is that there are many bases on which to judge success. So, what do firms typically choose to track? Billable hours. When you track hours, you send the unmistakable signal that you are interested in time — lots of time. After all, time spent equals money. However, where in that equation is the notion that time spent well is worth more than money? At the end of the day, you know the cost of the time spent. But, do you know the value to the firm or, more importantly, to the client?

If we defined success as delivering high-value services to clients, what would we track? If we defined success as building value within the firm as an institution, what would we track?

For law firm knowledge management, the issue of metrics is a persistent problem. We’ve chased various ways of trying to prove return on investment, but with little success. What should we track to show how our efforts provide value to clients and to the firm itself? Until we’ve conquered this challenge, we can’t expect to achieve any real measure of permanence within a law firm. And, that’s a problem when the economy is heading south.

I recently saw the perfect illustration of how we can get ourselves completely tangled up in unproductive activity by measuring the wrong thing. In this case, it was someone on Twitter who thought they had hit the jackpot because they had hundreds of followers. Further, this person was offering advice on how to increase the number of followers his readers had. This struck me as misguided at best. To be honest, there are folks I follow whom I’m sure don’t realize I exist. Equally, there are folks who follow me, but I’m largely oblivious to them because our paths don’t cross very often. So the numbers alone don’t tell the whole story and may, in fact, tell a misleading story.

The real issue isn’t size of following as much as it is scope of impact. How many of these folks are really paying attention to you? How many do you actually affect? Unless you know this, you don’t have a good understanding of your interaction with Twitter. Admittedly, there are Twitter stars whom everyone likes to follow. And, assuming we follow because of their established reputations, we’re more likely to pay attention to what those Twitter stars say. For the rest of us in the Twitter mob, however, the number of our followers is a poor (and possibly inaccurate) proxy for our impact.

Coming back to law firm knowledge management, take a moment to consider whether your efforts to measure the wrong thing are leading you into unproductive activity. Don’t focus on bulk — focus on impact. For example, counting how many times a particular document is opened via your portal or document management system may be interesting but not helpful. What you really want to know is how many times was it opened and actually used? And, how often was it exactly the thing the user was searching for? In the latter two cases, you learn much more about the quality of your content and the quality of your search engine.

Consider the following: a document was opened 10 times and used each time, but then opened 20 times and discarded because it was not on point. For someone looking at bulk alone, they’d say, the document was opened 30 times, declare victory and go home. However, someone measuring impact would say it was used 10 times not 3o, and then would ask why. When you ask that question you create the possibility of learning and insight. That’s when you know you’re on the path to using metrics intelligently.

[permission to use granted under a creative commons license]