I made a mistake the other day. As I was leaving for work, I checked the weather report to see how warmly I needed to dress. The forecast said 40 Fahrenheit. So, my brain went through the following fairly logical steps:
- On the Fahrenheit scale, freezing occurs at 32 degrees.
- Today’s temperature is only 8 measly degrees above freezing.
- Therefore, it is practically freezing and I should dress warmly to avoid practically freezing myself.
So I put on my winter coat and walked out the door. Moments later, it was clear that I had misunderstood the data. I saw people walking in light jackets and, in a couple of slightly crazy cases, in shirtsleeves. Where did I go wrong? While 40 is fairly close to freezing, in New York City in January it can feel balmy — especially if it comes on the heels of a cold snap. If you doubt this, think about how you dress in the autumn in New York City as the temperature is plummeting towards winter. Warmly, right? To be specific, would you wear a sweater if it were 50 degrees Fahrenheit in September? Yes, most probably. Now think about a 50 degree day in March. In New York City, you’re likely to see folks wearing shorts and T-shirts.
What is critical to this analysis is knowing that we’re talking about New York City rather than Miami AND we’re talking about specific times of year. Both elements of context have a huge impact on how we interpret the bald data of temperature. It is no different when thinking about the metrics you’ve so carefully collected (I hope!) to help understand the efficacy of your Enterprise 2.0 or knowledge management project. Knowing that activity levels have risen may be interesting, but knowing that happened against a backdrop of falling business levels makes for interesting analysis. What’s going on? Why? The metrics by themselves don’t tell the complete story. They need faithful, honest interpreters who can place them in their correct context and draw appropriate conclusions. We need to be those faithful, honest interpreters.
By the way, it’s snowing heavily in New York City as I write. I’ll be dressing warmly.
[Photo Credit: Qiao-Da-Ye]
Above and Beyond KM 
True, metrics need context. But let me take your analysis up a level. Imagine if that weather data was socialized. For example, say those other people who were about to head out the door commented about their choice of light jackets and shirtsleeves. It might have given you pause to think of that data in another way, with a new perspective.Now translate that situation to the enterprise. For example, a proposal is in the works for a major client. The proposal is about to be finalized, when someone looks at it and notices there’s a missing piece of crucial data — one that the author had not thought to include. If the proposal is stored in a “social knowledge management” system (as opposed to a traditional CMS), the astute employee can leave a comment about the missing data. (Here’s where the “social” comes in.) Other people see this comment, support it, comment back, tag it, and the proposal is now stronger than the original.Just as the metrics by themselves don’t tell the complete story, neither does most enterprise content. Through the wisdom of the community (or as you state, “faithful, honest interpreters”), socialized content provides context where the “good stuff happens.” Mike Cassettarihttp://blog.inmagic.com
Mike -Thanks very much for pushing the analysis further. You're right that thereare loads of benefits to be gained from socialized content. The key isputting the content in a place where it can be read and reacted to. AND,you need a culture that encourages people to interact with content this way.- Mary
“Accurate” metrics aren't always better.Temperature is a substitute metric for what really matters in this case, “How warmly should I dress.” Providing context is a start, but it really requires considerable context to make this metric meaningful. How sunny is it? How windy is it? What will the weather likely be in two hours? And of course, as you note, is it Miami or NYC? Assemble all that context, and you still have a complex system that doesn't directly address the question you meant to ask.Substitute metrics are as evil as bad, incomplete, or mismeasured metrics. They pretend to information they don't actually convey.As H.L. Mencken said, For every complex problem there is an answer that is clear, simple, and wrong. I love metrics, but they matter only when at least as much work is done on assuring that they directly measure what you need to change as is done on taking the measurements themselves. — Steven B. Levy Author, “Legal Project Management: Control Costs, Meet Schedules, Manage Risks, and Maintain Sanity”
Steven -Thanks very much for digging deeper into the quality of metrics. Many gliblytalk about the need for metrics, but few speak as frequently about theinherent problems with metrics. Metrics, like all numbers, areapproximations. Our first job is to ensure that the metrics we choose tocollect are close enough approximations that they answer the questions weneed to answer. You're absolutely right that poor approximations, orsubstitute metrics as you call them, can cause more problems than theysolve. A brilliant interpretation of the wrong data doesn't help anyone.- Mary
Steven -Thanks very much for digging deeper into the quality of metrics. Many gliblytalk about the need for metrics, but few speak as frequently about theinherent problems with metrics. Metrics, like all numbers, areapproximations. Our first job is to ensure that the metrics we choose tocollect are close enough approximations that they answer the questions weneed to answer. You're absolutely right that poor approximations, orsubstitute metrics as you call them, can cause more problems than theysolve. A brilliant interpretation of the wrong data doesn't help anyone.- Mary