Speakers:

• Ted Hopton, Wiki Community Manager, United Business Media
• Donna Cuomo, Chief Information Architect, the MITRE Corporation

Background:

[These are my quick notes, complete with  (what I hope is no more than) the occasional typo and grammatical error.  Please excuse those. Thanks!

From time to time, I'll insert my own editorial comments - exercising the prerogatives of the blogger.  I'll show those in brackets. ]

Notes:

Ted Hopton:

• His company organized the Enterprise 2.0 conference
• They use Jive software for their enterprise 2.0 platform
• Focus first on participation
  • Use the analytics module of your enterprise 2.0 tool to see who is visiting the site, where the activity is taking place, who is creating and viewing content, etc.
  • Analyze active members by level of activity
• Problem: Make sure the metrics tie back to your project goals
• Use qualitative measures to improve your understanding
  • Use a survey – ask how and how often people use the community
  • List possible positive outcomes and ask users which of these outcomes they have experienced
  • Ask why they don’t use the tool more
    • Use blunt, negative statements
    • Encourage them to tell you exactly how they feel
  • Use this information to benchmark (and draw out the venom – otherwise it festers)
  • Net Promoter Score – give users a scale of 1-10 and ask them how likely they are to promote your work.  Scores above 6 indicate that they will promote rather than detract your promoter. Subtract your scores below 6  from your scores of 6 and above. This yields your “Net Promoter Score.”  Obviously, the higher the better.
  • Track your success stories and share them
• Lessons Learned
  • While it’s good to have consistent metrics, be aware that metrics evolve and your methods should evolve too
  • Beware of benchmarks (e.g., the 90-9-1 standard of participation). Make sure the benchmark you are using really applies to E2.0 projects.

Donna Cuomo:

• The Mitre Company runs four differently federally funded programs (including for the Dept. of Homeland Security and the Dept. of Defense)
• Use Case 1:  Improve MITRE’s Research Program Selection Process
  • They used Spiggot to be their “innovation management tool”
  • They wanted to codify their research competition process
  • They wanted to stop people further down the food chain from weeding out ideas too early
  • They wanted to encourage broader participation (from a review perspective)
  • They created an “Idea Market”based on a SharePoint wiki
  • Their first-year metrics indicated broad participation
  • They were able to create widespread transparency
  • They used surveys to compare the new tools (and user satisfaction) against the old tools/methodologies
• Use Case 2: Social Bookmarking
  • Hypothesized that social bookmarking would inmprove resource sharing, leveraging the research of others across teams and the corporation
  • They also thought the tagging would help identify experts within the organization
  • They used a tool similar to Delicious
  • Bookmarks helped create a lightweight newsletter (this was an unexpected benefit)
  • You don’t need many participants in order to provide real value to the entire organization
• Use Case 3: Babson SNA Study
  • They identified super users of their internal social networks and social media (brokers) and then interviewed their colleagues
  • They discovered that these super users tended to be innovative and provide huge value to their networks
  • Frequency of interactions was not as important as the number of unique connections each broker had (indicative of their ability to have an impact on a wider range of people).

Exercise:

• What are the most important things you are NOW measuring?
  • Number of communities
  • Number of community members
  • Percentage of contributors versus consumers
  • Usage across geographies, business units, etc.
  • Number of visits
  • Dwell time (how long is each visit)
  • Number of concurrent users at any one time
  • Number of people editing (indicates collaboration)
  • Number (and identity of ) lurkers
  • Measuring conversion of lurkers to active participants
  • Participation in community activities (who is sharing, who is editing, who is tagging, etc.)
  • Utilization of the various social tools
  • Success stories
• What are the most important things you should be measuring?
  • Abandonment rate – when do visits/activity drop off
  • Tracking against business goals
  • Net Promoter Score
  • Day/time of highest activity
  • first and last page viewed
  • business improvement metrics
    • = correlation of usage to operating metrics
    • = correlation of usage to improved business process
  • Measuring cross-fertilization (the extent to which people choose to go outside their community for information)
  • Number of new ideas/ rate of innovation
  • What’s the reduction in other forms of overhead activities (e.g., now that the subject matter expert is posting answers on a social platform, what is the resulting decline in repetitive e-mail requests?)
  • Percentage of profile completion
  • Rating content
  • Ability to determine a dollar value to participation
  • Where was the content reused, how was it reused, and what were the results of the reuse (e.g., cost savings, process improvement, etc.)

• Presentations:  www.e2conf.com/boston/2010/presentations/workshop
  • User name: Workshop
  • Password: Boston
• Presentations also on Slideshare: http://slideshare.net/20adoption

For knowledge management professionals, it can be a tough one to answer honestly. Why? Because many of us struggle with proving the value of knowledge management efforts. We know that we’ve helped individuals, but we are often hard pressed to explain how much we have in fact helped.  For example, you might truly believe that the enterprise search engine you’ve painstakingly implemented will save lawyers time and effort, ultimately saving clients money.  But do you have any metrics to prove it?  Unlikely.  So how do you know that your search engine project creates value?

One approach is to sit next to your colleagues with a stop watch measuring the time spent on searches before and after your enterprise search engine is implemented. Then you should have the data necessary to prove value numerically.  But how do you measure user satisfaction? You could ask users to complete a survey.  With tools like zoomerang or surveymonkey, it’s almost too easy to do this.  However, the real challenge lies in how the survey is constructed and interpreted.  An additional problem is that it can be hard to coax busy lawyers to complete your survey.

If you’re looking for ways to show how much value you’ve created, consider the example of Morrison & Foerster.  On a page whimsically entitled “Geek Power,” the firm makes the following claims about their knowledge management program:

They back this up with an exercise they undertook in 2006 to prove value.  Specifically, they retained Bruce MacEwen of Adam Smith Esq to talk to MoFo attorneys about their experiences before and after AnswerBase.  According to Bruce MacEwen:

I was retained by Morrison & Foerster to lead an analysis and review of AnswerBase vis-a-vis its predecessor Knowledge Management system during last summer and fall, and reached the resounding conclusion that AnswerBase was strongly superior to the firm’s legacy systems, by providing highly relevant documents and discovering genuine subject-matter experts within the firm with impressive accuracy.   By interviewing a broad cross-section of lawyers at the firm’s New York offices, I was able to determine that the design and functionality of AnswerBase essentially replicate, as I put it in my report, “the way lawyers think” rather than reflecting technical considerations or limitations.

Admittedly, hiring someone of Bruce MacEwen’s caliber will be hard to justify for every small project on your to do list.  However, I’ve recounted this story to remind you (and me) that sometimes it makes a lot of sense to bring in an impartial third party to help you and your colleagues see what is right in front of you.  And if in the process you manage to demonstrate that your KM efforts have created value, that’s all the better.

[Photo Credit: Dave Elmore]

1. On the Fahrenheit scale, freezing occurs at 32 degrees.
2. Today’s temperature is only 8 measly degrees above freezing.
3. Therefore, it is practically freezing and I should dress warmly to avoid practically freezing myself.

So I put on my winter coat and walked out the door.  Moments later, it was clear that I had misunderstood the data.  I saw people walking in light jackets and, in a couple of slightly crazy cases, in shirtsleeves.  Where did I go wrong?  While 40 is fairly close to freezing, in New York City in January it can feel balmy — especially if it comes on the heels of a cold snap.  If you doubt this, think about how you  dress in the autumn in New York City as the temperature is plummeting towards winter.  Warmly, right?  To be specific, would you wear a sweater if it were 50 degrees Fahrenheit in September?  Yes, most probably.  Now think about a 50 degree day in March.  In New York City, you’re likely to see folks wearing shorts and T-shirts.

What is critical to this analysis is knowing that we’re talking about New York City rather than Miami AND we’re talking about specific times of year.  Both elements of context have a huge impact on how we interpret the bald data of temperature.  It is no different when thinking about the metrics you’ve so carefully collected (I hope!) to help understand the efficacy of your Enterprise 2.0 or knowledge management project.   Knowing that activity levels have risen may be interesting, but knowing that happened against a backdrop of falling business levels makes for interesting analysis.  What’s going on?  Why?  The metrics by themselves don’t tell the complete story.  They need faithful, honest interpreters who can place them in their correct context and draw appropriate conclusions.  We need to be those faithful, honest interpreters.

By the way, it’s snowing heavily in New York City as I write. I’ll be dressing warmly.

[Photo Credit: Qiao-Da-Ye]

So how do we restore context, nuance and meaning? And, more importantly, how do we help initiate needed change within our organizations? According to the folks at Anecdote, the answer lies in telling good stories and then listening properly to those stories:

Surveys and metrics can uncover trouble in an organisation, but they usually don’t help you identify the reasons for dysfunctions, let alone generate the resolve to springboard people into action. Instead, learn to use stories as listening posts and tap into the emotion to spark action. From time immemorial, stories have contained collective lessons in condensed form. When gathered and examined, stories that are told in your organisation reveal important themes and patterns that in turn indicate effective solutions.

To be clear, I’m not trying to trash quantitative analysis. However, I do believe there are some things that can be communicated best by numbers and other things that can be communicated accurately only through narrative. Be very sure that when you make your choices about what to measure, how to measure and how to report the results, you choose the right tools and methods. If you cut corners here you will compromise your project and, possibly, your credibility. Why risk it?

[Thanks to Stan Garfield for pointing out the Anecdote post.]

Let’s take the example of the typical law firm. How does it define success? Profits per partner? Long-term client relationships? Employee attrition? Recruiting rates? The reality is that there are many bases on which to judge success. So, what do firms typically choose to track? Billable hours. When you track hours, you send the unmistakable signal that you are interested in time — lots of time. After all, time spent equals money. However, where in that equation is the notion that time spent well is worth more than money? At the end of the day, you know the cost of the time spent. But, do you know the value to the firm or, more importantly, to the client?

If we defined success as delivering high-value services to clients, what would we track? If we defined success as building value within the firm as an institution, what would we track?

For law firm knowledge management, the issue of metrics is a persistent problem. We’ve chased various ways of trying to prove return on investment, but with little success. What should we track to show how our efforts provide value to clients and to the firm itself? Until we’ve conquered this challenge, we can’t expect to achieve any real measure of permanence within a law firm. And, that’s a problem when the economy is heading south.

The real issue isn’t size of following as much as it is scope of impact. How many of these folks are really paying attention to you? How many do you actually affect? Unless you know this, you don’t have a good understanding of your interaction with Twitter. Admittedly, there are Twitter stars whom everyone likes to follow. And, assuming we follow because of their established reputations, we’re more likely to pay attention to what those Twitter stars say. For the rest of us in the Twitter mob, however, the number of our followers is a poor (and possibly inaccurate) proxy for our impact.

Coming back to law firm knowledge management, take a moment to consider whether your efforts to measure the wrong thing are leading you into unproductive activity. Don’t focus on bulk — focus on impact. For example, counting how many times a particular document is opened via your portal or document management system may be interesting but not helpful. What you really want to know is how many times was it opened and actually used? And, how often was it exactly the thing the user was searching for? In the latter two cases, you learn much more about the quality of your content and the quality of your search engine.

Consider the following: a document was opened 10 times and used each time, but then opened 20 times and discarded because it was not on point. For someone looking at bulk alone, they’d say, the document was opened 30 times, declare victory and go home. However, someone measuring impact would say it was used 10 times not 3o, and then would ask why. When you ask that question you create the possibility of learning and insight. That’s when you know you’re on the path to using metrics intelligently.

[permission to use granted under a creative commons license]