Measuring the Value of Social Tech #KMWorld

KMWlogo_Stacked_Session Description: When it comes to enterprise social, it’s all about the business case. Whether your initiative is just getting started or relatively mature, sustained adoption and ongoing support depend on demonstrating clear business value. This session showcases an approach and practical examples you can use to model your business case and ensure that your investment in enterprise social creates an open, agile, and networked organization. Take away ideas you can apply immediately in any social technology platform.

Speaker: Euan Semple, Director, Euan Semple Ltd

[These are my notes from the KMWorld 2015 Conference. I’m publishing them as soon as possible after the end of a session, so they may contain the occasional typographical or grammatical error. Please excuse those. To the extent I’ve made any editorial comments, I’ve shown those in brackets.]


  • Slide Deck
  • Numbers.  How do you establish the value or ROI of social tools? The numbers may be interesting, but they do not always tell the full story. Unfortunately, numbers  are relatively easy to gather and are, therefore, sometimes all organizations track.
  • Monitoring. The better approach is to actually get into the social activity streams yourself and then monitor them. What’s happening? What patterns emerge? What does this tell you about the impact of the social tools? Euan actually subscribed to every community/activity stream in his enterprise social network (ESN). Over time he became more proficient and efficient at following the conversations. This helped him understand what was really going on in a way the numbers simply could not reveal.
  • What to report?  At the beginning, don’t set up the wrong expectations based on the numbers you offer to report. Remember that you are interpreting and communicating the data to people who do not always understand the tools, the context, and the nature of the activity that happens on the social platform. Even if you grasp the data, you won’t really be able to do anything with it unless you are truly a part of that activity.
  • “If you can measure it, it is probably not worth managing.”  Too often people/organizations get obsessed with their numbers and then allow those numbers to distort their behavior:
    • “Everyone from BuzzFeed to the Washington Post seems to be chasing after viral content because of the traffic it brings — but all this does is reinforce how doomed the page-view model really is.”
  • What should you monitor?  Treat your ESN as an ecology rather than a mechanical system. As you monitor activity, track where the energy is. Then deploy tools that help you follow and amplify that energy. What should you look for?
    • Who is using the ESN?
    • What are they doing with it?
    • Where are they using it from?
    • How often are they using it?
    • What are the reactions?
  • Thanks. The most important metric is the extent to which the system is actually helping people. To show this, Euan did a monthly search for the word “thanks.” Next he dug deeper to identify what help was provided and how that help made a difference. Then he told stories that explained the impact. A member of the audience reported that they have introduced simple hashtagging so contributors can tag success stories as #forthewin. Then the KM group can track that hashtag to collect the success stories.
  • How to report the ESN activities? While there may be pressure to report numbers, try to keep the focus on stories that matter in the context of the person enquiring about value.
  • The biggest value of an ESN? Often the biggest value of an ESN to an organization is that is shows what actually is happening inside the organization. Admittedly, sometimes uncovering this pushes the organization to a dark place. However, it is necessary if we want to manage and improve the organization’s performance.
  • Senior Sponsorship? Euan was lucky that he had sufficient network clout to move (several times) to a new sponsor who could provide the support necessary for the ESN initiative. His advice is to find a new boss and move if you need something your current boss cannot provide.
  • Reverse ROI. Euan pushes back on ROI inquiries. In his view, the social business activity is happening, so we should go with the flow rather than using ROI inquiries as a means of stopping the activity. He asks the inquirer to provide the cost of ignoring or disallowing ESN activity.
  • Fix Problems. Makes things concrete, not abstract. Find ways to fix problems for real people.
  • Work Horizontally. If you cannot go upwards, go horizontally. If you cannot get support from above, pull in people at your own level across the organization who can become involved, become persuaded, and then become part of the success story of the ESN.
  • Use the tools to manage and report on the tools. Try working out loud to surface things that may be challenges in the network. If you see tension or problems, work out loud by asking the network what is going on and what to do about it.
  • Good networks self-heal. It’s important to revert back to the network any issues, queries or concerns. Then let the network address it. This is important for its health and growth.
  • Get used to being misunderstood and under-appreciated.

Are You Creating Value?

Am I creating value? That’s the key question to start and end every working day.

For knowledge management professionals, it can be a tough one to answer honestly. Why? Because many of us struggle with proving the value of knowledge management efforts. We know that we’ve helped individuals, but we are often hard pressed to explain how much we have in fact helped.  For example, you might truly believe that the enterprise search engine you’ve painstakingly implemented will save lawyers time and effort, ultimately saving clients money.  But do you have any metrics to prove it?  Unlikely.  So how do you know that your search engine project creates value?

One approach is to sit next to your colleagues with a stop watch measuring the time spent on searches before and after your enterprise search engine is implemented. Then you should have the data necessary to prove value numerically.  But how do you measure user satisfaction? You could ask users to complete a survey.  With tools like zoomerang or surveymonkey, it’s almost too easy to do this.  However, the real challenge lies in how the survey is constructed and interpreted.  An additional problem is that it can be hard to coax busy lawyers to complete your survey.

If you’re looking for ways to show how much value you’ve created, consider the example of Morrison & Foerster.  On a page whimsically entitled “Geek Power,” the firm makes the following claims about their knowledge management program:

In order to take maximum advantage of the collective experience of our lawyers, we have developed a number of important knowledge management systems and tools.  These systems improve our efficiency.

AnswerBase. AnswerBase is our award-winning enterprise search engine.  The search engine enables us to access the firm’s best and most pertinent practice materials, internal research, attorney experience, client and matter information and other important firm information.  AnswerBase has won a number of awards, including an award from Law Technology News (“Best Collaboration in Implementing Enterprise Search”) and Citytech Global Tech Leaders Top 100 (“Law Firm Project of the Year”).

Knowledge Exchange. Our Knowledge Exchange database makes documents, forms, templates, precedent, briefs, practice materials and internal research available to all attorneys.

They back this up with an exercise they undertook in 2006 to prove value.  Specifically, they retained Bruce MacEwen of Adam Smith Esq to talk to MoFo attorneys about their experiences before and after AnswerBase.  According to Bruce MacEwen:

I was retained by Morrison & Foerster to lead an analysis and review of AnswerBase vis-a-vis its predecessor Knowledge Management system during last summer and fall, and reached the resounding conclusion that AnswerBase was strongly superior to the firm’s legacy systems, by providing highly relevant documents and discovering genuine subject-matter experts within the firm with impressive accuracy.   By interviewing a broad cross-section of lawyers at the firm’s New York offices, I was able to determine that the design and functionality of AnswerBase essentially replicate, as I put it in my report, “the way lawyers think” rather than reflecting technical considerations or limitations.

Admittedly, hiring someone of Bruce MacEwen’s caliber will be hard to justify for every small project on your to do list.  However, I’ve recounted this story to remind you (and me) that sometimes it makes a lot of sense to bring in an impartial third party to help you and your colleagues see what is right in front of you.  And if in the process you manage to demonstrate that your KM efforts have created value, that’s all the better.

[Photo Credit: Dave Elmore]


Collaboration’s ROI

Collaboration is like motherhood and apple pie.  Who will publicly say that it’s a bad thing?  Nevertheless, many knowledge workers have private work habits that inhibit collaboration.  Further many of their organizations don’t do enough to change these behaviors.  Why?  In many cases, because they have not yet realized the enormous benefits that can accrue to an organization that fosters collaboration.

Not convinced?  Looking for some hard numbers?  Take a look at these results from Cisco’s implementation of Web 2.0 and collaboration technologies in fiscal year 2008:

  • US$691 million saved
  • 4.9 %  increased productivity
  • The technology investments, which cost US$81 million to deploy, provided a 900 % return on investment (ROI).

Now your firm may not be as large as Cisco and you may not have the same access to state of the art technology or a workforce that is tech friendly.  Nonetheless, wouldn’t even a fraction of Cisco’s ROI be welcome at your firm?  Further, wouldn’t your firm benefit from improvements in the way information and expertise are shared among employees, customers and partners. What more do you need by way of incentives?

If you’re interested in learning more about how Cisco used Web 2.0 and collaboration technologies to achieve these impressive results, read their guide,  Creating a Collaborative Enterprise (PDF), which explains their framework for achieving collaboration with significant ROI.

[Thanks to John Tropea and Oscar Berg for letting me know about this resource.]

[Photo Credit:  Bee-side]


The editors of the ABA Journal have recognized Above and Beyond KM as one of the top 100 law blogs of 2009.  They are requesting your votes to help them determine which of these blogs are the most popular.  To vote for this blog and your other favorites, please click on the picture below.  Thanks a million!


Why Worry About Law Firm KM ROI?

From time to time, law firm knowledge management junkies twist themselves into knots trying to determine the best way of calculating the return on investment (ROI) of knowledge management efforts. I’m as guilty as the rest for engaging in this exercise. A few have suggested that thinking about ROI is not helpful to KM since knowledge management done properly should create client work product more efficiently, thereby reducing the number of billable hours required and the size of the bill presented to the client. For these folks, that’s reason enough not to talk too loudly about ROI.

Jordan Furlong has a better approach. In his post, Never mind the billables, he points out that we shouldn’t be conflating the cost to the client with the cost to the firm. The client will pay what the client is willing to pay. Therefore, the best way for the firm to protect itself is to reduce its own cost. Then, as the client imposes more constraints on the amount the law firm may charge, the firm can maintain or increase its profit margin by carefully containing its own costs. This is where knowledge management can help. Here’s how he describes this:

Every time you reduce your costs, you create an equivalent opportunity in your profit column, because the amount you spend to render a service to your clients has no effect on the value of that service to the client. (It never has.) Your client doesn’t care how much profit you make for yourself; the client only cares that you delivered excellent value in a cost-effective (to the client) manner. How you bill your services is between you and your client; how much it costs you to deliver those services has to be your number-one business priority.

So let’s take another look at knowledge management ROI in this context. Granted, for firms that can’t think beyond the billable hour, this may seem premature. But for firms with foresight, separating (at least for planning purposes) client cost from firm cost should help open a competitive advantage.


Envious of ROI

A newly-published article on knowledge management began with the following example to illustrate why organizations should even bother with a KM program:

Siemens, the global telecommunications giant, recently won a $460,000 contract in Switzerland to build a telecommunications network for two hospitals in spite of the fact that its bid was 30% higher than the competition. The secret to Siemens success was its knowledge-management system. This system allowed Siemens people in the Netherlands to draw on their experience and provide the Swiss sales reps with technical data that proved that the Siemens’ network would be substantially more reliable than the competition’s.

I can’t remember seeing a public report of an instance where a law firm was able to document a success like this attributable to KM.** Can you? Is this because law firms are hyper vigilant about confidentiality and, therefore, don’t tend to talk about how they work? Or is it because law firms don’t have comparable success stories? Or worse still, what if law firms do have similar successes, but they don’t know it or don’t know how to document it?

While envy is rarely a good thing, I suspect most law firm knowledge managers would be envious of their counterparts at Siemens. I can’t say I blame them.

** To be fair, several law firms have impressive KM technology. What I haven’t seen is published evidence of the ROI resulting from those tools.


Getting Your Money’s Worth Out of KM

Lately I’ve been thinking about whether law firms value knowledge management and how to measure knowledge management ROI. The underlying concern is that law firms don’t know how to measure and value knowledge management activities. (If you ask most law firm knowledge managers if their firms are doing a good job valuing KM, I suspect you’d receive only negative answers.) And, allied with this is the concern that knowledge managers don’t know how to assist this effort in a meaningful way. We often just throw up our hands and say that it is impossible to measure the ROI on knowledge management. Or, we bury law firm management in an avalanche of useless statistics regarding our activity rather than demonstrating true productivity. And then, we slink back to our corners and feel sorry for our undervalued selves.

Stepping back from that a little, I wondered whether there was a way to separate some basic human tendencies from the economics of law firm life. For example, is it a basic human tendency for all but the most arrogant of us to feel as if we are not properly valued for all the wonderful contributions we make to our organizations (and the world, the universe, etc.)? After all, how many of us actually ever have our egos stroked sufficiently??? If so, how does this color our assessment of our KM contributions? From the perspective of the organization, does the firm even know what it is spending on knowledge management? Does it know what it ought to expect for that investment? And, does the firm know how the lawyers value the results of that investment?

All of this came into sharp focus when I read a report of a recent airline initiative to charge $15 for the first checked suitcase and $25 for the second. Suddenly, the cost/value of having extra clothing and gear options on a trip became very stark. What if we were to charge $150 (or some other amount appropriate for the purposes of this exercise) for the first use by a lawyer of the KM system and $250 (or some other amount) for each subsequent use? Assuming we priced this correctly, and made these personal charges rather than client billable charges, would we then see what value lawyers placed on the KM system?

As long as KM systems are free to users, will those systems always be taken for granted, undervalued, and criticized? Conversely, if these systems are expensive to the user, will we have a better way of judging their true value in the firm’s internal market? Now let’s slink off into our corners and ponder those questions rather than feel sorry for our currently undervalued selves.


Measuring Knowledge Management ROI

We’ve achieved unprecedented levels of unverifiable productivity! That’s the punch line from a fabulous Dilbert cartoon I saw last year. And, it sums up so much of what passes for measuring the Return on Investment (ROI) of knowledge management. All too often knowledge managers report on their level of activity because that is concrete, but have a much harder time determining the true impact on their organizations of their KM activities.

In the context of law firms, the ROI of KM conundrum becomes even more vexing. When value to the organization often is summed up by the amount of billables realized, it isn’t always clear how a particular how-to guide or model document made a financial difference to the firm. Sure, having that KM content probably allowed a lawyer to work more efficiently (reducing billables, which may not be seen as a good thing), thereby permitting that lawyer to provide service to more clients (expanding the client base) or get home earlier (improved work-life balance) and maintain better morale (increased retention levels and decreased recruitment costs). But exactly how do you reduce this to dollars and cents?
Given this perennial problem, it’s worth taking a look at the commonsense advice contained in the European Guide to Good Practice in Knowledge Management — Part 4: Guidelines for Measuring KM. This guide proposes the following starting points:
1. define your goals — the clearer the goal of a KM initiative, the more obvious the measures
2. identify the stakeholders for your measures — each stakeholder defines KM success differently
3. define the measures — they need to be valid and reliable, and must yield actionable information
4. decide what data will be collected and how it will be collected
5. analyze and communicate the measures — communicate your findings in a way that helps the reader understand the value of the KM initiative — don’t just present raw data
6. review your combination of measures — finding the right measures comes through a process of trial and error, so you need to monitor and evaluate your measures regularly — adjusting them as your goals develop.
7. “measuring for the sake of measuring is a waste of time” — measure for a specific purpose
8. be sure that useful action will result from the measures
9. don’t measure everything — focus on what’s important
10. to the extent possible and productive, use measurement systems that are already in use within your organization — presumably, these measures exist because they relate to something important for the organization, and this piggybacking allows you to integrate your results within organizational reports.
[Thanks to the UK National Library for Health for pointing me to this guide.]