Dave Snowden Keynote: Big Data vs Human Data #KMWorld

KMWorld 2013Speaker: Dave Snowden, Founder & CSO, Cognitive Edge

[These are my notes from the KMWorld 2013 Conference. Since I’m publishing them as soon as possible after the end of a session, they may contain the occasional typographical or grammatical error. Please excuse those. To the extent I’ve made any editorial comments, I’ve shown those in brackets.]

Session Description: Will information come from the misty mountains of the internet or the cloud with no human engagement as Big Data suggests? Don’t we need human sensors to share knowledge? Our popular and provocative speaker discusses the cycles of techno-fetishism that try and ignore the importance of human intelligence, seeking to create the great algorithm which will answer the questions of life, the universe, and everything else. Big Data is important, but it’s only the start of the journey, and savvy organizations realize they need a synthesis of machine and human intelligence. Get lots of insights and ideas to take home to your organization.


  • What’s the Role of KM? This talk is part of a series of talks Dave Snowden has given in an attempt to drag KM professionals away from fruitless activities, such as simply storing information. For him, the role of KM is to enable innovation and support better decision making by the organization. To do this, we need to manage the entire knowledge environment — not just the bits that are easily codified.
  • Technology vs the Human: Technology in its place is good. What’s its place? To augment not replace human capabilities. As we use technology, it changes us. Consider the smartphone — it has become so deeply entrenched in our lives that it is almost an extension of our brains. And as we come to rely on it, we lose some capabilities we had before. In fact, human beings can lose specific capacities over the course of 1-2 generations. He cites the example of the slide rule. People trained to use it also developed the ability to understand numbers in a particular way. Further, it turns out that people who learn to use the slide rule seem better able to see errors in computer code. By contrast, people who use calculators exclusively do not develop either of these abilities. In this case, the slide rule augments human capability while the calculator replaces (and possibly diminishes) human capability.
  • Re-discovering the value of the human brain: The human brain is 1.4 kg of fats and tissues. Yet it can still outperform many algorithms. The human brain has developed to do well at pattern recognition, not process information. Computers have been designed to process information. When we force people to absorb and store information, we are not allowing them to do what their brains are designed to do best.
  • Narrative Learning: This is the best way of humans to make sense of the world around them. We tell stories to identify patterns and convey information. Some stories are oral, some are in text. However, don’t make the mistake of ignoring stories in drawings  — often they are the richest stories.
  • Narrative Mapping: It understands the basic patterns by which humans operate and then helps identify which patterns could or should be changed by amplifying the useful and dampening the less useful. “To change a culture, tell more stories like this and fewer stories like that.”
  • The Efficient Brain: There are a host of responses of the body and brain that happen without conscious thought. These are autonomic responses For example, if your hand gets near an intense heat source, you automatically move it. You don’t have to think about it first. This is an efficient way of operating because it helps the brain use less energy. (It is already a huge energy hog.) There are even more things that can become almost autonomous, with sufficient practice. For example, after 2-3 years, we can drive cars without much conscious thought. Similarly, it takes 2-3 years before we can reliably recognize errors in computer coding. This isn’t just an information processing issue. It is a matter of experience, training and judgment. This is why we need to bring back apprentice programs. They permit repeated practice and, most importantly, they create a reasonably safe environment in which to experiment and fail. This is critical because we learn more from failure than from success.
  • The Brain Evolves: Our brains evolve to respond to inputs and the environment. For example, over 2-3 generations of constant input or practice, there are resulting biological changes in the brain that make that practice unconscious.
    • Aristotle: “Knowledge must be worked in the living texture of the mind, and this takes time.”
  • Brain Constraints: The brain can handle only 3-4 concepts at one time. The only way to handle more concepts or more complex concepts is through aesthetics — through art, through metaphor. By abstracting things we can absorb much more complexity and nuance.
  • The Impact of the Environment:  Place, our physical environment, can have a huge impact on who we are and how we work. As economics and the drive for cost savings are forcing people into cubicles or common work spaces, the new work environment can have the effect of eliminating diversity of thought.
    • Dissent is more important than consensus. It is a myth that everyone should be aligned. It is important to tolerate tension and support diversity of thought.
  • The Problems with Current Approaches to Knowledge Management: We spend a lot of time trying to stop people from working in silos or encouraging them to share across silos. We should forget about it since we can’t stop them from working the way they do. The better approach is to have them create metadata. People tend to be much more willing to share metadata with people outside their silos. The shared metadata can spark new ideas on the part of the people who receive that metadata.
  • The Power of Narrative: In Iraq, the troops had no use for doctrine. What they valued most was blogging from the front lines. The secret of narratives is that they can handle ambiguity, they can be complicated and messy. Because of this, you can convey more information and more complicated information. Further, each listener will extract from the story the elements that are most relevant to the listener in the moment. Years later, that person may not remember the details, but they will remember the gist of the story.
  • Wisdom of Crowds: We have come to believe that crowd-sourced information is uniformly good. However, sometimes this so-called wisdom is nothing more that “the tyranny of the herd.” By way of example, consider the Dutch Tulip Mania,  South Sea Bubble,  and the recent sub-prime mortgage crisis.
  • Human Sensor Networks: This involves using people to elicit oral histories from a larger group of people. In Wales, they are using school children to ask people in the community what matters in their community. This project will replace polling and focus groups. It will provide the basis for evidence-based policy-making. Better still, once this network has been designed and created, it can be reactivated later to provide answers to specific questions as the need arises. Further, these networks can be used to disseminate information rapidly.
  • Proactive Foresight: Ideally, we ought to create networks that do more than provide restrospective coherence. We need to build networks that help us develop proactive foresight — the ability to sense what is likely to happen and then prepare for it.
  • Repositories vs Networks: If you have a choice between building a repository or a network, choose the network.  Snowden: “Repository rhymes with suppository. Guess which is better?” On a more serious note, real-time data (gathered through the network) are more valuable than data that have been pruned and polished later. In fact, fragmentary data are hugely valuable, but they are often culled and lost forever in the polishing process.
  • Big Data vs Human Narratives:
    • While big data can tell you what happened (e.g., Joe got on the subway at 8:45am), only stories can tell you why it happened.
    • Another problem arises from the way we tend to interpret data. Typically, we eliminate the outliers and look for the general trends. The problem with this approach is that the strategic opportunities and threats often exist in those outlying data points.
    • Search algorithms also disregard outlier data. They focus on the most commonly searched concepts and on popular links. What are we missing by disregarding the outliers?
  • Exaptation: Adaptation is when we develop for a specific function. Exaptation is when we develop for a specific function and then that new capability is used for a completely different purpose. [Perhaps this is a human example of “off-label use”?] We need to create a KM ecosystem for managed exaptation.
  • Judgment: We need to create trust and training to help people exercise human judgment. Human sensor networks allow us to express opinions on important issues before the political climate requires us to take a hard and fast position that has to be defended to the death.
  • Focus on Designing an Ecology, Not a Machine: Think about people and computers working together in an environment, rather than building a system. If we fall into the pattern of letting computers do what humans ought to be doing, humans will lose the capability to do that which they must do. Respect technology, but respect human capability more. Design technology to augment human capability, not replace it.
  • The final words go to Hugh McLeod:
    • “Change is not death. Fear of change is death.”
    • “What we Are is changing quickly. What we MUST BE, even more so.”

Dave Snowden:Finding New Solutions to Wicked Problems #KMWorld

Dave Snowden, is the Founder and Chief Scientific Officer of Cognitive Edge.

[These are my notes from the KMWorld 2012 Conference. Since I’m publishing them as soon as possible after the end of a session, they may contain the occasional typographical or grammatical error. Please excuse those. To the extent I’ve made any editorial comments, I’ve shown those in brackets.]


  • Wicked Problems are Intractable Problems. Intractable problems are ones to which traditional solutions have been applied, but have not worked. These problems present a really strategic opportunity to Knowledge Management. If you can solve these problems, you will become vital to your organization. On the other hand, if you address only conventional problems, your knowledge management will never be anything but conventional — it will never be strategic.
  • Complex Adaptive Situations. These situations are not causal, they are dispositional. In other words, there is no direct link between cause and effect. Therefore, there are no drivers that can be identified and applied. Further, traditional failsafe design does not work. So don’t waste time hiring consultants or doing copious amounts of research. Instead, run several rapid safe-to-fail experiments to “probe” what is going on by testing quickly any coherent theory that emerges and then move on depending on the results of your results. Your portfolio of theories of test must include some contradictory theories. Otherwise, you haven’t cast your net widely enough and have missed something. In addition, some of the portfolio must be oblique — something intended to solve another problem altogether. (See Obliquity by David Kay.) Some of the theories in your portfolio should be naive, which means that they should be formed from the perspective of one who is not an expert.
  • Complicated Problems. These problems can be addressed by a traditional fail safe design. For these problems, there are leading theories or process that can be implemented.
  • How to design a Safe-to-Fail Experiment that addresses an Intractable Problem. Begin by answering the following: (1) Name of experiment, (2) Rationale for experiment, (3) Indication of success, (4) Indications of failure, (5) Amplification strategy (after success), (6) Recovery strategy (after failure), (7) Actions (to follow success), (8) Responsibility for actions (after failure). After you have done your preliminary experiment design, conduct as many as five rounds of ritual dissent to tighten your design.
  • Ritual Dissent. The benefit of ritual dissent is that it is an extraordinarily effective way of identifying weaknesses and potential problems with your experiment before you actually fund and carry out your experiment. Between each round of the ritual dissent, the planning group takes the criticism they received in the round just completed and use it to improve their experiment design. Dave Snowden recommends that you do as many as five rounds of ritual dissent to ensure that many people have had a chance to test your experimental design.



Topspin and Tacit Knowledge

Do you know what you know? And, more importantly, do you know how to communicate it effectively to someone else? For far too many of us, the answer to both of these questions is “No.”

To be fair, we may think we know the extent of our knowledge and may even believe we can be effective teachers of that knowledge, but Malcolm Gladwell suggests that we are just fooling ourselves. Take a look at the brief video clip below of Gladwell discussing why people succeed. He recounts instances where professional tennis players believed they were giving an accurate account of their knowledge and practice, and yet a video of their game proved the inaccuracy of what they said. At the end of the day, the explanation they gave about how they hit a topspin forehand did not match what they actually did.  Rather, their instructions would have led to a sprained wrist. Were they just dumb? Gladwell doesn’t suggest that.  Instead, he says that their knowledge as extremely competent professionals was instinctive and they really weren’t able to reduce it to words that could produce a topspin forehand if put into practice by someone else.

Now consider the implications of this for knowledge managers who seek to “capture tacit knowledge.” It is an article of faith in knowledge management that some of the most valuable knowledge is tacit knowledge:  that part of knowledge that comes through experience and cannot easily be codified into explicit knowledge.  It’s prized and it’s elusive.  Dave Snowden years ago reminded us that we know more than we can say and we say more than we can write down.  Yet so many of our knowledge management systems depend upon the written word. If you’re lucky, your knowledge management system will contain merely incomplete information.  If you’re unlucky, your attempts to render tacit knowledge explicit may result in information that is just plain wrong — like the instructions on how to hit a topspin forehand.

What are the solutions? Rather than asking experts to write everything down, consider making a video.  But have that video focus on what the experts are doing — not what they are saying. As we discovered with the tennis players, verbal explanations may be no more accurate than written explanations. Better still, facilitate knowledge transfer by having the experts work directly with less knowledgeable people.  It’s this old-fashioned apprenticeship approach that maximizes the flow of tacit information.

Granted, instituting an apprenticeship isn’t quite as cool as implementing new technology. But if you really want to learn how to hit a topspin forehand, you will have to learn by watching and doing.  If you rely on the incomplete transfer of tacit knowledge into verbal or written instructions, you may end up with a sprained wrist.

You’ve been warned.




Coalition of the Willing

Lawyers in most firms are given a lot of freedom to decide how to manage their own knowledge. In fact, it’s a rare law firm that can demand that its lawyers handle their knowledge in a particular way. For many, the battle began and ended with the document management system. At this point, most firms with document management systems have persuaded their lawyers to create and store documents primarily within the DMS. This has the signal benefit of ensuring that the firm’s work product is located in one place.  The problem, of course, is that while you can require that documents be created within the DMS, it’s much harder to get lawyers do anything more than the most rudimentary profiling of their documents.  As a result, it has until recently been extremely difficult to capture much metadata regarding a document. What’s changed? In part, it’s that lawyers are beginning to learn the value of metadata to assist in the document searches they do every day.  In addition, new document management systems are more intelligently designed and allow simpler filing of documents, coupled with the ability to let new documents “inherit” metadata from the folder in which they are placed.  Couple this with the metadata extraction capabilities of some work product retrieval systems, and the burden on the individual lawyer to create metadata is lightened considerably.

So the good news is that after nearly 20 years of document management systems, we’re finally getting to a point where the technology allows them to work more seamlessly and intuitively for lawyers.  This should encourage greater use (and more rewarding use) of the DMS by lawyers. The bad news is that relatively little of a firm’s knowledge in contained in its work product. What’s your strategy for dealing with that problem?

Unless your firm is run by Attila the Hun, you won’t be able to compel lawyers to share their knowledge via a central repository or medium.  Further, you will run into the problem observed by Steve Denning (see The Economic Imperative to Manage Knowledge) regarding the behavior of “experts” with respect to their knowledge:

As preliminary efforts to establish what the organization knew were launched, it started becoming apparent – to the surprise of many – that the organization did not know what it knew. Inquiries as to the cause of the hesitancy revealed that even the experts were not sure of what they knew. The experts even contested whether they were responsible for sharing their knowledge. They often contended that their job was to meet with their clients and deal with their needs, not sit in an office in headquarters and assemble best practice manuals.

What’s the solution? If you can’t compel sharing, you’ll need to coax sharing.  The best way to do this is to work individually with your experts to identify their personal knowledge management challenges and then find ways to address those needs in a manner  that results in a solution that is satisfactory for that expert AND yields rich material in a selectively shared content repository. Notice, that I used the words “selectively shared.”  Unless you can promise some measure of control over the knowledge, you’ll have a hard time winning the cooperation of your experts.  They will undoubtedly want the freedom to gather and organize the content as they see fit — not as necessarily as the IT department dictates. The key here for technologists and knowledge managers alike is to provide very lightweight systems that provide the individual flexibility cherished by experts. One obvious choice is the range of Enterprise 2.0 tools now available, but I could imagine implementing some firm-wide systems in a way that encourage personalization, sensible organization and sharing rather than the unmanageable wilderness currently found in everyone’s favorite content repository — Outlook.

One challenge is that your work with these individual experts will result in information silos.  However, you can go some distance in managing these new silos by ensuring that the content can be shared easily. Then, see the good that happens when your intelligently-designed system interacts with what Dave Snowden observed as our basic tendency to help in times of true need.

The bottom line is that you have to build a coalition of the willing — willing experts, that is.  Once you’ve helped them organize and find what they know, they’ll be better equipped to share that with others.

[h/t to John Tropea for pointing out the Steve Denning piece]

[Photo Credit: lumaxart]


Defining Ourselves

One of the thorniest problems we’ve faced in knowledge management has been how to explain what we do. Ray Sims set out to determine if there was a definition of knowledge management that could help with this. What he discovered was not one or two, but rather 62 definitions of KM. That’s more than one for every week of the year!

Into this murky mess steps Dave Snowden. Although he has nothing to prove, his recent post, Defining KM, demonstrated yet again why he is considered one of the foremost KM experts in the world. Here’s how he defines KM:

The purpose of knowledge management is to provide support for improved decision making and innovation throughout the organization. This is achieved through the effective management of human intuition and experience augmented by the provision of information, processes and technology together with training and mentoring programmes.

The following guiding principles will be applied

  • All projects will be clearly linked to operational and strategic goals
  • As far as possible the approach adopted will be to stimulate local activity rather than impose central solutions
  • Co-ordination and distribution of learning will focus on allowing adaptation of good practice to the local context
  • Management of the KM function will be based on a small centralized core, with a wider distributed network

There’s a lot to chew on in this definition.  I heartily agree with his assertion that the point of KM is to support “improved decision making and innovation.” If this isn’t why you’re involved in KM, what are you doing?  It also raises an interesting question for people engaged in KM 1.0.  What proof do you have for your operating premise that larger document repositories improve decision making and innovation?

For those who are KM empire builders, his guiding principles will give pause.  He is clearly favoring local, grassroots solutions rather than centralized, large-scale solutions.  This will require placing people close to the frontline who are knowledgeable enough about KM to provide some light guidance to the knowledge workers who have the most immediate need of KM systems.  Better still, to my mind it encourages every knowledge worker to be an effective knowledge manager.  In this context, a global KM Czar is going to be superfluous and unwelcome.

In separate correspondence with Dave Snowden, I’ve asked if he can elaborate on his notion of “effective management of human intuition and experience.”  It’s not clear to me exactly how one manages either intuition or experience.  It will be interesting to what additional guidance he can provide.

In the meantime, stayed tuned.  By offering this definition, he’s given us an opportunity to define ourselves and our work again.  Let’s see how far we get this time.

[Photo Credit:  jovike]


Linear is Not Always Best

Our society has made a fetish of linear thinking. We’ve been trained to expect that A will lead to B, which in turn will lead to C. We breathe a sigh of relief whenever we experience what Webster’s New Millennium Dictionary of English describes as a “step-by-step progression where a response to a step must be elicited before another step is taken.”  All of this is deeply comforting — even when it is not entirely appropriate.

In the June 2009 issue of KMWorld Magazine, Dave Snowden recounts an experience from the beginning of his career in which he elected to design a new system in a manner that didn’t fit well within established design methods.  He was creating something that had never existed before and decided early on that IT’s usual linear approach wasn’t going to work.  In fairness, it sounds like he initially did try to conform.  However, once he set about to gather requirements he quickly discovered that

…few if any of the users had any idea of the capabilities of software.  As a result, if you asked them what they wanted, they told you what they currently did, or asked for automation of existing processes.  To use an adage of that time, `Users say they know what they want until they get it, and then they want something different.

Instead of IT’s traditional linear approach, he adopted an iterative method whereby he and his clients engaged in a more curvaceous  “co-evolutionary process” to develop the new system.  Drawing on his own substantive experience of the work his clients were trying to do, he approached the design effort in the following way:

…I could talk with the users in their own language; go away and develop a module with real data; and create reports, monitoring screens and other processes based on a synthesis of my knowledge, the stated needs of the client and my knowledge of the technology.  The application would work in novel ways, users would find new ways of working, and modifications would be agreed upon.  Over the course of a year, a powerful application emerged that was very different from anything that either the user or I could have defined.

In many ways, this is a textbook description of how to implement social media tools within the enterprise.  Work iteratively with your users, create opportunities to learn from each other and from the tool using a series of “safe-fail” experiments, stay in beta for as long as it takes to reflect user reality in your tool, and don’t be afraid to step off the straight and narrow path of linear thinking.  To be clear, this is not a recommendation that you abandon all logic in your design and implementation.  Rather, it is a reminder that there can be great beauty and greater rewards in following a more circuitous route.

[Photo Credit:  Headsqueeze]


Knowledge Management 101?

Sometimes we just want someone to tell us what needs to be done and how to do it.  For those moments, there are hundreds of “how-to” books that purport to tell us how to “do” knowledge management — beginner’s guides, dummies’ guides, idiot guides, lazy person’s guides, etc. They are in the bookstores and their advice is regurgitated in any number of blog posts.  Unfortunately, too many of them are a waste of time. They will point you in the direction of KM 1.0, which invariably requires lots of people and technology to attempt the nearly impossible task of compelling your colleagues to convert their tacit knowledge into explicit knowledge, and then capturing and organizing the relatively small amount of converted information that results from your efforts.  Too few of these “helpful guides” actually explain how to align your activities with the strategic goals of your organization or the futility of trying to comprehensively capture knowledge.  Further, they encourage us in the delusion that it is possible to “manage knowledge” in a controlled, top-down manner.  In other words, few of these guides have a realistic view of how human nature invariably trumps neat centralized schemes and how critical it is to work directly with the workforce in a grassroots way if you’re serious about creating and perpetuating an effective knowledge sharing culture.

In short, these guides make fundamental errors that folks wiser than me (in this case, John Bordeaux) have already identified:

Believing that knowledge is only transferred once it has been made explicit leads to mechanistic, engineering approaches to knowledge management that have not proven their worth.  Crank it out of people’s heads, churn it into a shared taxonomy or tag it somehow, and then – and only then – is it useful to others.  I would like to know the exact date that the apprentice learning model was made obsolete by advanced information technology.

While a tidy approach to KM (actually more an approach to information management), the call to “make tacit knowledge explicit” ignores much of what we know about how the world actually works.  To be more precise, we are learning the limitations of what we can know as a result of research across the disciplines of sociology, neuroscience, anthropology, and others.

A far better approach is to think hard and then think harder again about human nature — how we learn, how we know and how we share what we know.  And then, put your organization and colleagues under a microscope and study them until you have an accurate understanding of how the knowledge ecosystem within your organization works.  When you’re ready to do this, here are some useful guides to help you along your way:

These are not idiot guides — they are invitations to deeper study and thought.  Better still, they contain truths that will outlive the quick take-aways of the how-to guides.  The best way to use these recommended materials is to read them with a critical eye, and then find some trusted colleagues with real KM experience and discuss* with them what you’ve learned through your reading and work.  Then, rinse and repeat.

* I’d be delighted to have that conversation with you online.  Just let me know by leaving a comment below.

[Photo Credit:  tsmyther]


When Failure is Fine

Every so often, we’re fortunate enough to hear about an organization that has mastered the art of innovation. In the arena of social media, Best Buy is getting a reputation for innovation and success. This week I learned about an extraordinary feature of Best Buy’s corporate culture when I read Cam Gross’ blog post regarding their implementation of Mix, which he described as “a start-up offering a mashup of email, SMS and Twitter-like functionality. ” When I wrote about Mix in an earlier post, I was impressed by their repeated attempts to broaden the conversation among Best Buy’s employees. Given their success with Blue Shirt Nation, I assumed that they ultimately would be successful with Mix. (Blame the lawyer in me for relying excessively on precedent.) What I didn’t fully understand was that, apart from their prior achievements, they had one of the most critical ingredients necessary for success. Here’s how Cam Gross describes it:

We have had almost zero conversation about Mix outside of the development group. When the article by Laura Fitton (@Pistachio) came out … word traveled inside Best Buy Corporate. A couple of departments have already raised their hands anxious to test/sample Mix. You just can’t beat having an environment where people want to try and are OK with “fail” as long as something is learned. [emphasis added]

Now, be honest. When was the last time you heard someone say that about your company?

We talked earlier about the value of mistakes when pursuing growth and innovation, and Dave Snowden has included in his Seven KM Principles the truth that we learn more from failure than from success. But have we ever gone so far as to say in our workplaces that it’s okay to fail as long as something is learned? Rather, don’t we circumscribe our actions and ambitions in order to avoid failure at all cost? Admittedly, if we do find ourselves staring failure in the eye, we’re usually willing to attempt to redeem that failure by looking for a nugget of learning. But I’d suggest that the Best Buy attitude goes further than that. It sounds like they don’t circumscribe actions and ambitions for fear of failure, but rather choose to risk doing something new — knowing that they can learn from it. With that orientation to innovation, they are bound to succeed.


7 Principles of Law Firm KM

Dave Snowden‘s 3 Rules of knowledge management have expanded to 7 Principles, now that he is focusing on law firm knowledge management. (Perhaps there is just something about lawyers that invites the creation of more rules). Here are the 7 Principles:

1. Knowledge can only be volunteered, it cannot be conscripted.
2. We only know what we know when we need to know it.
3. In the context of real need few people will withhold their knowledge.
4. Everything is fragmented.
5. Tolerated failure imprints learning better than success.
6. The way we know things is not the way we report we know things.
7. We always know more than we can say, and we always say more than we can write down.

This is a list worth chewing over. I expect I’ll come back to it several times. In the meantime, I’d urge everyone involved in law firm knowledge management to take a hard look at their KM programs and measure them against these 7 principles. A large number of firms are engaged in classic KM 1.0 efforts: trying to convert tacit knowledge into explicit knowledge, creating precedent collections and brief banks, writing practice guides to convey best practices, etc. These methods seem to violate one or more of the 7 Principles. It would be worth spending a little time to determine if you are achieving the levels of success you and your firm anticipated from this efforts. If not, how much of that is due to the fact that your projects do not conform to these principles? If you are truly successful in your KM 1.0 approach, we should talk. You may have identified an interesting exception to the 7 principles.

[Hat tip to Dennis Kennedy’s microblogging on Twitter.]