Enterprise Search Analytics and Reporting #ILTA12

The moderator is Tanisha Little (Senior Manager Knowledge Management, Morrison & Foerster), and the speakers are Rizzi (Directors, Knowledge Delivery, White & Case), and Michael Williams (Director of LIfecycle Information Management, eSentio Technologies).

[These are my notes from the International Legal Technology Association’s 2012 Conference 2012. Since I’m publishing them as soon as possible after the end of a session, they may contain the occasional typographical or grammatical error. Please excuse those. To the extent I’ve made any editorial comments, I’ve shown those in brackets.]


  • Why Focus on Search Analytics?. To report on the positive behaviors, but also to understand the negative outcomes such as which lateral partner hasn’t been integrated, which lawyer is a confirmed luddite, etc. Michael Williams reminded us that the analytics help you understand how the search engine is being used and how it needs to be modified to be more useful.
  • Why should firms invest in analytics?. Ennio Rizzo pointed out that nothing works immediately, most systems need to mature. Analytics can help show trends and behavioral information that add credibility to your claims of success. Michael Williams says that enterprise search that is poorly implemented or poorly tended will result in a poor search experience. However, by using analytics, you can learn quickly about the user experience and how to improve it. This will avoid people turning away because they have been disappointed by their results. Therefore, analytics are a great way of protecting and exploiting your investment in an enterprise search system.
  • What are all the uses of analytics?. There are four key uses of analytics: utilization, demographics, stability of performance and user experience. Depending on where you are in your project, you can weight one more than the other or focus on one in particular.
  • What kinds of information should you be collecting?. Ennio Rizzo showed us some of the information they are collecting at White & Case. For example, they look at search penetration by practice group and title weekly to understand where they need to increase training or provide additional support. They look at who the top users are and then decide whether those people should be recruited as advisors to the enterprise search project. They also produce a heat map regularly to show how, where and when the search engine is used and whether they need to provide additional support. Finally, they can look at practices and industries that most interest specific individuals. This helps identify the expertise of those people. Ennio noted that enterprise search is a big, expensive investment by any firm, so information like this helps the firm improve its return on investment.
  • When to deploy analytics?. Ennio Rizzi strongly suggests that you shouldn’t go to production before you have the analytics in place. Michael Williams agrees that having the analytics ready helps the engineers understand the guts of the tool. He also believes that the analytics that come bundled with the tool are not sufficient. Therefore, most firms need to purchase an additional analytics tool from another vendor. White & Case was able to use easily available web parts to extract the data from the analytics native in Recommind and then presented it via SharePoint. Ennio acknowledged that while the native analytics measured tons of elements, the real art was to extract the relevant information and present in an actionable form. Michael Williams showed an eSentio/Recommind dashboard that could send an alert to the law firm knowledge management group letting them know when a lawyer has a failed search so the KM group can contact that lawyer and help with the search. While this may feel a little creepy or Big Brother-like, Ennio believed that most lawyers appreciated the assistance.
  • Dealing with Irrelevant Results. You should use analytics to help understand why we are getting back too many results or irrelevant results.
  • What should you do before you deploy?. Spend a huge amount of time educating firm leadership and KM champions on the pros and cons of the search engine as built. This will help them come on board and provide you and your project “air cover.” All of this is part of the change management exercise you will have to do when you bring a tool as powerful [and disruptive] as enterprise search to your firm.
  • Relevancy. Michael warned the audience not to fiddle with relevancy too much. If you are not careful, you can quite quickly degrade the search results. That said, judicious tweaking can be very helpful. Just be sure to proceed very carefully.
  • Analytics is all about trending. You can show a change in usage. As long as you can continue to show good trends, that should help to make your case with management. Ennio also suggested using the analytics to find out whether there are resources people are searching for, but don’t exist. The KM team can then approach the practice group and offer assistance to create the missing resource.

A Better Way for Lexis and Westlaw

Legal researchA Notice to Lexis and Westlaw:  You would be well-advised to read this post since it proposes a better model for your products.  (As for the royalty checks you’ll owe me, let’s talk…)

For a few brief months early in my legal career, I was a litigation associate.  As such, the main focus of my days was legal research. Being a diligent sort of person, I’d spend hours tracking down one case after another and yet, despite my conscientious approach, I could never entirely dispel the nagging feeling that somewhere out there was the one case that could blow my research out of the water.  Surely, I’m not the only lawyer who has confronted this.

The cause for this anxiety was the knowledge that my search results were only as good as my search technique.  And I was honest enough with myself to know that I wasn’t nearly as expert at searching as, for example, a research librarian.  However, in those days, it really wasn’t the done thing to have a research librarian do your research for you. So I soldiered on and tried to refine my search methodology.

With the benefit of hindsight, it seems obvious that the current approach to legal research is fundamentally flawed.  Lexis and and Westlaw have created these enormous databases of case law that cannot be completely mastered unless you have world class research skills. In fact, what they’ve created is a frustrating game of “find the needle in the haystack.”  The problem is that the people who need the cases aren’t always the best equipped at finding the cases, and the people who are expert at finding cases aren’t the best equipped to analyze and use them. Further, most law firms don’t promote lawyers on the strength of their research skills. Rather, they promote lawyers on the strength of their analytical, advising, negotiating, writing and business-winning abilities.

It would be better if Lexis and Westlaw aligned themselves with their customers’ need to improve analytical capabilities.  Here’s the new model I propose:  instead of forcing lawyers to come up with appropriate search queries, Lexis and Westlaw should ask lawyers questions to elicit information about the case at hand.  In other words, the role of the lawyer searching for precedent would be to analyze their own case and strategy and provide that information to Lexis and Westlaw: what are the pertinent facts of the case, what jurisdiction, what procedural approaches is the lawyer considering.  Then, Lexis and Westlaw would deliver to you links to groups of cases that match your facts within your jurisdiction.  You could then review them to see how closely aligned they are to your situation.  Ideally, this approach would reveal the array of ways in which lawyers before you had handled this fact pattern in your jurisdiction and would highlight opportunities for following precedent or striving for innovation.  Better still, this should reduce (if not eliminate) the nagging worry that you’ve missed a case.

What’s different about this approach? Success depends on a lawyer’s ability to understand that facts and context of the case at hand, not that lawyer’s ability to construct the perfect search query.  It would no longer be a hit-or-miss proposition depending on your skill at constructing boolean or natural word search queries. (Of course, it would also depend on Lexis and Westlaw profiling and organizing their cases differently, or adopting search technology that is more sophisticated than simple word searching.) The result is that both lawyers, on the one hand, and Lexis and Westlaw, on the other hand, start focusing on their strengths.  Lawyers build their analytical muscles and the online research companies get better at organizing and delivering comprehensive results to you.

Now imagine a law firm in difficult economic times considering whether to drop one online research provider in favor of another. What if one provider were to offer these comprehensive (and more appropriate) search results? Would there be any contest?

In this blog post I’ve provided information on my innovation to Lexis and Westlaw (and any other online legal research provider interested in conquering this lucrative business).   I wonder which one will be hungry enough to adopt this proposal?

[Photo Credit: Garry Willmore.  I’d highly recommend that you click on the photo above to read Garry Willmore’s commentary on this photo.  It’s priceless!]


A Google Instant Experiment

Sometimes my curiosity gets the better of me. That happened this morning, when I wondered what Google Instant would do to some typical searches of mine. The results were thought-provoking:

  • When I typed “law” >> Google Instant served up information on Law via Wikipedia and then information on the television show “Law and Order” first.
  • When I typed “law firm” >> the first law firm name Google Instant showed me was the New York firm, Paul Weiss. Kudos to the Paul Weiss marketing department. (I hope the marketing department of every NYC law firm is paying attention to this!)
  • When I typed “law firm kn” >> Google Instant provided a link to an old blog post of mine on Personality and Law Firm Knowledge Management.  The best part (for me) is that this post appears on the first page of results. (Whew!) Better still, Google Instant helpfully provides on that first results page a link to all my blog posts that have been tagged to the topic “law firm knowledge management.”
  • When I chose the “law firm knowledge management” search query provided by Google  >>  Google Instant offered on the first page of search results links to knowledge management materials from Lexis, the American Bar Association, Amazon.com, and … AboveandBeyondKM.com.

Since Google tailors search results to the user, I asked a friend living in a different part of the state to run a parallel search using Old Google.  Here’s what we found:

  • When she typed “law” and clicked search >> Old Google gave her  the Wikipedia Law article, then Law.com, and then FindLaw.com.
  • When she typed “law firm” and clicked search >> Old Google gave her “Philadelphia Law Firm,” then another personal injury lawyer’s web site, then web design for lawyers, and then the Wikipedia Law Firm article.
  • When she typed “law firm kn” and clicked search >> Old Google gave her the Martindale & Hubell link for  “KN Hyde and Associates” — which appears to be a commercial law firm with offices in several parts of the former Yugoslavia — and then gave her a direct link to that firm’s website as her second search result. The third result is to a law firm in Bangalore, India.
  • When she typed “law firm knowledge management” and clicked search >> Old Google offered exactly the same results to her as Google Instant offered to  me — complete with the link to this blog.  Very nice!

What this test with an unscientifically small sample suggests is that Google Instant knows a fair amount about who I am.  (Although it hasn’t figured out that I’m not a Law and Order devotee.)  That said, I was delighted to discover that this blog ranks well in the search results of an interested party like me, as well as in the search results of my friend who does not work in the law firm world and has never searched for my blog via Google.

Out of sheer vanity (or perhaps prudence), I tried a new search — for my name.  I had to type “Mary Abrah” before I turned up in the search results.  But that’s not too bad.  After all, I do have a fairly common name. When I typed “V Mary A” one of my social media activity streams and my blog appeared on the first page.  I had to type “V Mary Abrah” before Google Instant figured out that I was looking for myself.  Interestingly, when I tried these searches again a few minutes later, I got slightly different results.  However, I don’t know what prompted Google to change the results order.

At the end of the day, it appears that keywords still matter.  However, if your keyword or phrase is too long, users may never get to it if they are offered other options by Google Instant.  Accordingly, it may well be time for you to reconsider your keyword strategy.  If your website gets a fair amount of traffic thanks to Google searches, you may want to think harder about the implications of Google Instant.  I know I will be.

[Photo Credit: dullhunk]


Google Blink Not Google Instant

Google Money (Enterprise Irregulars)

Google unveiled Google Instant on September 8. By most accounts, it’s an impressive step forward in search technology that brings us closer to immediate gratification.

Until now, most of the time related to searching was spent typing a query and then evaluating the answers.  Typically, one would type a query, hit enter, review the results and then click through to see if an item on the results page was responsive.  And then, rinse and repeat until you found what you wanted or gave up trying. With Google Instant, Google offers search results to you before you finish typing your query.  With every character you type, Google attempts to predict your query and serves up search results that it thinks will meet your need. In other words, Google plays a version of Name that Tune with you — trying to identify the song you have in mind using the fewest notes possible.

So how much time will Google Instant save?  Google estimates that a typical search takes approximately 25 seconds.  With Google Instant, you can shave 2-5 seconds off every search.  What does this mean in aggregate terms?  According to Google, “If everyone uses Google Instant globally, we estimate this will save more than 3.5 billion seconds a day. That’s 11 hours saved every second.”

Even still, I couldn’t help thinking “close but no cigar.”  To be honest, while it’s awfully nice that Google can make accurate predictions based on just a few letters in a search box, what I really want is for Google to predict my query and serve up helpful results … before my fingers even touch the keyboard!  This is what I call Google Blink (i.e., getting results in a blink of an eye).  As it turns out, Google’s engineers have already considered this possibility, but called it MentalPlex.  With MentalPlex, all you have to do is stare at the Google MentalPlex page, “project a mental image of what you want to find” and then that action alone will trigger the display of responsive search results.  Now that’s more like it!  Although this may be my fondest wish for search technology, as far as I know, it’s not yet more than an April Fool’s joke for Google.

All joking aside, why should we care about Google Instant or, better still, what I’m calling Google Blink? To quote Eric Schmidt, CEO of Google, “Never underestimate the importance of fast…speed matters cuz your time matters.”


If you’ve got 90 minutes to spare and would like to learn more about Google Instant, here’s the video of the launch announcement:

Alternatively, here’s a quick overview that will take less than two minutes of your time:

Finally, here’s Google Instant served up with a side of Bob Dylan:


Are We Organizing the Right Stuff?

As a self-confessed pack rat, I’ve had a morbid fascination for folks who preach and practice the virtues of minimalism and a clutter-free existence.  Call it a form of self-abuse, but I just can’t help reading their propaganda.  And then, I come every workday to a business that never has a shortage of stuff to organize.  No matter how much we try to move away from the old KM 1.0 chores of building, organizing, maintaining and searching data repositories, the reality is that as long as we work in a document-intensive business, law firm knowledge management will always include a significant portion of KM 1.0 work. Vendors promise silver bullets in the form of super search and auto-profiling to help us manage all this data, but even still we hear stories of people spending hours searching fruitlessly.

Recently Kathleen Hogan reported that studies show that managers and executives spend 6 weeks each year just looking for stuff. My initial reaction was what a waste of time! Can’t they get a better search engine? And then, I had a radical thought — what if we really aren’t supposed to save and organize all this stuff? What if there really is too much to organize effectively? What if there’s no reasonable way to stay on top of it all? Are our taxonomies and search engines designed to cope comfortably with our exploding data collections? What if they can’t?

If you take a look at all the anti-clutter propaganda I’ve been saving for a rainy day, you’ll soon discover that the first step to organizing stuff is — get rid of what you don’t need.  By so doing, you reduce the amount of material you actually have to organize and maintain.  I know we think we need to organize all the data in our firms, but do we really?  Does it all matter?  Does it all need to be saved and organized for posterity?  Or, is some of it truly ephemeral?

Greg Lambert suggests that law firms have been saving all this stuff for all the wrong reasons:

There are certain things we should legally and/or ethically keep for a specific period of time. But, most of the data that we handle does not fall under these requirements. In fact, I’d wager that 90% of the emails, electronic documents, or paper documents we keep, we do because we are implementing the “CYA” rule.

Folks who drink the super search kool-aid will say that the cost of saving and searching data is becoming increasingly trivial, so why spend any time at all trying to weed the collection?  Rather, save it all and then try Filtering on the Way Out.  On the other hand, look at the search engine so many of us envy — Google.  It indexes and searches enormous amounts of data, but even Google doesn’t try to do it all.  Google doesn’t tackle the Deep Web.

So why are we trying to do it all?


For information on searching the Deep Web:

From my deep stores of anti-clutter information:

[Photo Credit:  Jason Rust]


What Do Your Searches Disclose About Your Work?

When you use a search engine, you’re thinking aloud.  It’s almost as if you’re standing in the middle of Central Park or Hyde Park shouting, “Does anyone know anything about [X]?” In Central Park, at least, people are likely to ignore you and just keep walking.  What would happen, however, if someone stopped, paid attention, and made a note of your request?  And, what if they then noticed that other people were asking about [X] and that these people worked with you?  Could that someone reach the conclusion that you and your colleagues are interested in [X]? Now, what if X=Initial Public Offering, or X=Merger, or (more likely these days) X=Bankruptcy? What would that attentive person know about you and your work?

I’ve heard reports of investment bankers and lawyers around the world beginning their research assignments on popular internet search engines.  What if someone noticed that lots of people at a particular firm were interested in Company [Y] and the topic “initial public offering.”  That’s normally the kind of information that is considered highly confidential within a company, an investment bank or a law firm.  However, do our searches on public search engines, social media sites or commercial subscription databases reveal information to a careful observer that we don’t intend to disclose?  What could that observer do with that information?

Do you remember when Amazon reported on which books seemed popular in certain organizations?  (E.g., we’ve been selling lots of books on bankruptcy to people at the ACME Company.)   It is possible for providers of public search engines or commercial databases to gather this data and make sense of it.  Is anyone doing that now with respect to our searches outside the firewall?  Should we be doing something about that?

[Photo Credit:  pavel1998]


The KM Solution?

In a recent meeting,  a vendor said with great enthusiasm, “Let me show you our KM solution.”  For a brief moment of intense joy, I actually thought I was about to experience KM enlightenment.

I should have known better.

After a bit of fanfare, he unveiled … a search engine.  Admittedly, it appeared to be a very fine search engine.  Nonetheless, if search and retrieval were the entire KM solution, most of us engaged in law firm knowledge management could have retired years ago.  The reality is that while good search and retrieval are important components of a law firm knowledge management program, they cannot fairly be described as the complete answer.

I know that and you know that.  When will the vendors figure it out?

[Photo Credit:  Sharon Pazner]


KM in an Era of Information Snacking

In a recent letter to his customers about the eBook reader Kindle, Amazon’s Jeff Bezos discussed the current tendency to engage in “information snacking”:

We humans co-evolve with our tools. We change our tools, and then our tools change us. Writing, invented thousands of years ago, is a grand whopper of a tool, and I have no doubt that it changed us dramatically. Five hundred years ago, Gutenberg’s invention led to a significant step-change in the cost of books. Physical books ushered in a new way of collaborating and learning. Lately, networked tools such as desktop computers, laptops, cell phones and PDAs have changed us too. They’ve shifted us more toward information snacking, and I would argue toward shorter attention spans. I value my BlackBerry—I’m convinced it makes me more productive—but I don’t want to read a three-hundred-page document on it. Nor do I want to read something hundreds of pages long on my desktop computer or my laptop. As I’ve already mentioned in this letter, people do more of what’s convenient and friction-free. If our tools make information snacking easier, we’ll shift more toward information snacking and away from long-form reading. Kindle is purpose-built for long-form reading. We hope Kindle and its successors may gradually and incrementally move us over years into a world with longer spans of attention, providing a counterbalance to the recent proliferation of info-snacking tools. 

As a reader, I’m supportive of Bezos’ attempt to give reading the advantages of the electronic age, but I can’t help wondering if he is entirely right about the impact of tools on how we read.  Perhaps it isn’t just the tools that nudge us away from long-form reading and towards information snacking.  Could information snacking in fact be a by-product of the pace of modern life?  Or does the abundance of available information at our fingertips force us to find faster, more efficient ways of retrieving and reviewing what we need?

Wikipedia’s article on Information Foraging, which appears to draw on the work of Peter Pirolli and Jakob Nielsen, notes that once users have been trained by search engines to go to a variety of sites to gather information, they will be reluctant to invest much time on a single site. Instead they make quick strategic raids to collect specific information and then leave the site.

If information snacking and short attention spans are an unavoidable part of the knowledge terrain of the 21st century, this leads to some interesting questions for law firm knowledge management:

– Are we providing lawyers with tools that encourage information snacking over long-form reading and analysis? (Bloggers and Twitterers may want to take the Fifth on this one.)
– Are there risk management issues arising from the increasing tendency to engage in information snacking?
– If information snacking by lawyers is a bad thing, how do we counteract it?
– If information snacking is inevitable, how do we adjust our KM systems to accommodate it?

As we deploy increasingly more powerful search engines within our firms, we give lawyers the ability to find more information with Google-like ease.  And that information is fragmented, providing a perfect opportunity for information snacking.  Perhaps the most important thing knowledge management systems can do now is to ensure they provide adequate context for and connection among these fragments so that we diminish some of the negative side effects that result from information snacking.  


Finding the Right Stuff

The best search engine in the world cannot convert garbage into useful content.

So, before you spend megabucks on the latest cool search tool, think about what repository you’re searching. If your office is anything like most offices, you’ve got tons of ephemera — stuff that probably isn’t going to matter 30 minutes from now — and tons of materials that really ought not to see the light of day. However, a decent search engine has the effect of shining a klieg light on the shortcomings of your content.

In fairness, a great search engine can help filter out much of the suboptimal content, but if there isn’t anything worth finding, what have you really achieved?

One solution is to stack the deck: make sure you put a goodly amount of useful content into your repository before you turn the search engine on. And what constitutes “useful content”? Not just something that someone might possibly need someday. (That’s the sort of rationale the turns a document repository into a cluttered and confusing packrat hell.) “Useful content” is content that is vetted: someone has reviewed it and given it a seal of approval or it’s been used with good results enough times for you to know it’s a great resource.

With that kind of content, you’re going to look like a genius every time the search engine does its thing. How often can you say that about a KM project?

For more information on the benefits of vetted content, see my article “Loading the Deck” in ILTA’s 2007 Knowledge Management White Paper.


What are People Searching For?

What are people searching for and where are they looking? That’s the question asked and answered in a thought-provoking article in the March 2008 issue of KMWorld. While working with an admittedly small sample, the survey yielded some interesting findings:

– 62% of respondents said that they first search the Internet before searching more specialized resources such as their own company’s website or intranet
– while 13% of the time the respondents said they were searching for information about their own companies, they began their search on their company intranet only 2% of the time
– respondents tended to ask their colleagues for help before they tried their company intranet
– business users spend a lot of time searching for information at work (approximately 9.5 hours per week)
– knowledge workers tend to search using general indices like Google and Yahoo rather than specialized web sites or search engines

The picture that emerges is troubling:

– companies aren’t doing a good job of making their intranets the first choice for company information
– despite the hours spent searching, many knowledge workers are not searching efficiently
– knowledge workers don’t seem to understand the inherent weakness of general web search engines like Google and Yahoo when it comes to finding specialized, high-value content
– searchers tend not to use content aggregators, specialized vertical search sites or topical sites to find data

For knowledge management, these findings pose some real challenges. In many companies, it’s the knowledge management group that’s responsible for the intranet. The findings of this survey are a real indictment of the job we’re doing. So what must we do differently to make our intranets the first choice research resource for our colleagues? It might be worth asking them.

And while we’re talking with them, we should investigate why it is they are using sub-optimal search methods. Is it a lack of awareness about how search engines like Google and Yahoo work? Do they simply not understand that high-value content can get buried in the Web, but will tend to be more visible on specialized web sites? According to the author of this article, people in the online industry know that “the `good stuff’ gets hidden if it is thrown into the larger web grab bag. And very often, it isn’t even in the grab bag because it isn’t indexed.” Clearly the average knowledge worker doesn’t know this or they wouldn’t be using the grab bag search engine.

Despite this (or perhaps because of this), the author notes that there are some signs of progress in the growing recognition of the value of finding high-quality information rather than merely relevant information. As a result, there is renewed interest in recommendation engines, contextual search and vertical search sites. These are “tools that will tell [knowledge workers] what they need to pay attention to in the pile” of information they face. In this age of overload, this sounds like a step in the right direction.