The moderator is Tanisha Little (Senior Manager Knowledge Management, Morrison & Foerster), and the speakers are Rizzi (Directors, Knowledge Delivery, White & Case), and Michael Williams (Director of LIfecycle Information Management, eSentio Technologies).
[These are my notes from the International Legal Technology Association’s 2012 Conference 2012. Since I’m publishing them as soon as possible after the end of a session, they may contain the occasional typographical or grammatical error. Please excuse those. To the extent I’ve made any editorial comments, I’ve shown those in brackets.]
NOTES:
- Why Focus on Search Analytics?. To report on the positive behaviors, but also to understand the negative outcomes such as which lateral partner hasn’t been integrated, which lawyer is a confirmed luddite, etc. Michael Williams reminded us that the analytics help you understand how the search engine is being used and how it needs to be modified to be more useful.
- Why should firms invest in analytics?. Ennio Rizzo pointed out that nothing works immediately, most systems need to mature. Analytics can help show trends and behavioral information that add credibility to your claims of success. Michael Williams says that enterprise search that is poorly implemented or poorly tended will result in a poor search experience. However, by using analytics, you can learn quickly about the user experience and how to improve it. This will avoid people turning away because they have been disappointed by their results. Therefore, analytics are a great way of protecting and exploiting your investment in an enterprise search system.
- What are all the uses of analytics?. There are four key uses of analytics: utilization, demographics, stability of performance and user experience. Depending on where you are in your project, you can weight one more than the other or focus on one in particular.
- What kinds of information should you be collecting?. Ennio Rizzo showed us some of the information they are collecting at White & Case. For example, they look at search penetration by practice group and title weekly to understand where they need to increase training or provide additional support. They look at who the top users are and then decide whether those people should be recruited as advisors to the enterprise search project. They also produce a heat map regularly to show how, where and when the search engine is used and whether they need to provide additional support. Finally, they can look at practices and industries that most interest specific individuals. This helps identify the expertise of those people. Ennio noted that enterprise search is a big, expensive investment by any firm, so information like this helps the firm improve its return on investment.
- When to deploy analytics?. Ennio Rizzi strongly suggests that you shouldn’t go to production before you have the analytics in place. Michael Williams agrees that having the analytics ready helps the engineers understand the guts of the tool. He also believes that the analytics that come bundled with the tool are not sufficient. Therefore, most firms need to purchase an additional analytics tool from another vendor. White & Case was able to use easily available web parts to extract the data from the analytics native in Recommind and then presented it via SharePoint. Ennio acknowledged that while the native analytics measured tons of elements, the real art was to extract the relevant information and present in an actionable form. Michael Williams showed an eSentio/Recommind dashboard that could send an alert to the law firm knowledge management group letting them know when a lawyer has a failed search so the KM group can contact that lawyer and help with the search. While this may feel a little creepy or Big Brother-like, Ennio believed that most lawyers appreciated the assistance.
- Dealing with Irrelevant Results. You should use analytics to help understand why we are getting back too many results or irrelevant results.
- What should you do before you deploy?. Spend a huge amount of time educating firm leadership and KM champions on the pros and cons of the search engine as built. This will help them come on board and provide you and your project “air cover.” All of this is part of the change management exercise you will have to do when you bring a tool as powerful [and disruptive] as enterprise search to your firm.
- Relevancy. Michael warned the audience not to fiddle with relevancy too much. If you are not careful, you can quite quickly degrade the search results. That said, judicious tweaking can be very helpful. Just be sure to proceed very carefully.
- Analytics is all about trending. You can show a change in usage. As long as you can continue to show good trends, that should help to make your case with management. Ennio also suggested using the analytics to find out whether there are resources people are searching for, but don’t exist. The KM team can then approach the practice group and offer assistance to create the missing resource.
Above and Beyond KM 