Intelligent Search in Action

Speakers: Brendan Heck (Senior Manager, Knowledge Management Platform Modernization, Charles Schwab & Co., Inc.) and Gloria Burke (Director, Knowledge Management & Field Communications, Charles Schwab & Co., Inc.)

Session Description: Hear how one organization used intelligent search to enable frontline professionals to streamline access to knowledge and information reducing average handle time of client service requests and increase speed to proficiency for new hires.  Our speakers share the steps to determine a vendor, the path forward of implementation and successes since the launch.

[These are my notes from the KMWorld Connect 2021 Conference. I’m publishing them as soon as possible after the end of a session so they may contain the occasional typographical or grammatical error. Please excuse those. To the extent I’ve made any editorial comments, I’ve shown those in brackets.]

Notes:

• Engagements with two major consulting firms identified knowledge management as an area of need in their company. These firms were able to convince management of the need to invest further in KM.
• Intelligent Search is one part of a wider transformation of their KM capabilities. They are building a Knowledge Center of Excellence.
• Objectives of their Intelligent Search Project
  • improve time from search-to-click (68 seconds)
  • first call resolution
  • reduce average handle time
  • “Google like” search experience
• Vendor selection
  • Scope: internal, external (involving clients), or both?
  • Buy or build? Configure or customize?
    • if you are working in a core competency of your business, then build. If not, buy from a vendor who is committed to providing and maintaining a best-of-breed product.
  • What primary connectors to content sources do you need? (They cost extra!)
  • Calculate your ROI with your leadership
  • Start with search or with content? 
    • while they understood that the “garbage in garbage out” principle would suggest that they start with content, they decided to start with search on the theory that it would give them a better sense of their content landscape and needs.
  • They worked with Gartner and Forrester to find the best-of-breed vendors
    • they identified several key evaluation areas they would use to choose a vendor and then created a questionnaire to send to vendors
    • they used spider charts to graph the questionnaire responses from candidate vendors. Interestingly, they decided not to go with the vendor that seemed to have the most well-rounded responses because they did not seem to be as good a fit with Schwab. 
  • They narrowed the list of candidate vendors to the top two and then ran a 3-month proof of concept.
  • They created a system usability scale to score both vendors.
  • They chose Coveo and their intelligent search
• Implementation
  • Setup (system access, identifying content source, connecting to content sources, crawling and indexing those sources)
  • Limited scope of initial rollout to their Service Teams because that would give the best results for clients and best return for the company
  • Think about bringing search to where your people are working — this reduces the need to swivel from where they are working to a separate search system.
  • How can search work better for you? Are there data in other systems that can enable search results before a user does anything? (They did this by folding in data from their customer call and response system.)
  • They took 6 month to implement and release to production (but believe it could be done in as little as 3 months if all coordinated systems were ready to go.)
  • They ran the user interface through their UX lab to make sure what they designed would work well for their users.
  • Beta testing with a subset of users in prduction
  • They helped the system by adding specific “featured results” and synonyms specific to their business.
  • According to Coveo, approximately 30K interactions are needed before machine learning will kick-in properly. However, Schwab saw benefits far earlier.
  • They established new baselines so that they could collect and check data periodically to make sure they were heading in the right direction.
• Results
  • Reduced average handle time by 11.25% during the beta test (as compared to the non-beta users who were doing similar searches outside the intelligent search system.)
  • Their initial average click rank baseline was 1.8. This means that the correct answer the user was looking for was within the top 2 search results. (Their goal was top 3.) Since general release, the click rank has improved to 1.6.
  • They reduced search-to-click time by 71% initially. Now the actual reduction is 84%, which translates to real savings of time and money for the company.
  • Adoption: currently they have 47% adoption within their initial audience. They will be broaden access and expect their adoption rates to get higher then.
• What’s next?
  • They are remediating content and meta data tagging, both working in concert with the new search capabilities
  • They plan a scaled expansion to the enterprise
  • Implementing Coveo Q&A Snippets: this is similar to the Google function in which they provide a short answer to the user’s query above the links to the responsive web pages.
  • A/B Testing to examine proposed improvements before they roll them out. This testing approach enables data-driven decision making regarding proposed enhancements
  • Refinement
• Things to Remember
  • Objectives — know where you are headed
  • Be clear about your ROI — and be able to articulate it for your leadership
  • Choose a vendor that fits your needs and will help you meet your objectives
  • Eat the whale one bite at the time — you can’t to the whole project at once, so do it in sensible phases
  • Measure your results — “You can’t manage what you don’t measure.”