I’ve previously noted that keyword searching has seen better days. Numerous recent e-discovery court cases and KM blogs provide worthy critiques of keyword searching as an inadequate way of retrieving knowledge.

Recently I have taken a more in-depth view of the next-generation of search, known as concept, natural language or semantic search. The basic goal or purpose of semantic search, is creating structure out of chaos – thus the title of this post.

In the e-discovery world the challenge is finding the right knowledge within massive data stores. A lawyer can only present a relatively finite set of information at trial. So the trick is getting to the best information cost effectively. Given terabyte and larger discovery challenges, this is no small feat. An effective semantic search tool is the next answer to this question.

At its most basic level, semantic search tools are able to analyze human language and make sense out of it (like a human does). For instance, they can determine what a period is in a sentence versus a decimal point. For us humans this is a simple task. But this is well beyond a keyword search engine’s ability.

At this point in my thinking, I am crossing from one paradigm into another. For so long we have been focused on extracting structured data from data bases and making that knowledge universal. This has been the promise of XML. But the real magic comes when technology can find structure in unstructured data. This may sound very geeky, but it is definitely a real world problem.

For KM, it may be THE real problem. Most of the knowledge we seek to manage is buried in unstructured BLOBs. Tools that makes sense of this chaos and deliver the results in human-understandable ways will be very powerful.

More to come on this topic I’m sure, as my research continues.

  • Read your post with interest. You may want to take a look at the semantic search engine we are working on at http://www.cluuz.com. This engine’s strength is identifying the relationships between people and entities.

    We’ve had a number of inquiries with respect to its application in the eDiscovery space. We’d appreciate any feedback.



  • Michael,

    At first look at cluuz, it looks amazing. I’m going to start playing with it and maybe do a posting here on the geeklawblog later describing what all you can do with this type of semantic search engine.

  • Thank you for your interest and for sharing Cluuz. That may well have an e-discovery application. I can more directly see the KM enterprise search use, but I am biased in that direction. Is Cluuz a tool you can deploy within an enterprise?

  • Toby,
    You are absolutely correct – we do have a product which is Cluuz enterprise search that is exactly meant for searching and making sense of internal unstructured data sources be it for pursposes of eDiscovery or general KM. Moreover, Cluuz Enterprise provides ability to reconcile internal and web search results into an easy to understand semantic graph which gives users complete information picture across all the sources that they have access too.