I’ve previously noted that keyword searching has seen better days. Numerous recent e-discovery court cases and KM blogs provide worthy critiques of keyword searching as an inadequate way of retrieving knowledge.
Recently I have taken a more in-depth view of the next-generation of search, known as concept, natural language or semantic search. The basic goal or purpose of semantic search, is creating structure out of chaos – thus the title of this post.
In the e-discovery world the challenge is finding the right knowledge within massive data stores. A lawyer can only present a relatively finite set of information at trial. So the trick is getting to the best information cost effectively. Given terabyte and larger discovery challenges, this is no small feat. An effective semantic search tool is the next answer to this question.
At its most basic level, semantic search tools are able to analyze human language and make sense out of it (like a human does). For instance, they can determine what a period is in a sentence versus a decimal point. For us humans this is a simple task. But this is well beyond a keyword search engine’s ability.
At this point in my thinking, I am crossing from one paradigm into another. For so long we have been focused on extracting structured data from data bases and making that knowledge universal. This has been the promise of XML. But the real magic comes when technology can find structure in unstructured data. This may sound very geeky, but it is definitely a real world problem.
For KM, it may be THE real problem. Most of the knowledge we seek to manage is buried in unstructured BLOBs. Tools that makes sense of this chaos and deliver the results in human-understandable ways will be very powerful.
More to come on this topic I’m sure, as my research continues.