[Ed. Note: Please welcome guest blogger, Ravi Soni, data scientist from Casetext. I was introduced to Ravi by Casetext’s Vice-President, Pablo Arredondo, and asked to publish Ravi’s discussion on how he uses analytics at Casetext to determine if “the holding in a case is more procedural or more substantive,” and how to leverage that information to potentially predict outcomes. – GL]

___________________

One of the biggest constraints to innovation in legal research is how hard it is to scalably classify and quantify information without significant human intervention. At Casetext we’ve made real progress using advanced analytics to better leverage the wealth of content within the law to predict certain outcomes with more precision. The applications for this can range from anything between practice management, case strategy, or in my case, legal research. There is one such challenge I’m particularly interested in, namely, how to quantifiably determine whether the holding in a case is more procedural or more substantive.

I started with a collection of 47,464 briefs written by top law firms in the country. Using the citations and nature of suit (NOS) code associated with each brief, I was able to determine how many unique NOS codes were associated with each case. I defined this as how “polytopic” a case is. In other words, I counted all the unique NOS codes from the briefs that cited to each case and assigned that number as the polytopic score for each case. Ultimately, my goal was to use polytopicness as a proxy to measure proceduralness.

The idea behind using polytopicness to measure proceduralness comes from a simple concept. Let’s say we have a lawyer at an AmLaw 50 firm working on a massive M&A, a public defender in a small county appealing a death penalty verdict, and a boutique immigration firm working on a deportation case, and they all cite to the same case. What does this case have that all three of these attorneys found useful? The short answer is probably nothing substantive. What is more likely is that they are all citing to this case because it is a foundational case that sets the framework for some common motion that transcends practice area.

Let’s look at a concrete example. If I ask a roomful of lawyers if they know about A to Z Maintenance Corp. v. Dole 710 F. Supp. 853 (D.D.C. 1989), it’s quite unlikely that any of them would be able to tell me much, or anything at all. If I asked about a case like Bell Atl. Corp. v. Twombly 550 U.S. 544 (2007), any attorney in the room should be able to tell me how it changed the standards for dismissal. Looking at Figure 1, we can see how there is a difference in citation count and polytopic score between these two procedurally distinct cases.

In this example comparing these two metrics clearly show a difference between the procedural and substantive case – but does this hold for all cases in the data set? 
To find the answer, I first looked at the average number of citations per distinct polytopic score, as seen in Figure 2. To clarify what that means, I’ll use the point at roughly (50, 2500) as an example. This point can be translated to the following: Cases that have a polytopic score of 50 will on average be cited for a total of 2500 times in the briefs data set. The fact that it is a positive slope is intuitive and somewhat trivial; since a case that has a polytopic score of 5, must have been cited at least 5 times. The interesting piece here is the exponential growth, which means that proportionally, the cases that have a higher polytopic score will have a higher citation count. This finding was the first bit of evidence used to confirm our initial assumptions.

Next, I wanted to see what the distribution of polytopic scores look like in order to better understand how many cases are monotopic, bi-topic, etc. To do this, I aggregated the count of cases based on polytopic score (see Figure 3). Easily we can see that most cases in our brief data set are mono or bi-topic. However, when looking closer at the NOS codes (there are 102 in total) it seemed like some of the NOS codes could have been clustered together to make larger groups. For instance, there were codes like Personal Injury: Other, Personal Injury: Marine, Personal Injury: Automotive, etc. that could have been grouped together to make our groups more distinct from one another. As such, after grouping it seemed like any case that is associated with a polytopic score of 6 or more could be considered more procedural.
Although looking at polytopic score is useful, there are some corner cases where this metric would fail in measuring proceduralness. For instance, if a case has a polytopic score of 7, and it has only been cited 7 times ever, then to say it is procedural may not be correct. This is due to the fact that such a small number of citations may not be enough to give us an accurate polytopic score. As such, we need to account for how often cases are cited and adjust the polytopic score accordingly. Looking to Figure 4 we can see the overall distribution of case citations to better understand how often cases are cited. Figure 4 specifically looks at cases that have been cited at least once. 
Here, we can see that roughly half of all cited cases are cited less than 20 times. (In the same light, of the 8.99 million total cases that make up the common law, 5.65 million or about 63% have never been cited at all.) Using this citation information and the polytopic score for each case, I was able to distill an updated polytopic score that accounted for the number of times a case is cited. 
With the help of lawyers, I was able to manually go through 10% of cases that were most procedural and 10% of cases that were most substantive based on our polytopic scoring. I used this to determine whether or not this measurement was accurate in determining if a case is procedural or not. Overall, our assumptions were verified and we can say with some confidence that using polytopicness is a reliable measure of proceduralness for a case. For reference, here are the 10 cases that were shown to be the most procedural. 

ASHCROFT V. IQBAL 556 U.S. 662 (2009)
BELL ATL. CORP. V. TWOMBLY 550 U.S. 544 (2007)
CELOTEX CORP. V. CATRETT 477 U.S. 317 (1986)
ANDERSON V. LIBERTY LOBBY, INC 477 U.S. 242 (1986)
MATSUSHITA ELEC. INDUSTRIAL CO. V. ZENITH RADIO 475 U.S. 574 (1986)
LUJAN V. DEFENDERS OF WILDLIFE 504 U.S. 555 (1992)
CONLEY V. GIBSON 355 U.S. 41 (1957)
DAUBERT V. MERRELL DOW PHARMACEUTICALS, INC 509 U.S. 579 (1993)
KOKKONEN V. GUARDIAN LIFE INS. CO. OF AMER 511 U.S. 375 (1994)
FOMAN V. DAVIS 371 U.S. 178 (1962)

While this analysis has shown a strong relationship between polytopicness and procedurality, there is still some fine tuning needed to address the small subset of corner cases. The next step in continuing forward with this would be to see how clustering of NOS codes could be used to further refine the polytopic score. In the same light, this analysis has also opened up different avenues to explore. Some of which include, looking at different relationships between a brief and the cases they cite, how citation counts for cases differ in briefs and court opinions, or if we can predict what a case is about using substantive citations in the case documents. 
If you have any questions, comments, or concerns, please feel free to send me an email at ravi@casetext.com.

_________________________________________
Ravi Soni is a recent University of California, Berkeley graduate with a degree in Applied Mathematics. He is currently working as a Data Scientist at Casetext Inc., a legal technology company using AI to enhance legal research. Prior to joining Casetext, Ravi spent some time at other legal technology companies and worked as a legal assistant at a boutique IP firm where he focused on trademarks.

[Ed. Note: Please welcome back guest blogger, Marcia Burris, Research & Information Services Consultant for HBR. – GL]

A lot of attention has been given lately to the trend of law firms cancelling subscriptions to expensive online resources. This is often referred to as going “Sole Provider” since it has long been assumed (for a few decades, at least) that “good” law firms subscribe to both of the Big Two legal research providers, Lexis and Westlaw. In recent years however, many firms have decided they no longer need both. In an effort to measure the trend, law library surveys, including the one administered by HBR Consulting, routinely ask about whether firms are planning to (or already have) cut Westlaw or Lexis. However, while the term Sole Provider is easy to say and generally understood in the law library community as cancelling one or the other of these two services, it really isn’t the best way to describe current practices in the world of legal information, and in fact can cause harm to the conversation. So here’s why Sole Provider isn’t really a thing, and why I’m not going to say it any more.

  1. First of all, it isn’t true. Certainly not in Big or Medium Law, and probably not even in the vast majority of Small Law. No firm uses only one source for all its legal research needs. Cancelling one of the historical duopoly providers doesn’t mean attorneys will be limited to just one single source for all their legal research questions (although some attorneys may by choice return to the same well over and over again.) Law firms will continue to offer a variety of information resources – and formats – to meet their attorneys’ practice needs.
  2. Using the term “Sole Provider” needlessly reinforces the expectation of legal research Duopoly by implying that firms are choosing one and cancelling one, and fails to adequately describe the variety of different choices firms are making today. In doing so, it devalues the contributions of numerous providers beyond the traditional duopoly, whose innovations are creating new ways to think about and use legal information. This can cause real harm, as holding to the outdated duopoly concept hamstrings the decision process, limiting creative thinking about what resources firms should be offering to their attorneys and distracting from important discussions about new opportunities to evolve and modernize research services.
  3. In addition to reinforcing the idea of duopoly, the Sole Provider concept is often associated with cost reduction efforts, and this creates a value judgment which critics can leverage against firms (and librarians), no matter which way they go with the decision. Firms which keep both traditional major providers can be criticized for overspending, while firms which cancel a major service are criticized for prioritizing cost reduction over efficiency and service. (This reminds me of the working mom vs stay-at-home mom controversy – truly a debate with no winners.) Just as the duopoly concept narrows thinking about options beyond the Big Two, the question of Cut vs Keep limits the discussion to an either/or which fails to address the nuanced resource needs of individual firms, which ultimately drive their purchasing decisions.
  4. By referring to only a single facet of resource selection, the term devalues the important work law librarians do in carefully curating information collections to best meet their firms’ needs, and distracts from the question we should really be asking: What is the best mix of resources to meet our firm’s needs now and into the future?

It’s time to reframe the discussion. Instead of referring to “Sole Provider” decisions, let’s start talking about *Legal Research Optimization*. The discussion should include subpoints related to content (primary & secondary), efficiency of use and administration, attorney support, resource interrelatedness and content integration, cost, practice-specific needs, business needs, evolving technology, and client demands. Rather than allowing the status quo to set the tone of our discussions, let’s ask what should we include as we build the law library of the future for our firms. Firm needs and information resources continue to evolve, and libraries today have the opportunity to do more than ever before to support attorney practice needs. With the baggage of the sole provider conversation left behind, we can move forward and continue working to align information resources with firm needs, with freedom to explore the best fit for the future.

Definition of algorithm : 


noun al·go·rithm ˈal-gə-ˌri-thəm  – a step-by-step procedure for solving a problem or accomplishing some end especially by a computer

When I attended the WestPAC Law Librarian meeting in Jackson Hole, WY a couple of months ago, I had the opportunity to sit in on 
University of Colorado Law School’s Susan Nevelow Mart’s presentation on legal researcher’s reliance on algorithms for online legal research. Susan’s presentation discussed her SSRN Paper entitled “The Algorithm as a Human Artifact: Implications for Legal {Re}Search” where she breaks down the algorithmic affects of Westlaw, Lexis Advance, Fastcase, Google Scholar, Ravel Law, and Casetext. 
The key thing to remember, says Mart, is that we “need to remember that the algorithms that are returning results to them were designed by humans.” That includes all the “biases and assumptions” that come with the human experience. In other words a little bias and assumption on the part of the people developing the computer algorithms can cause dramatic changes in the results produced with similar content and search terms. As a researcher, Mart states that it is important that we “acquire some expertise about the technology at the meta-level.” How can you trust the results if you are not familiar with the way the tools are designed to index, search, and retrieve those results? The problem with this argument is that most legal research providers don’t want to reveal very much about the processes that go on behind the scenes to pull those top 10, 25, 50, or 1000 results. Mart is calling for more “Algorithmic Accountability” from our legal databases in order to help legal researchers better understand the biases present in the retrieved results.
Mart’s paper and research behind it attempt to test the different legal research databases on same search terms and same data content, and evaluate the results to see where results overlap and differ. The experiment wields results that are, in Mart’s words “a remarkable testament to the variability of human problem solving.” The top ten results from each resource showed very little consistency, and “hardly any overlap in the cases, and only about 7% of the cases returned were in all six database results. That low of a return rate should cause a bit of a shudder to run up the spine of legal researchers.
What is a researcher to do in this day and age of very little Algorithmic Accountability? First, researchers need to call upon these database providers to give us more detailed information about how their algorithms are set up, and the technical biases that result from these rules. Mart states that “the systems we use are black boxes,” that prevent us from understanding how these technical biases skew the results of our searches. “Algorithmic accountability will help researchers understand the best way to manipulate the input into the black box, and be more certain of the strengths and weaknesses of the output.”
Until we better understand the processes that go on in the background, researchers today should expand their searches, and use multiple databases in order to reduce the effects of technological bias. Mart explains that, “[t]he uniqueness of results may show something about the world view of each database that suggests that searching in multiple databases may be the 21st century version of making sure that multiple authorial viewpoints are highlighted in a library collection’s holdings.”
Within the SSRN paper, Susan Nevelow Mart presents the findings of her Empirical Study and breaks out the results by:
  • Uniqueness of Cases
  • Relevance
  • Relevant and Unique
  • Number of Results Returned by Each Query
  • Age of Cases
The different databases have individual strengths and weaknesses in each category, and the results, read as a whole, back up Mart’s suggestion of searching multiple databases. Until legal research providers begin to open up their black boxes and adopt more Algorithmic Accountability, researchers will need to expand our own legal information literacy with a better understanding of how each database compiles, categorizes, indexes, searches, and prioritizes the results. Hopefully, Mart’s research, and pressure from lawyers and researchers will help push these providers to shine a little more light into their algorithmic black boxes.

[ed. note – Updated at 11:30 CT to include Ravel Law as part of the databases reviewed by Susan Nevelow Mart. – GL]

As many of you that follow 3 Geeks know, I’m a big fan of the products that are coming out of Stanford University’s CODEX program. One of the latest insights comes from a CODEX fellow, Casetext, with their new CARA platform. Casetext’s VP of Legal Research, Pablo Arredondo, has been talking with me about CARA for a number of months now, and I’ve seen a few versions as he prepped CARA for release. If you’ve ever seen Pablo demo this new product, then you know how excited he is about the value that he believes this brings to legal research.

What is CARA? CARA is a ‘brief-as-query’ legal research tool, in which instead of using a keyword query you drop an entire brief in as the input. Users can input a brief in either Word or PDF format. From my use of it, I would explain CARA as a tool to analyse your brief (or the other side’s brief) to find potential missing points of law, or alternative arguments not cited within the brief.

CARA data mines the inputted brief and uses the gathered information to form a sort of ‘mega-query’ that runs against Casetext’s database of case law. CARA takes a look at the brief and analyzes how much cited cases are discussed within the brief as well as the other text within the brief. Arredondo explained to me that “[t]he analysis CARA runs looks not only at direct citation relationships (Case A cites to Case B) but also ‘soft citation’ relationships – Case A doesn’t cite Case B directly, but Case C cites to both A and B.” He also says that, “CARA also discounts heavily cited procedural cases like Celotex” so that these citations do not skew the results. Attorneys and researchers upload their drafts to check for missing cases before filing and uploading their briefs, and they check their opponent’s briefs to see if there are missing cases that they might be able to exploit.

Screenshot of CARA results.

CARA outputs a list of cases, all linked to the full text on Casetext. The results are displayed along side

  1. concise summary of case holding; 
  2. most cited passage from the case; 
  3. link to a law firm client alert discussing the case.  

Because attorneys are uploading drafts/work product, there is always a question of “who can see my draft?” Arredondo told me that “Casetext has applied stringent security to CARA.  Inputted briefs are not stored; once the data is extracted the brief is deleted. The reports CARA generates are accessible under a unique URL with a long hash; only those possessing the link can see the report.”

A geek like me loves to learn about how the back-end of these systems work, so bear with me for a paragraph as I repeat Pablo’s explanation of how CARA processes the brief that you drop in for analysis. CARA uses what is called topic modeling system (latent semantic analysis) to sort results based on how well they match the topics in the brief.In law-librarian terms, this means that CARA uses the full text of the brief and compares it to the full text of existing cases, looking for similarities in both legal terms and regular nouns. It’s not magic, it’s math. But, sometimes math can look like magic. Alright… end of uber-geek discussion.

The second serious question that most of us ask when we see products like this is “does it replace Westlaw or Lexis?” The answer is simply, no, and it is not meant to. CARA is designed to supplement traditional research systems. It can catch cases you missed using regular tools or help you find cases that you would have found anyway, only much faster. In these modern days of “efficiency” in legal practice, getting to the answer, faster, is a competitive advantage, and that is what CARA sets out to do.

Pablo mentioned that CARA will be made freely available to the judiciary, and that trials for firms or individuals are available for anyone who wishes to evaluate CARA. He said that there were no strings attached to the trial, and that you can contact him for details on the trial at pablo@casetext.com.

In his post the “Great Google Debate“, Mark Gediman suggested I was wise to not touch the debate on Google, and while I am happy to take the compliment, it also makes me wonder if somewhere down the road we (and by we, I mean those industry insiders, you know who you are) can’t create a Google equivalent to support the Legal industry. Imagine a single source that allows researchers to bridge the chasm between the business of law and the practice of law.  Let me explain.

On that same panel at AALL, I was asked where CI should report, my reaction drew a chuckle and was rapidly tweeted and retweeted. It was something to the effect of “I am tired of having this debate”. And I am, for a variety of reasons.  Where any of the research types or “information and analysis brokers” – Library, CI, KM, Research etc. – report is in my opinion, irrelevant and but an administrative imperative. How and where we add value to the firm and most importantly its bottom line/top line is what matters. I tweeted yesterday that information, intelligence, analysis when used effectively and systemically by firms could be the next disruptive factor, akin to the AFA.  Research, and the information professionals who undertake these tasks are embracing technology and are “to be congratulated for navigating really difficult times in the industry” according to Aric PressBig Law Is Here to Stay, and if its information professionals are going to continue to step up their game in this rapidly changing industry, they need proper tools, a collaborative environment and a checking of the proverbial egos (and related reporting structures) at the doors. 

Throughout the day, information professionals on the business side of the equation, search Google, subscription databases (what’s your favourite??), social media feeds, securities filings, traditional and  new media outlets and should be doing some kind of primary research i.e. talking to people and working the network (that’s a blog post for another time). On the practice side of the equation, legal researchers search corporate precedents, case law, filings, treaties, judgments, dockets, summaries, briefs, memos and other subscription databases. Imagine if you could put it all together, search one platform – a Googlesque type platform minus the paid SEO and get whatever research you needed in one place. How much more efficient, smart and focused on client and legal service could we and our firms be with one magnificent tool at our finger tips. 

Its pie in the sky, but that’s where dreams live, right? Here’s a use case. A proposed change in legislation relating to construction zoning in a particular jurisdiction is announced.  You  – Research Warrior/Maven/Guru  access the details of the proposed changes, and are able to fire it off to the relevant attorneys for an opinion, a LinkedIn Post, or a Client Update, while at the same time researching the number of public (and private, it’s a dream database, right?) companies in the jurisdiction who will be affected. You can also access which of those companies are your clients, your competitor clients, or prospects, and you can analyze the text of the proposed change to determine what the percentage of prior proposals with similar language were accepted, or rejected. With this data in hand, you can do a historiographic or timeline analysis to determine the likelihood of the proposal becoming law and using the same magic portal you can determine which other jurisdictions may adopt similar changes based on a cursory review of relevant local media and social media reactions and commentary.  And let’s not stop there, with a few clicks, you can output all the data into neatly branded reports complete with charts and graphs – a data visualization panacea. At that point, who really cares where you report? You just saved lawyers time, developed new leads, created an opportunity to demonstrate the firm’s value and demonstrated the information professional’s propensity for serial innovation.  Not bad in a day’s work!

Yes, there will be those that suggest it can’t be done, or those who won’t trust the data in a single platform even if it is pulling from multiple (triangulated and vetted) sources.  And course there will be a myriad of UX considerations, search/browse convergence discussions, taxonomy whoas and other finicky things to figure out.   But it would stop the where should we report and should we use Google debates….

I just returned from the AALL annual meeting in Philadelphia and had an interesting discussion with a colleague about Google.  First, let me set the scene: I was on a panel with Zena Applebaum and we had just answered a question about our favorite CI resources.  A member of the audience then asked why neither of us had included Google in our lists. As I began to answer, Zena wisely tweeted:

There is a debate going on, both within our institutions and in the research community.  Is Google a tool or a resource?  I feel that Google is just a tool, an excellent one that allows us to access a universe of information.  Unfortunately, the quality of the information is always in doubt.  Information from a fake website or a misleading post could be included in the search results, maybe even at the top of the list. The same reasons you don’t rely on Wikipedia apply even more to Google.  Google has never laid claim to delivering only quality, vetted information.  In fact, they have taken great pains to do the opposite.  Look at the disclaimer at the bottom of the page here and listen to the conversation Richard Leiter and Company had with Google Scholar’s Chief Engineer here.

As a researcher, I know the importance of confirming anything I find on Google and noting if the information is suspect and cannot be verified.  In CI as in law, it is important to have a high degree of confidence in the information that your analysis and recommendations are based on.  Google alone does not instill that confidence. 

There is a reason we pay for services like Lexis Advance and WestlawNext.  These services ensure that their subscribers have access to current and vetted content, often with editorial review.  I’m not saying Google isn’t useful.  I am on Google several hours each day.  However, it is for these reasons I don’t conduct legal research on Google when I have these services and others like them at my fingertips.  Just like any tool, a thorough understanding of its limitations is necessary to get the most out of it.

We all know this coming of age story. A boy leaves home to study abroad, sows his wild oats, and returns home a grown man, wiser and ready to take on the world. Except this coming of age story has a bit of a twist. The boy is actually a computer. And that computer’s name is Watson.
ROSS Intelligence, which is making headlines for its novel application of the IBM Watson machine learning platform to legal research, has been hard at work training the system to understand law. The team originally worked with Canadian legal content and lawyers, teaching Watson what “good” results looked like. But yesterday, the ROSS team announced they are bringing Watson back to the States to tackle US case law. They also announced support and funding from a powerful investor: Silicon Valley’s Y Combinator. ROSS is starting small with bankruptcy and, in a similar fashion to their original work north of the border, has partnered with a number of pilot law firms. But make no mistake, this first small step is likely to create tremendous ripples in the legal profession as their program expands.
I sat down recently with two of the co-founders of ROSS Intelligence, Andrew Arruda (CEO) and Jimoh Ovbiagele (CTO), to learn a bit more about ROSS and their experience with Watson.
One of the first topics was whether ROSS complemented or replaced the likes of LexisNexis and Westlaw. Arruda’s perspective was that it complemented traditional legal research for now, but the goal is ultimately to replace them. In reality, it is a bit of an apples and oranges comparison. Traditional legal research vendors generally provide data and a search box, leaving much of the heavy lifting to lawyers. This approach was well-suited to the “leave no stone unturned” philosophy that guided legal research in the golden age of law. ROSS, on the other hand, serves up insights based on a more natural dialogue between the lawyer and its Watson-based system. This approach fits better in a post-recession world where clients are cost-conscious and expect efficiency in their law firms.
For now, ROSS is still relatively targeted in its scope and utility. LexisNexis and Westlaw have massive stores of content they either own or license, and they have spent decades gathering and curating this content. Matching their breadth and depth of content will be a daunting task, to say the least. But the big vendors would be foolhardy to ignore this threat. Anyone familiar with Clayton Christensen’s The Innovator’s Dilemma and the concept of disruptive innovation knows that incumbents are often unseated when entrants perfect their technology downstream then move to compete directly. As Arruda says, “Think of us like the Netflix of legal research; we are going to keep adding capabilities and original content until lawyers no longer have a reason to stay with their traditional providers and can cut the cord.”
ROSS’s pilot approach is consistent with this notion. They start by turning associates loose, using ROSS just as they would other research tools (yes, Google and Wikipedia, you’re included in that list). As ROSS returns results, associates can provide feedback on whether ROSS’ answer was helpful. If it is not, the result is dropped and the next most relevant one is shown. This user feedback loop helps ROSS understand what is relevant for a particular topic.
As ROSS gets a more sophisticated understanding of an area of law, the pilot then moves upstream to senior associates and, ultimately, counsel and partners. This incremental approach to learning is a recurring theme in the world of deep learning, where AI systems learn in much the same way as children. In this instance, the ROSS team took Watson to law school and is now guiding ROSS through its first years at a law firm.
I asked Arruda and Ovbiagele about some of the challenges they faced adapting Watson to the legal profession. I have some familiarity in this space, having built several AI systems for LexisNexis back in the early 2000’s. One of the key issues is the structure of the typical legal document. If you break it down, much of the text in a brief or agreement is not really that important. It’s filler text or scaffolding where the real meat of the argument is hung. Take the heading of a court filing, for instance. It may say “In the 2nd district court of appeals,” but really all that matters is 2nd and appeals. All that extra text, like “in the court of,” or the “by and between parties” in an agreement, really don’t mean much. But to a system trying to extract and make sense of concepts, the extra text is a real problem.
Arruda and Ovbiagele confirmed they experienced the same issue. Much of their work has been tailoring and building an infrastructure around Watson to make legal text understandable. While some may cry foul at this level of intervention, that is the reality of where we are with AI. There is currently no “silver bullet” general purpose AI that is fully automated. But that does not stop the creation of targeted, specific-purpose AI like ROSS. And as has been shown in many other domains, that level of targeted AI is usually sufficient to disrupt an industry.
We also discussed how Watson is designed for a very specific type of question/answer interaction. Developers are constrained to a very specific formula of content ingestion, topic extraction, and tuning of relevant answers to questions. There are many other machine learning techniques out there – clustering, classification, prediction – that Watson does not do. ROSS, like many other Watson applications, layers their own special sauce on top of Watson to make results even more relevant and meaningful. “ROSS is a composite of AI technologies with Watson at its center, but we have a dedication to using the best methods available for this grand challenge,” explained Ovbiagele.
So what’s next for ROSS? With their move to the largest legal market in the world, it is clear they are setting their sights on broader application, both in terms of practice areas and law firm customers. But much remains to be seen. How fast will this occur? What will the business model and cost ultimately look like? How will other legal research providers like LexisNexis and Westlaw and intelligence system providers like Kira Systems react? And perhaps most importantly, will lawyers embrace help from a computer as it becomes more human-like?
One thing is clear. Disruption is coming to legal, as it has to so many other industries, and this time there is a feeling of inevitability. Lawyers and firms will have a choice: adapt, or perish.

++++++++++++++++++++++++++

Matt Coatney is an AI expert, data scientist, software developer, technology executive, author, and speaker. His mission is to improve how we interact with smart machines by making software smarter and teaching people how to work (and cope) with advanced technology. Great things happen when smart people and smart machines work together toward a common goal.
Follow Matt on LinkedIn and on Twitter @mattdcoatney. Follow the conversation at #BridgingTheAIGap.

[Ed. Note: Please welcome guest blogger Noah Waisberg, CEO of Kira Systems, and a good friend of the 3 Geeks. Noah was on an ILTA Panel with me last year, and will participate in the follow up to that panel this year called, Legal Technology Innovation: Bolstering AND Destroying Legal Work.  This post originally appeared on the Kira Inc. Blog. – RM]

[CC] Gengiskanhg

Watson is almost certainly the most significant technology ever to come to law, and it will give lawyers permission to think innovatively and open up the conversation about what is possible in a field that has been somewhat “stuck.” 

 –10 predictions about how IBM’s Watson will impact the legal profession“, Paul Lippe and Daniel Martin Katz, ABA Journal

IBM’s Watson AI has received a lot of attention for how it might change law practice. Should it? Or should commentators expecting “Watson” to change the world instead refocus their attention on “artificial intelligence” or “machine learning”?

Recently, legal market observer Ron Friedmann wrote, in a post on potential business models for Watson in the legal space:

if we aim to improve the efficiency of the legal market, there is no lack of technology to choose from. Whether Watson is the best place to bet remains an open question.

  1. What is Watson currently good it, and is it even the best avenue for automating tasks where it is strongest?
  2. Is Watson strong in the most promising areas for legal automation?
  3. Will Watson grow to become leading machine learning or AI technology across the board, or will it remain high quality only for question answering?

Watson, Currently

From Wikipedia: “Watson is an artificially intelligent computer system capable of answering questions posed in natural language”. Today, it appears to be a leading machine learning offering for question answering of a very specific sort (as we will cover below). IBM looks to be attempting to build Watson out into leading general purpose artificial intelligence software, but there is no consensus that it is at or better than the state of the art in areas beyond question answering. Indeed, early reviews on released Watson APIs have been underwhelming. As Ron Friedmann points out, it is not even clear whether Watson is a better technological approach for legal question answering tasks than, say, Neota Logic.

Apparently, one vendor is currently using Watson to extract data from contracts. I have yet to see any data suggesting that identifying contract provisions is in Watson’s sweet spot (by data, I mean information such as, say, published provision extraction accuracy numbers for a system built using Watson; the vendor claims their “goal today is to deliver a 20% cost reduction for a law firm in a typical diligence exercise”, which would not stand out relative to claims from other contract review software vendors (e.g., our clients tell us they find from 20–45% time savings on page-by-page review using our contract review software, and 60–90% time savings when they rely on it more heavily)).

Is Current Watson Right for Legal Problems?

Watson is currently a leading technology for question answering tasks. Are most legal tasks that could be impacted by software question answering tasks?

“Question answering” could be very broad, and most or all legal tasks could be interpreted as giving answers to questions. However, today Watson only stands out for performance on a narrow definition of question answering.

Each of the following questions illustrates a type of legal problem technology could help solve:

  1. Is it illegal for individuals to have ferrets in the state of California? (Watson-type question answering)
  2. Which of these 1 million documents are relevant to determining if anti-competitive behavior occured in this specific case? (eDiscovery technology-assisted review)
  3. How long does it tend to take for cases to get to trial in front of Judge Vernon Broderick? (Lex Machina)
  4. Who will the Supreme Court decide for in King v. Burwell? (Katz/Bommarito/Blackman algorithm)
  5. Can you draft a brief for us to submit to the court for this case? (NarrativeScience, Automated Insights (neither appear to be currently targeting legal))
  6. Do any expense items on this legal bill seem inappropriate? (SimpleLegal)
  7. Which of these contracts have change of control or exclusivity clauses? (us (Kira) and others)
Despite how all of these legal automation areas were phrased as questions, current Watson seems to only have documented high quality performance on tasks similar to the first, legal researchey issue. Is there any data to support the idea that Watson could best other currently-existing technology solutions on the other questions?
Alternatively, is being great at legal research question answering sufficient to make Watson the leading legal technology? Are all the other areas insignificant compared with an ability to know the law? As an ex-corporate lawyer, not to me. I know there are large numbers of lawyers that hardly ever do legal research.*

Will Future Watson Be Better Than Alternatives?

A Watson proponent might say maybe Watson is only truly great at question answering right now, but it will grow into the accross the board leading AI technology. After all, Watson won Jeopardy!, and IBM is pouring tons of resources into bettering it. As Friedmann states, after listing off a number of companies—including us—who build contract review software:

But as Paul points out, Watson’s R&D investment is probably 100x all these companies combined, and so has the potential to ride a much steeper performance curve.

Since machine learning does not yet have one approach that is better than others across the board, it is hard to say how much value Watson’s extensive R&D investment matters in the contract review software space (or, for that matter, in most other areas of technology, legal or otherwise). The argument that Watson will dominate outside its core because of overall R&D investment is akin to arguing that Lance Armstrong, straight off winning seven consecutive Tour de Frances, would win the 2006 New York Marathon. After all, he had incredible aerobic capacity, slow twitch muscle training, toughness, and more spent on his training than others in the field. Plus, running is basically just putting one foot in front of the other. For what it’s worth, Armstrong finished in 856th place his first try, and 232nd the next year.
Winning Jeopardy! is great, but there have been many other very impressive machine learning feats, including self driving cars; translation, including of live speech; and writing decent enough quality news articles that human reviewers could not necessarily tell the difference. Even on question answering, was Ken Jennings easy competition at Jeopardy! relative to a different current AI system like Google DeepMind?
Moving past Jeopardy!, IBM may be putting significant resources into Watson, but other companies are doing the same. Some equally large companies to IBM, including Google, HP, Facebook, and Baidu, are also putting a lot of resources into machine learning. Why will IBM beat them? Why will IBM even beat out newer AI focused startups such as DeepMind (bought by Google in 2014 for $650 Million), MetaMind, or many others? IBM itself appears to recoginize that others are building valuable machine learning technology, acquiring deep learning focused AlchemyAPI in May
Lots of companies are building AI technology for specific verticals (like us with contract review). Current machine learning is quite problem-specific, and these companies are getting experience honing their technology for their particular use cases. Will Watson’s technology really be better for specific verticals than companies focused on those specific verticals? Would you use Watson for eDiscovery ahead of offerings from companies who have been focused on that challenge for years? Will Watson do machine learning fraud detection better than well-funded Sift Science? Or movie recommendation better than Netflix? Will Watson even be better on legal research than something Thomson Reuters builds? IBM may be a lot bigger than TR, but TR is not small and has to nail this, whereas IBM does not need to get legal research right.
There are a lot of different legal tasks ripe for automation. There are also a lot of different technological approaches to solving AI problems. I suspect we’re a long way out from saying any one vendor’s technology is going to transform law practice on the whole.
* There is one other way Watson could transform law practice that I did not discuss here. Perhaps question answering lies behind as-yet-undiscovered-but-transformative legal applications. No doubt, there is a lot of opportunity to improve law practice through technology.

Image [cc] photologue_np

Over the past few years I have been less than impressed with the types of new research tools that have entered the legal market. Especially from the major players. In the past five years, all of the major vendors have re-vamped their flagship products, or have merged with other companies and have updated the interface, and the back end. This makes for a slicker look and feel and some enhancements on the user’s experience, but when you really break it down, it’s really just the same concepts with a few new features and (hopefully) better functionality. When I worked for the Oklahoma Supreme Court Network, way back between 1999 and 2002, I felt like the legal technology field was on the cusp of something really great. Thirteen years later, I feel like I’m still waiting on that greatness to actually arrive. It’s been over a decade of technologies just not quite reaching that threshold, but maybe my wait is finally over.

In the past week I’ve talked with a number of people that have come out of Stanford University’s CodeX, the short name for The Stanford Center for Legal Informatics program. It may be the first time in a decade that I’ve actually gotten excited enough about legal information technology that I thought I need to quit my job immediately and find a way to get involved in these start ups coming out of California. The ideas coming out of CodeX are actually novel concepts, rather than what we’ve seen for many years of simply repackaging old ideas into cheaper, better, easier, or more accessible platforms. CodeX is having a FutureLaw Conference this week, and I’m sorry that I’m not going to be there to see first hand what is the latest technology being incubated in CodeX.

I want to touch on three products, not as a full product review of those products, but rather just from the idea of how they are looking at things differently. All got their start through the Stanford program, and all have some truly unique and original concepts of how to pull relevant information from legal documents.

First up, Lex Mechina.

Lex Machina isn’t new on our radar. We did a bit of a review on this last year. The idea comes from what they call “Legal Analytics” of parsing large amounts of information about judges, lawyers, and other points of data regarding IP Litigation. The concept of analyzing the data to help “predict the behaviors and outcomes that different legal strategies will produce.” The most impressive review of Lex Machina came from an attorney that told me he was tired of getting beat by opposing counsel because they had this product. That is perhaps the best quote to ever hear from your attorneys when you are contemplating buying a new product. It’s hard to argue against.

Second is Ravel Law.

Jean O’Grady has reviewed and talked about Ravel Law, so there’s no need for me to rehash that here. As with many law librarians, sometimes we have to see with our own eyes before we actually “get it” when it comes to new products. I have to admit that happened to me with Ravel Law. I saw Ravel Law’s Co-Founder and CEO, Daniel Lewis, present alongside Fastcase’s Ed Walters at the ARK Group’s Law Library conference back in February, and have to say it was at this time that the “light went on” in my head that we were looking at a different approach. Information laid out in a readable and effective method, along with visual representations that allow a researcher to quickly spot the relevant information quickly and move in a non-linear method toward additional information. The Judge Analytics is one of the most interesting ideas I’ve seen in a while.  It was pretty amazing to watch it all unfold, and come to realize that they were definitely on to something with this product.

Finally, there is Casetext.

Just as with Ravel Law, I just didn’t immediately “get it” when it came to Casetext. However, after having a two and a half hour long call with Pablo Arredondo last week, I immediately became a fan. Just as with the other products, the information is compiled and displayed differently than we typical researchers are use to seeing. Heat maps and summaries and context and innovative citing methods are used to create visually stimulating and logical organization of the information all within the visible screen area. Add to this the ability for users to add in relevant information, upload briefs, and join communities, it just shows the potential of this platform and a truly novel approach at leveraging a community of legal researchers and practitioners.

Are We Seeing the First Steps Away From Keywords?

This is something I think I will come back and visit in later posts, but I wanted to touch on it here. It is my belief that in the next five to ten years we will no longer look at keywords as the primary way to research legal information. I think we are seeing the genesis of that concept here with these three products. In a way, we are looking at a high-level of compiling documents, information, topics, and insights through advanced algorithms or crowd sourced trends and actions. Think of it as the traditional digest system, only automated and always morphing as new information is added or the actions of individuals change throughout the research process. It is a fascinating idea to contemplate, and I really think that we are on the edge of a monumental change in how we typically “find the law” in legal research.

Content Is Still King

What I’m seeing with these product is that we are simply scratching the surface of what is coming next. Lex Machina is taking a tiny slice of the legal information world with its IP Litigation docket process. Ravel Law and Casetext are doing great things with a core set of case law. Imagine what would happen if these and other products start parsing larger amounts of data. No one seems to be touching statutes and regulatory information. Dockets are a logistical mess, but the potential is huge. News, law reviews, blogs, internal documents, state, federal, and foreign and international information are ripe for exploitation from these new thinkers. It will be interesting to see if there are ways that these powerhouses of idea generations will be able to team up with the mega information holders, whether that be governments or private holders, and really test the limits of how we conduct legal research in the future. I, for one, am excited to see what’s next.

Image [cc] Ana C.

It seems that Chief Justice John Roberts, Jr. is not a fan of law reviews. Back in 2011, Roberts joked that he found law reviews irrelevant, and found no need to know why there was any influence on 18th century Bulgaria by philosopher Immanuel Kant. In fact he went further and said “I would have to think very hard” in order to recall any recent law review articles he read, or found useful.

Ouch.

Let’s admit it, most of us outside the ivy covered walls of Academia rarely rush to the library to grab the latest law review before our peers in order to have a competitive advantage. To be fair, however, law reviews are kind of like archives. Most of the time, you never need anything from the archives, but when that need arises, you sure are glad it is there. Whether law reviews are relevant, or useful, or readable is beside the point here. What is interesting is that Roberts’ comment was actually about a real law review article, written by George Washington University Law Professor Orin Kerr. And, as any good writer will attest, always take advantage of the opportunity to turn one piece of writing into two pieces of writing.

Kerr is publishing an answer to Roberts’ comments in an upcoming law review article in The Green Bag. I’m actually looking forward to reading that one from this unconventional law review. One side note for The Green Bag…  please update that awful looking website. Just because your law review was created in 1997, doesn’t mean your website needs to look like it was created that same year.

Sorry. Got off topic.

Kerr got the idea of writing the response to Roberts when he was named a scholar-in-residence at the Law Library of Congress in 2012. In an interview to the National Law Journal (h/t to Rich Leiter), Kerr mentioned that the staff at the Library of Congress was amazing, stating that “They can find anything.” That without the help of the Library research staff, specifically Peter Roudik, the article couldn’t have been written. Then came the quote that I’m printing out, framing, and hanging on my office wall:

“The lesson of the article is that you can do anything with an amazing research librarian.”

As happy as I am to see this quote, I have to really as Orin Kerr one thing, “you’re just now figuring that out??” I bet there are some folks back at George Washington that would love to introduce you to the research librarians at the law library. Helping you find the obscure text from 1859 is something that many of us do on a regular basis. In addition, we can probably get it to you overnight (or within hours), without costing you a fortune. Law Librarians, and Researchers have connections, and those connections have connections. I hope for Kerr’s sake that he’s located the law library researchers back at GW when he got back from the Library of Congress.

So let this be a lesson to all of the Professors out there writing the next great Law Review Article. Go find the law library and introduce yourself to the research staff. Tell them what you’re working on, and make them a part of the team. They probably won’t be able to make your article any more appealing to Chief Justice Roberts, but they will definitely help you thoroughly research the topic.