by 3 Geeks (Ryan McClead, Greg Lambert, and Toby Brown)
This is part 2 in a 3 part series. The first part is here. Part 3 is here.
The Big Idea: We found a much better dataset, though still small, from which to extrapolate actual effects of Generative AI on the legal industry.
Key takeaways:
- We got anonymized and summarized data for 10 corporate legal departments from LexisNexis CounselLink
- The data showed that almost 40% of time entries, representing 47% of billings, could potentially use Generative AI.
- We estimate that a realistic initial upper limit for Generative AI would be to reduce that work by half, or 20% of time entries and 23.5% of revenue
In the previous post, Ryan got tired of hearing the Goldman Sachs “44% of Legal is going away” stat being quoted uncritically and decided to actually look into the underlying data used in their report. Ryan’s exploration of the data is an interesting story in and of itself, but the bottom line is that the data is fuzzy at best, the sample size is laughable, and the breathlessly unquestioning reporting on Goldman’s study has been remarkably sloppy.
After writing up his findings, Ryan shared that post with Greg and Toby, and the question quickly arose, “can we find some actual, useful data to better understand the effect that Generative AI might actually have on law firms?” Gregreached out to Kris Satkunas from LexisNexis CounselLink, a recent interviewee on the Geek in Review, to see if CounselLink could share some anonymized benchmark data for us to analyze.
LexisNexis CounselLink Data
As a reminder the Goldman data was using survey questions about how important certain “work tasks” were for their jobs. Those tasks included things like “Getting Information”, “Identifying Objects, Actions, and Events”, and “Scheduling Work and Activities”. These are quite vague and wide open to interpretation.
In an attempt to find more useful data for our purposes, we asked Kris for the percentages of all time entries that included the keywords “Draft” or “Review” in the description. Our assumption is that those two terms will capture a large percentage of actual time entries in which lawyers are likely to use Generative AI. We fully recognize that this simple heuristic will not produce a clean data set from which to extrapolate definitive results, but as a first pass at some real data, we believe this gives us a nice estimate of tasks that could potentially be ripe for automation with Generative AI.
Kris suggested that the best way to do this would be to look at law firms’ time entries for individual corporations who use CounselLink e-billing for all of their legal work. This gives us the percentage of time entries for a company’s entire legal portfolio that contain our keywords. She selected data from 10 CounselLink customers of differing sizes and representing a broad range of industries. We don’t know the companies or the law firms that they used, but we know that 7 of them had a material portion of their legal work performed by AmLaw 50 firms, and the remaining 3 had material work performed by AmLaw 100 firms.
This is still an extremely small data set, but we believe these numbers are significantly more representative of actual legal work than Goldman’s O*Net professions survey data.
| Keywords | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | Average |
|---|---|---|---|---|---|---|---|---|---|---|---|
| “DRAFT” | 13.57% | 16.07% | 9.31% | 15.28% | 12.7% | 12.91% | 13.87% | 12.64% | 13.5% | 12.18% | 13.02% |
| “REVIEW” and not “DRAFT” | 27.4% | 26.32% | 26.26% | 25.36% | 23.42% | 26.57% | 29.96% | 28.37% | 25.7% | 21.3% | 26.17% |
| Combined Total % | 40.98% | 42.39% | 35.57% | 40.64% | 36.11% | 39.48% | 43.82% | 41.01% | 39.19% | 33.48% | 39.19% |
As you can see above, “Draft” and “Review” time entries make up about 39% of all entries in our data set. Even if Generative AI could completely replace all of that work (which it can’t and won’t), less than 40% of legal “work tasks” (as Goldman considered them) could be automated using Generative AI. If you take an optimistic view, AI could potentially replace about half of the hours in those entries. In other words, a more realistic estimate would be that about 20% of legal “work tasks” could potentially be automated by Generative AI. That is not “nothing to see here”, in fact, it’s significant and impressive, but it’s not the horror headlines that so many publications ran back in March and April.
Follow the money
Of course, the number of time entries, isn’t really that interesting on its own, even though that is essentially the data that Goldman Sachs was actually reviewing. The real question, and the headline that should truly strike horror into the heart of every law firm partner is, “What is Generative AI going to do to my paycheck?”
Thankfully, Kris took the same data she provided above and represented it as a percentage of total fees billed.
| Keywords | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | Average |
|---|---|---|---|---|---|---|---|---|---|---|---|
| “DRAFT” | 19.19% | 19.33% | 19.05% | 24.01% | 19.48% | 15.88% | 27.54% | 18.73% | 17.19% | 19.72% | 19.77% |
| “REVIEW” and not “DRAFT” | 26.65% | 25.86% | 29.45% | 27.55% | 23.67% | 22.09% | 33.86% | 34.15% | 31.59% | 22.46% | 27.16% |
| Combined Total % | 45.84% | 45.19% | 48.49% | 51.56% | 43.15% | 37.97% | 61.4% | 52.88% | 48.78% | 42.18% | 46.93% |
While the number of “Draft” and “Review” entries made up 39% of all billed tasks, those same tasks make up 47% of all fees on average. Using our back of the napkin estimation from above, if Generative AI could potentially remove 50% of that work, it would result in a revenue reduction of about 23.5% to law firms in the very near future.
It was at this point that Toby pointed out that Generative AI is unlikely to hit law firm revenue evenly across all timekeepers. In fact, it’s much more likely to replace or reduce lower level legal work like that typically performed by associates.
I think we just heard many partners just breathed a huge sigh of relief, but they shouldn’t. Because if Generative AI is going to affect associate work more than their own, that means that firm profitability (i.e., the money the firm splits and hands to partners at the end of the year) is going to be substantially smaller.
But more about that in the next post.
In the next installment, we’ll look at how firm profitability is likely to be effected by Generative AI.
This will be discussed further in a free LVN Webinar on August 15th.