This is the first in a 3-part blog post, it first appeared on The Sente Playbook. The other 2 posts are co-authored by Toby Brown and Greg Lambert and will follow later this week. Apologies for the length of this post, but I was channeling my inner Casey Flaherty.
The Big Idea: The data that Goldman used is insufficient to make the claims about Generative AI’s effect on legal that their report did.
Key Take-Aways:
- Reporting about this report is sloppy
- Reporting within this report is sloppy
- The underlying data doesn’t tell us much meaningful
- 3 Geeks attempts to find meaningful data
On March 26th, 2023 Goldman Sachs sent shockwaves through the legal industry by publishing a report claiming that 44% of “something” in the Legal Industry was going to be replaced by Generative AI. I didn’t question that stat at the time, because it sounded about right to me. I suspect that was true for most people who know the legal industry. As I’ve heard this stat repeated by multiple AI purveyors actively scaring lawyers into buying their products or services, I eventually started to question its validity.
I started by looking into the press coverage of that 44% number and was immediately confused. (All emphasis below added by me.)
Law.com – March 29, 2023
Generative AI Could Automate Almost Half of All Legal Tasks, Goldman Sachs Estimates
“Goldman Sachs estimated that generative AI could automate 44% of legal tasks in the U.S. “
Observer – March 30, 2023
Two-Thirds of Jobs Are at Risk: Goldman Sachs A.I. Study
“The investment bank’s economists estimate that 46% of administrative positions, 44% of legal positions, and 37% of engineering jobs could be replaced by artificial intelligence.
NY Times – April 10, 2023
A.I. Is Coming for Lawyers, Again
“Another research report, by economists at Goldman Sachs, estimated that 44 percent of legal work could be automated.”
Okay, so which is it? Generative AI is going to replace 44% of legal tasks, positions, or work?
Because those are 3 very different things; each of which would have extremely different impacts on the industry if they came to pass. Lest you think I cherry-picked three outlying articles, go ahead and Google “AI Replace 44% Legal Goldman Sachs” and see what you get. Those 3 articles are in my top 5 results.
My top result as of this writing is a news article from IBL News, writing last Tuesday that Goldman says, “AI could automate 46% of tasks in administrative jobs, 44% of legal jobs, and 37% of architecture and engineering professions.”
We should probably just go back to what the Goldman Sachs report actually said and then we can chalk this up to lazy tech journalism. Well, not so fast. Because while the Goldman researchers clearly say “current work tasks” (see below) even that begins to fall apart once you dig into the underlying data.
What Goldman Sachs actually said in the report
“…we estimate that one-fourth of current work tasks could be automated by AI in the US (Exhibit 5, top panel), with particularly high exposures in administrative (46%) and legal (44%) professions…” (emphasis added)
So, they are estimating that a quarter of all current work tasks in the US could be automated by AI… someday?
Let’s assume they mean sometime in the very near future, although the only reference to a timeframe is in the middle of page 5, “While much uncertainty remains around both the capability and adoption timeline of generative AI, these developments suggest that AI is well-positioned to advance rapidly and grow in scale in the coming years.” (Emphasis mine.)
Further confusing the issue, the Goldman researchers reference a chart (Exhibit 5) showing the “Share of Industry Employment Exposed to Automation by AI” in the US.
The Goldman researchers weren’t specifically targeting the legal industry with these findings. They were looking at potential for AI automation in ALL industries in the US and it just so happened that “Legal” was the second most vulnerable industry.
Except… does anyone else find it strange the the most vulnerable industry identified in this report is “Office and Administrative Support”, the fifth is “Business and Financial Operations”, and the seventh is “Management”? I’ve done my fair share of office and administrative support, business operations, and management over the years, but I’ve never considered myself a part of one of those “industries”.
Even Goldman is confusing the terms in their analysis. This chart would probably be better titled “Share of Professions Exposed to Automation by AI: US”. That would make a lot more sense, except of course the highlighted column should then be labeled “All Professions” not “All Industries”.
However, this clarification highlights another problem with this data. Exactly what jobs are included in the “legal profession”, or the legal industry for that matter? And how do we know that 44% of tasks those jobs perform are vulnerable to automation by AI?
What the underlying data tells us
Thankfully the Goldman researchers included a high-level description of their methodology so I could go back to the original data and figure out exactly where they got that 44% number and what it really means.
Unfortunately, their description is verbose while still somehow remaining cryptically vague. You can follow along at the bottom of page 5 in the report.
In particular, we use data from the O*NET database on the task content of over 900 occupations in the US … to estimate the share of total work exposed to labor-saving automation by AI by occupation and industry.
…we classify 13 work activities (out of 39 in the O*NET database) as exposed to AI automation, and in our base case assume that AI is capable of completing tasks up to a difficulty of 4 on the 7-point O*NET “level” scale (see Appendix for more details). We then take an importance- and complexity-weighted average of essential work tasks for each occupation and estimate the share of each occupation’s total workload that AI has the potential to replace.
Having dug into the underlying data, I can now begin to translate some of that:
- O*NET database – The Occupational Information Network, a Department of Labor sponsored database containing voluminous occupational information. This is where all of Goldman’s data comes from.
- 39 Work Activities – O*NET defines 39 work activities that apply to each job title they track (I count 41, but maybe Goldman dropped a few intentionally?)
- 13 Work Activities – Goldman identified 13 of the 39 work activities that are potentially vulnerable to AI automation, including:
-
- Getting Information
- Monitoring Processes, Materials, or Surroundings
- Identifying Objects, Actions, and Events
- Estimating the Quantifiable Characteristics of Products, Events, or Information
- Processing Information
- Evaluating Information to Determine Compliance with Standards
- Analyzing Data or Information
- Updating and Using Relevant Knowledge
- Scheduling Work and Activities
- Organizing, Planning, and Prioritizing Work
- Documenting/Recording Information
- Interpreting the Meaning of Information for Others
- Performing Administrative Activities
- “Level” Scale – O*NET gives each work activity a Level from 1 to 7. O*NET simply describes this as a “level”, but extrapolating from the Level Examples (called Anchors by O*NET), and published in the report Appendix on pg. 17, Goldman interprets the “Level” metric to mean “Difficulty”.
- Base Case – For the purposes of this study, Goldman assumes that Generative AI is Capable of performing work activities up to level 4. They suggest that findings may be significantly different, one way or another, depending on how much capability you attribute to AI.
- Weighted Average – In addition to a level, each work activity is given an “Importance” score. Goldman combines the importance- and complexity- (level?) scores of essential tasks to come up with a percentage of the workload for each occupation, that AI could potentially replace. They don’t provide that formula.
What Goldman Sachs does not say in the report
O*NET classifies Work Activities by individual Job Titles, not by industry or profession, so what should we consider “legal” in this report?
With a little digging, I discovered that O*NET provides a list of occupations that it considers a part of the
“Legal Job Family”
“Legal Job Family”
- Administrative Law Judges, Adjudicators, and Hearing Officers
- Arbitrators, Mediators, and Conciliators
- Judges, Magistrate Judges, and Magistrates
- Judicial Law Clerks
- Lawyers
- Legal Support Workers, All Other
- Paralegals and Legal Assistants
- Title Examiners, Abstractors, and Searchers
I found it odd that O*NET did not include “Legal Secretaries and Administrative Assistants” in the Legal Job Family, even though that is a title that they track. I soon discovered that they were listed in the Office and Administrative Support job family. Which sounded very familiar to me.
The O*NET Job Families (presented in a familiar order):
- Office and Administrative Support
- Legal
- Architecture and Engineering
- Life, Physical, and Social Science
- Business and Financial Operations
- Community and Social Service
- Management
- Sales and Related
- Computer and Mathematical
- Farming, Fishing, and Forestry
- Protective Service
- Healthcare Practitioners and Technical
- Educational Instruction and Library
- Healthcare Support
- Arts, Design, Entertainment, Sports, and Media
- Personal Care and Service
- Food Preparation and Serving Related
- Transportation and Material Moving
- Production
- Construction and Extraction
- Installation, Maintenance, and Repair
- Building and Grounds Cleaning and Maintenance
What Goldman Sachs confusingly calls “Industries” in Exhibit 5 is actually O*NET’s “job families”.
Level Classifications of Work Activities are not Integers
O*NET gathers data by surveying practitioners of each occupation. The surveyed individuals are presumably asked to rank each activity Level using integers 1 to 7 (7 being the most complex or difficult.) However, the values for the Level metric in the data contain decimals, likely because they are calculated as an average result of all respondents.
This means that Goldman’s statement, “AI is capable of completing tasks up to a difficulty of 4 on the 7-point O*NET ‘level’ scale” has a problem.
Do we take “up to 4” to mean 4.0 and below or anything less than 5.0?
Interestingly, it makes a big difference, because many of the Activities in the Legal Job Family are in the 4-5 range. Below is a table of the Levels of the 13 potentially exposed work activities as performed by lawyers according to the O*NET data.
| Work Activities | Level for Lawyers |
|---|---|
| Getting Information | 5.92 |
| Monitoring Processes, Materials, or Surroundings | 3.69 |
| Identifying Objects, Actions, and Events | 5.51 |
| Estimating the Quantifiable Characteristics of Products, Events, or Information | 2.83 |
| Processing Information | 5.09 |
| Evaluating Information to Determine Compliance with Standards | 5.86 |
| Analyzing Data or Information | 4.61 |
| Updating and Using Relevant Knowledge | 5.37 |
| Scheduling Work and Activities | 4.09 |
| Organizing, Planning, and Prioritizing Work | 5.43 |
| Documenting/Recording Information | 4.12 |
| Interpreting the Meaning of Information for Others | 4.26 |
| Performing Administrative Activities | 3.88 |
If we assume that “up to 4” means that AI can only perform activities 4.0 and below, then only 3 of Goldman’s 13 Lawyer Work Activities are exposed to AI Automation and 3 of 39 total activities (7.7%) are vulnerable.
If we assume that it instead means that AI can perform any activities below 5.0, then 7 of 13 exposed Lawyer Work Activities and 7 of 39 total Lawyer Activities (17.9%) are vulnerable.
Of course, we haven’t yet taken into account Goldman’s proprietary “Importance- and Complexity- Weighted Average”, nor have we factored in the other Jobs in the Legal Job Family, but if your concern was that 44% of Lawyers jobs are going away sometime soon, it’s not happening.
| Work Activities | Lawyer | Judicial Law Clerks | Administrative Law Judges, Adjudicators, and Hearing Officers | Arbitrators, Mediators, and Conciliators | Judges, Magistrate Judges, and Magistrates | Paralegals and Legal Assistants | Title Examiners, Abstractors, and Searchers |
|---|---|---|---|---|---|---|---|
| Getting Information | 5.92 | 5.69 | 5.2 | 5.87 | 6.05 | 4.76 | 5.41 |
| Monitoring Processes, Materials, or Surroundings | 3.69 | 2.24 | 1.74 | 2.13 | 3.06 | 3.95 | 4.56 |
| Identifying Objects, Actions, and Events | 5.51 | 4.89 | 4.9 | 4.3 | 5.27 | 4.23 | 5.1 |
| Estimating the Quantifiable Characteristics of Products, Events, or Information | 2.83 | 0.8 | 1.99 | 1.14 | 2.61 | 2.12 | 2.53 |
| Processing Information | 5.09 | 5.21 | 4.8 | 4.48 | 5.26 | 5 | 5.14 |
| Evaluating Information to Determine Compliance with Standards | 5.86 | 4.54 | 5.24 | 4.48 | 6.17 | 5.02 | 4.36 |
| Analyzing Data or Information | 4.61 | 5.14 | 4.43 | 4.7 | 4.34 | 4.29 | 4.69 |
| Updating and Using Relevant Knowledge | 5.37 | 4.32 | 5.28 | 5.04 | 5.48 | 5.01 | 4.09 |
| Scheduling Work and Activities | 4.09 | 1.13 | 3.34 | 3.27 | 4.53 | 3.88 | 2.69 |
| Organizing, Planning, and Prioritizing Work | 5.43 | 4.16 | 4.98 | 4.36 | 4.98 | 5.46 | 4.48 |
| Documenting/Recording Information | 4.12 | 2.52 | 3.79 | 4.13 | 4.25 | 4.21 | 4.92 |
| Interpreting the Meaning of Information for Others | 4.26 | 4.19 | 4.05 | 4.22 | 4.2 | 3.5 | 4 |
| Performing Administrative Activities | 3.88 | 1.71 | 4 | 3.22 | 4.23 | 4.68 | 2.56 |
We can break out all of the Legal Job Family the same way.
| Occupation | 4.0 or less | Between 4.01 and 4.99 | 5.0 or greater | % Tasks at Risk (Low) | % Tasks at Risk (High) |
|---|---|---|---|---|---|
| Lawyers | 3 | 4 | 6 | 7.7% | 17.9% |
| Judicial Law Clerks | 5 | 5 | 3 | 12.8% | 25.6% |
| Administrative Law Judges, Adjudicators, and Hearing Officers | 5 | 5 | 3 | 12.8% | 25.6% |
| Arbitrators, Mediators, and Conciliators | 4 | 7 | 2 | 10.3% | 28.2% |
| Judges, Magistrate Judges, and Magistrates | 2 | 6 | 5 | 5.1% | 20.5% |
| Paralegals and Legal Assistants | 4 | 5 | 4 | 10.3% | 23.1% |
| Title Examiners, Abstractors, and Searchers | 4 | 6 | 3 | 10.3% | 25.6% |
| Legal Support Workers, All Other | 0 | 0 | 0 | 0% | 0% |
|
Combined
|
27 | 38 | 26 | 8.7% | 20.8% |
Of course, this is just number of tasks, it doesn’t take into account Goldman’s Calculation of the Importance/Complexity weighted Average. And it doesn’t include anything about the “Legal Support Worker, All Other” category, that O*NET includes in the Legal Job Family, but does not provide any data for.
But let’s get to why absolutely none of this really matters…
Sample Size
The O*NET data provides the sample size for each job title. We know how many people provided information to them.
- Administrative Law Judges, Adjudicators, and Hearing Officers – 26
- Arbitrators, Mediators, and Conciliators – 23
- Judges, Magistrate Judges, and Magistrates – 19
- Judicial Law Clerks – 16
- Lawyers – 31
- Legal Support Workers, All Other – 0
- Paralegals and Legal Assistants – 34
- Title Examiners, Abstractors, and Searchers – 16
All of the information about the Work Activities in the Legal Job Family comes from 165 people in the legal industry.
Goldman is not a disinterested party
I have no direct insight into Goldman’s investment portfolio, but I have no doubt that they are heavily invested in Generative AI and are riding the current wave of interest to record gains. Publishing a report that gets lawyers all worked up that they are going to be replaced can only help drive those gains.
A better headline for this study might be the one I gave this article.
Putting it all together
So, let’s review, shall we?
- Goldman Sachs – an investment bank that likely stands to benefit substantially from greater hype surrounding the delivery of Generative AI –
- Published a study about the effect of AI on ALL Jobs in the US – based on surveys completed by thousands of people, but only 31 lawyers and 134 people in legal related jobs –
- That indicates that Generative AI could potentially replace 44% of “legal work activities” – which are at best vague and fuzzy “activities” and at worst kind of meaningless –
- As long as you factor in a proprietary importance/complexity weighted average – that Goldman does not share –
- At some point between now and the heat death of the universe – although, likely sooner.
- The press latched on to this and reported that Goldman Sachs says 44% of either legal tasks, jobs, or total legal work could be replaced by AI – depending on which account you read – and
- Lawyers and Legal Journalists all freaked out.
In that context, 44% doesn’t really seem quite so dire, does it?
In part 2 we find some better data from which to make a more informed calculation of the effect of Generative AI on law firms, in particular.
In part 3 we explore what that data means for law firm profitability.
This will be discussed further in a free LVN Webinar on August 15th.