I am disappointed every time I guest lecture a law school class.

Because anecdote is often more compelling than data, I’ll start with an example from two weeks ago. An adjunct professor who teaches one of those great law school classes with cool titles like Tomorrow’s Lawyer had his students take the Word module of my Legal Tech Assessment. They performed exactly as you (well, I) would expect of an untrained group: poorly. Their anonymized scores are below:

I’ll explain the scoring in a subsequent post. But, for now, just focus on the accuracy. On average, students were able to correctly complete less than a third of the following tasks in a live Word document:

• Accept/Turn-off changes and comments
• Cut & Paste
• Replace text
• Format text (font, margin)
• Footers
• Insert hyperlink
• Apply/Modify style
• Insert/Update cross-references
• Insert page break
• Insert non-breaking space
• Clean document properties
• Create comparison document

To put it in anecdotal context (though these numbers are representative of the larger data set), around the same time the class was taking the assessment, a similarly-sized pilot group at a large law firm achieved an average accuracy of 68% while a group of lawyers and staff who had just been through LTA-specific training achieved an average accuracy of 95%. Training matters.

Training matters even for so-called digital natives like the law school students I’ve tested. Acquiring a Twitter account in utero does not engender natural facility with technological tools because most technological tools are not intuitive.

My wife thinks my sons are geniuses. One piece of evidence she submits in their favor is how well they use an iPad. I agree with her that the fact that my 1.5 year old can use an iPad is a testament to genius. But not his. For me, it is a testament to the genius of the designers at Apple who created a device so intuitive that 1.5 year old can use it. The kid touches a picture, it moves. Congratulations to him!

While we can discuss design principles that would move work software in the direction of our consumer experience, I don’t actually believe that individual apps are the best basis of comparison for complex business software. Rather than thinking of Word as an app, we should think of it as a bundle of apps. Each of the icons on the ribbon is a solution to a particular problem. The challenge is that there are so many icons (just as there are so many apps). While design principles can bridge some of the gap, there remains a tradeoff between depth and intuitiveness. Most of us therefore only become comfortable with a few functions while ignoring the rest (just as most of us use a limited number of mostly single-purpose apps):

In this, the younger generation is no different than their predecessors. Survival is the threshold they achieve with most of their technology. That their milieu demands facility with a few more social apps sooner does not change the fundamental fact that using technology properly is a collection of acquired skills, not some innate talent that Lamarckian evolution bestowed on those under the age of 30. Expecting them to automatically know how to use complicated technology because of their familiarity with basic technology is like expecting them to automatically know how to prepare a gourmet meal because they know how to cook a Hot Pocket in the microwave.

Digital native first entered the popular lexicon in 2001. In an article entitled Digital Natives, Digital Immigrants, education consultant Marc Prensky explained that “Our students today are all ‘native speakers’ of the digital language of computers, video games and the Internet.” In many respects, the article looks terrible 14 years later. Prensky, for example, speaks glowingly of Digital Natives’ ability to multi-task, despite the fact that contemporary research (subsequently extended and validated) had already demonstrated the opposite. But just because it suffers from some poor assumptions and hyperbole (Prensky claims we’ve already arrived at a singularity) does not mean the thesis needs to be totally rejected.

You can reconcile (i) a belief that, on average, younger generations are more accustomed to technology (and the rapid evolution thereof) than their predecessors with (ii) the recognition that this comfort does not automatically translate into proficiency. Indeed, in his seminal article, Prensky talks about the need to train a bunch of Digital Natives on a CAD program that “contained hundreds of new buttons, options and approaches.” Prensky takes pride in efforts to gamify the training and “create a series of graded tasks into which the skills to be learned were embedded.” For Prensky, this approach translated training into the language of the Digital Native. He reports that the main impediment was the reluctance of the Digital Immigrant professors to adjust their pedagogical approach. But even Prensky realized that the Digital Natives still needed training (competence-based assessments paired with synchronous, active learning).

Prensky’s article was anecdotal, not empirical. It did not address the fact that most of what Digital Natives did with technology was related to consumption, not application. It also ignored the inconvenient fact that it was Digital Immigrants who had invented the technologies on which the Digital Natives relied. And it introduced a term that conflates general familiarity with specific facility. A decade later, the London School of Economics would publish a paper entitled Digital natives: where is the evidence? The paper concluded that there was no real evidence of fundamental differences between generations. What differences existed were best explained by “breadth of use, experience, self-efficacy and education.” Or as another academic paper would find:

Young people’s engagements with digital technologies are varied and often unspectacular – in stark contrast to popular portrayals of the digital native. As such, the paper highlights a misplaced technological and biological determinism that underpins current portrayals of children, young people and digital technology.

Subsequently, the Organization of Economically Developed Countries (OECD) ran an international study. Instead of asking people about their general comfort with technology, the study asked them to actually solve basic problems using technology (again, a competence-based assessment). Millennials did not fare well. In the words of The Washington Post, “U.S. millennials performed horribly.” Or, as Fortune summarized it:

We hear about the superior tech savvy of people born after 1980 so often that we tend to assume it must be true. But is it?

…. It turns out, says a new report, that Millennials in the U.S. fall short when it comes to the skills employers want most: literacy (including the ability to follow simple instructions), practical math, and — hold on to your hat — a category called “problem-solving in technology-rich environments.”

The advocacy group Change the Equation put out a related report about the High Cost of Low Technology Skills, which included the following graphics:

Digital natives are not at fault for the fact that comfort does not automatically translate into skill. The myths surrounding the digital native, however, have done them a disservice. The belief that they already know everything about technology has convinced us and them that they do not need training in technology. As discussed in the last post, the notion that they are already tech savvy introduces barriers associated with metacognition and mindset.

Metacognition is thinking about thinking. The related concept of metaignorance is ignorance about our own ignorance. We don’t know what we don’t know. Not recognizing how incompetent we are results in unfounded confidence in our own skill level. For this reason, the people most in need of training are the least likely to recognize it. Moreover, confidence begets ego. Those who have a high opinion of themselves are the least inclined to admit facts that undermine their self image.

The problems of ego are compounded by mindset. People with a fixed mindset believe that cognitive traits are stable. You are either smart or you aren’t in the same way that you are either tall or you aren’t. You are born with it, or you are not. To the extent tech acumen is treated as product of age, it will be approached with a fixed mindset, which means that people will try to hide their deficiencies rather than recognize and remedy them.

Because we take a fixed mindset approach, we tend to act as if neither older professionals nor younger professionals will benefit from training. The former because they lack the capacity. The latter because they lack the need. We are wrong on both counts.

I am not trying to pick on anyone. I understand why both older and younger generations buy into the myths surrounding digital natives. But they are myths with pernicious consequences. Many of the decisions about training (or lack thereof) made at law schools and legal employers rest on an illusory foundation. It isn’t always what we don’t know that gives us trouble, it’s often what we know that ain’t so. 

Part 2 (originally posted separately)

90% of people don’t know how to use CTRL+F to find a word in a document or web page. Instead, they search the old-fashioned way, manually skimming the text.

This preponderance of ignorance is stupefying to me. But I want to be very clear that I am using the word “ignorance” in its most neutral form–i.e., lack of information or knowledge–rather than to convey any judgment or pejorative connotation. Ignorance is unavoidable. The only settled part of the debate as to who was the last person to know everything is that the person is long dead.

The curse of ignorance is that you don’t know what you don’t know. Previous posts have touched on this obstacle of metacognition, and our ignorance of our own ignorance. But there is another side of the coin: the curse of knowledge. The curse of knowledge is that once we know something, it is really hard to imagine not knowing it. This incapacity undermines communication and, especially, instruction because of the lack of shared information and assumptions. If I, for example, were going to put together some tips on internet research, I doubt that, absent the article cited above, I would have thought to include CTRL+F. I would have assumed that most everyone already knew it. I would have been wrong.

Indeed, I am a posterchild for both curses. I’ve told the story many times that my inflection point in using technology involved a client discovering that I printed and scanned to create PDFs. But how was I supposed to know what I didn’t know–there’s an app for that–without already knowing it? Yet, several years later, I delegated a task where one of the steps involved converting a large volume of documents into PDF. I was shocked (shocked!!!) to find that the person was spending hours scanning and printing. I assumed that because I knew how to convert a file to PDF, they knew it, too, despite the fact that I had been Exhibit A that this was not knowledge everyone possessed.

Thus, whether we know something or not, we too often assume that others know it. The tech-averse frequently fall into the trap of thinking the tech-comfortable know everything there is to know about tech (i.e., they can’t tell the difference between someone who knows slight more than them and someone who knows infinitely more than them). And those who know tech sometimes assume that their tech knowledge is widely shared.

Both curses are reason that competence-based assessments are such excellent training tools. Figuring out what people do and do not know is superior to speculation. But assessments alone are not enough. The primary objective of identifying gaps is to tailor the training to fill them. In this regard, I have been an abject failure in speaking to law school classes.

I speak to law school classes for free. I provide them a copy of my Legal Technology Assessment (“LTA”) for free. I then provide a copy of the LTA Training Edition (which pairs the competence-based assessment with synchronous, active learning) for free. Finally, they can retake the LTA (for free). Not only do have the opportunity to address identified deficiencies in their skill set but a qualifying score is also something they can add to the bottom of their resume to replace the meaningless “proficient in MS Office.” After speaking to hundreds upon hundreds of students, I’ve had exactly zero take me up on my full offer.

The class I wrote about last week is representative. Twelve students took the LTA because it was a class assignment. The results (below) were bad, as usual. I spoke to them for 40 minutes and offered the Training Edition to anyone who wanted it. Only two of the twelve emailed to ask for the Training Edition. And, if history is any guide, neither of them will return to take and pass the LTA.

In approaching these classes, my idea is that taking the LTA beforehand will puncture delusions of adequacy. We won’t get bogged down in an abstract conversation about how fluent they are with technology. 32% correct on some fairly simple Word tasks leaves little room for debate:

Pretty bad but not unexpected. As I try to communicate to them, it is not their fault. Everyone just assumes that they know things that they had no way of knowing absent training. They are not stupid, lazy, or untalented. They are smart, hard working, and full of promise. They simply lack training in one particular area that has the potential to make their lives better.

On the issue of their immediate future, I point out that their most recent predecessors are miserable human beings. In fact, the students are auditioning for the unhappiest job in America.

I then try to persuade them that technology plays a role in this dissatisfaction. Before technology takes our jobs, it can make them easier. At least, in theory. The technology has to actually be good, and we have to use it correctly. Otherwise, it is a source of frustration rather than leverage. Technology initially substitutes for labor at the most severe pain points. Machines can reduce the hours spent reviewing, proofing, conforming, collating, updating, and otherwise fiddling around the edges of the substantive legal work. Using technology well can improve both speed and accuracy, as I try to convey in the video below, and thereby alleviate a fair amount of the agony associated with being a young lawyer:

My contention is that having the right technology and learning to use it correctly will permit legal professionals to reduce the amount of their finite time and attention that is directed towards misery-inducing busywork. I’ve added to my spiel some recent confirmation of this theory from the cover story of last month’s American Lawyer. AmLaw’s annual associate satisfaction survey found that technology, including technology training, has a material effect on satisfaction:

One unsung key to retention could be technology. We found that overall satisfaction of midlevel associates, as measured on our survey, was strongly statistically correlated to their law firm’s scores on four questions involving technology. (The questions ask respondents to rate their firms’ technology generally, as well as technology training, support and use of technology in meeting client needs.) 

….In fact, eight of the top 11 firms in the national satisfaction rankings also were at the top on the technology questions. Conversely, many of the firms that occupy the bottom of the national satisfaction rankings also place low in the technology survey.

The AmLaw conclusions comport with an earlier study I cite from the National Conference of Bar Examiners that surveyed recent law graduates about the most important skills for young lawyers. Out of 30 skills, using basic office technology ranked 6th:

Seeing basic office technology ahead of legal reasoning is a bit jarring, even for me. But the incongruence is heightened by the fact that, unlike the rest of the skills listed above, using technology is not taught in most law schools (or, generally, in most colleges or high schools). 

Then again, the idea that law school is not geared towards turning out practice-ready lawyers is well-worn territory. As discussed in Mark’s previous post, a LexisNexis survey found that “95% of hiring partners and associates believe that recently graduated law students lack key practical skills.” The dissatisfaction of associates is mirrored (and, maybe partially driven) by the dissatisfaction with associates. This is not just abstract griping. Anecdotally, partners report writing off massive amounts of associate time for perceived inefficiency. These claims appear to be borne out by the Georgetown Law and Peer Monitor realization data (which I dug into here):

 

So that’s my story. You’re great. You just haven’t gotten the training you need in technology. This training will benefit you directly in the form of improved satisfaction and performance. Here it is, for free. Followed by crickets.

I’m not quite sure how to interpret my utter inability to make any progress with these students (thankfully, the people who actually pay me are considerably more engaged). Am I, yet again, suffering from the curse of knowledge? Is there some assumption that I am making about these students that is impeding communication? As I try to put myself in their shoes, I increasingly come to conclusion that there isn’t anything I can say.

In general, it is challenging to get anyone to use their precious spare time to buckle down and really learn something new, even if they are persuaded that they should. The last time I decided to tackle a new area of study, I felt compelled to pay for online courses that included tests and graded assignments. I needed real stakes and real structure to have the discipline to systematically engage with the material (all of which I could have found for free on the internet). Here, the students took the LTA as a diagnostic because it was an assignment, and, I have no doubt, that they would have trained for and passed the LTA if that were also assigned. As a law student, I suspect I would have behaved much the same way (I know my scores would have been just as bad).

Stakes and structure matter. These students have had both all their life. From speaking to them, I get the sense that they believe this will continue. They believe that law school is designed to prepare them for law practice. They believe that whatever they do not know upon leaving law school, their firms will teach them. And, more than anything, they believe that they do not need to worry about this tech stuff because they will have secretaries to do it for them. More on that last point in my next post.

For me, the primary myth of the digital native is that, by virtue of their age, they already know what they need to know with respect to using technology. The corollary myth is that which they do not already know is not worth learning. But there exists a softer formulation that hits much closer to the truth. Rather than automatically knowing that which they need to know with respect to technology, we (and they) tend to believe that people who grew up with technology have the capacity to learn it and will do so when the situation requires.

The older generations seem to think that the situation will somehow mandate the acquisition of new skills. In this, they are not totally wrong. Most people, including the older generations themselves (with their fancy new iPhones and Surface Pro 4’s), learn what they need to learn to get by with technology. Some people learn more. But most satisfy the bare threshold of survival. This results in massive underutilization of extant technology. And study after study has shown that younger generations are the same as their predecessors in this regard–i.e., learn the minimum to get by.

The younger generations, on the other hand, think that they will quite literally be required to learn it. Someone in a position of authority is going to lay out a curriculum, objectives, and a timeline. At that point, they will do what they’ve always done: work hard to meet the expectations set for them. A few will fall short. Some will excel. But most will quite effectively do what they are asked to do. I, for one, think we ought to oblige them.

At some point, I will dig deep into my data. But, on average, people (lawyers and staff) in practice outperform the kids in school on the LTA. In part, this reflects a general raising of the baseline as the skill set required for bare survival expands upon entering the professional workforce. But there is still significant interorganizational and intraorganizational variance.

The variance between organizations appears to be entirely attributable to mandatory training. Different organizations have different attitudes towards training (is it available, is it mandatory, does it include competence-based assessments) that, unsurprisingly, have an appreciable impact on how well trained their employees are.

The variance within organizations stems from outside training. Frequently, I learn that the person who well outpaced her colleagues on a diagnostic assessment had some previous career that demanded a more robust technology skill set. Sometimes, I meet people who, like me, had some sort of rude awakening and decided they did not like being embarrassed. Every now and then, I encounter a true tech geek (meant with love and affection) who happens to also work in law.

My own data reinforces previous empirical empirical findings that, rather than age, facility with technology is a product of “breadth of use, experience, self-efficacy and education.” Technology training is important for everyone, including the digital natives. I just wish I could convince them of that.

++++++++++++++++++++++++++++++++++++

Casey Flaherty is the founder of Procertas. He is a lawyer, consultant, writer, and speaker focused on achieving the right legal outcomes with the right people doing the right work the right way at the right price. Casey created the Service Delivery Review (f.k.a., the Legal Tech Audit), a strategic-sourcing tool that drives deeper supplier relationships by facilitating structured dialogue between law firms and clients. There is more than enough slack in the legal market for clients to get higher quality work at lower cost while law firms increase profits via improved realizations.

The premise of the Service Delivery Review is that with people and pricing in place, rigorous collaboration on process offers the real levers to drive continuous improvement. Proper collaboration means involving nontraditional stakeholders. A prime example is addressing the need for more training on existing technology. One obstacle is that traditional technology training methods are terrible. Competence-based assessments paired with synchronous, active learning offer a better path forward. Following these principles, Casey created the Legal Technology Assessment platform to reduce total training time, enhance training effectiveness, and deliver benchmarked results.

Connect with Casey on LinkedIn or follow him Twitter (@DCaseyF).