While I was sadly unable to attend the 2016 AALL annual conference that wrapped yesterday in Chicago, I have it on good authority that the most recent ALM Law Library Survey  caused some intense discussion around industry surveys and their value. The issues raised go well beyond law libraries and seem to fit almost any annual survey where industry statistics or trends are presented. Those in legal marketing, would no doubt agree that directory rankings, akin to surveys of the industry are just as flawed in their research and survey methodology. Not to mention the concentrated amount of work they represent for a dubious ROI.  Nevertheless, since it was a hot topic yesterday, let’s go back to the ALM Law Library Survey as the catalyst for exploring the topic of surveys and their value. 

On the surface, the question “Do you plan to eliminate a majority of your print collection within the next five years?” seems innocuous. But if you think about it for more than a second, the question itself is a bit like leading the witness. There is no way to answer the question without distorting the data.   If you answer yes, the analysis will jump to suspicions of shrinking libraries, if you answer no, the analysis will jump to law librarians not embracing the future and e-resources.  Either way, the survey will point to a definitive trend about the value of librarians and libraries as a result of one poorly worded question the value and timeliness of which is in and of itself outdated. Not to mention that the results are often skewed by lumping all respondents together regardless of budget size, head count or prior culling of collections that have already been accounted for in previous surveys.  

The question, simple on the surface, points to a lack of understanding about the industry and most certainly limits the usefulness of the aggregated responses.  Print resources are an indication of – what?  Surveys, much like directories, league tables and the like provide those who create these industry research pieces with dramatic headlines, website traffic and social media click-throughs, generating soft leads and business or consulting opportunities. Surveys also provide data and data should provide insights, at least that is how it is supposed to go down.  But there has to be integrity and mindfulness in both the collection and analysis – the question above and its binary answers with no context, provides for neither. 

I am a data junkie, you all know that. I love metrics, and analytics makes me happy.   I understand the desire to create some benchmark that compares everyone to everyone else to see who’s winning and who’s falling behind. It can be the American Lawyer rankings, or it can be a Cosmo survey. The problem is that if those that are producing the survey don’t ask thoughtful questions and implement solid statistical analysis, they can pretty much make up the results (the old “I can make data say and anything I want”) to serve their end needs.  Now, I may be naïve or a bit of a Polly Anna, but I honestly don’t believe that those who create these surveys do so with malicious intent. I do believe that ALM or Bloomberg, Lexis or Thomson genuinely care about the clients they serve and do want to provide value to their clients by way of empirical data and industry insights.  

As I see it, the current model of survey and reporting as discussed at AALL has two flaws.  The first: survey creators are often on the sidelines, looking in on the action – they support the game but aren’t in it the same way industry leaders would be.  Therefore, to really add value, survey creators need to find a way to include industry leaders or practitioners in the creation of the survey so that the same questions are not asked year after year skewing the results in to a cumulative data mess. If you want to provide real usefulness, start by asking insightful well-crafted questions.  The second: despite working in an industry of word smiths who manipulate language to serve the needs of their clients, many of us (myself included) need to brush up on our written communication and analysis skills. That is, we need to be able to draft better questions, that result in stronger more representative and meaningful findings.  The size of print collections in the earlier example is no more an indication of a library’s strategy than the number of beds in a hospital speaks to the quality of care. Correlation is not causation, and poorly constructed survey questions, limit analysis to trite observations adding little benefit to the working body of industry knowledge.

I’ll stop the rant now, but hope next time the discussion turns to surveys, we can actually discuss the results and what action we want to take, rather than the methodology.