Here's why I'm skeptical of Roland Fryer's new, much-hyped study on police shootings

As a general rule, when somebody claims that a new academic study finally looks into the data behind a controversial news issue, you should be skeptical.

Often, this kind of study is a little like Columbus’s “discovery” of America. Yes, something’s being brought to the attention of a different (and sometimes bigger) audience, but there were people living and working there before.

And if you insist on overlooking those people and seeing a new and wild wilderness, you’ll probably lose out on learning some valuable lessons about what to do with the newly discovered thing.

The latest example of this: a new study from Harvard economist Roland G. Fryer Jr., published by the National Bureau of Economics Research and hyped by the New York Times. The study claims to show there isn’t a racial disparity when police use deadly force against people they’ve stopped or arrested — and it’s being presented as the best data yet on the question of police use of force.

The study is definitely useful in some regards: Researchers examined the circumstances of police shootings (like whether police were attacked before firing their guns) with much more detail than previous work has, for example. But it’s narrower than a lot of existing research on racial disparities in criminal justice — so narrow, in fact, that it kind of misses the point of why police killings of young black men are so frequently outrageous.

You won’t know that from reading write-ups that treat this study (or any study) as a Columbus-level discovery. On the bright side, that makes this study a useful object lesson for a few tips in how to read social science research in general (and criminal justice research in particular).

Ask yourself: is there really no data out there already? Are we sure?

If you’ve been following the nationwide debate about police killings of black Americans, you know that we don’t have super-reliable data on how many people (and which ones) get killed by police.

But there is some data out there: from the FBI’s Uniform Crime Report, media-compiled databases from the Washington Post and the Guardian, and volunteer-compiled databases like Fatal Encounters (which Vox uses for our maps of police killings).

Joe Posner/Vox

So when the Times article summarily dismisses existing data as “poor,” and doesn’t explain what that data actually is, that should be a red flag — a clue that the article’s author isn’t going to provide you with an explanation of why this new data is so much better than the old data, and you’re going to have to do that yourself.

When Fryer (an economist by training) tells the Times that he got interested in police shootings because of “his anger after the deaths of Michael Brown and Freddie Gray,” and (in Fryer’s words) “decided I was going to collect a bunch of data and try to understand what really is going on,” that should be another humongous red flag.

It implies that Fryer assumed he was doing something pioneering, rather than asking first what work was already being done and what he could add to the existing conversation. This is something that often happens when people in “quantitative” social sciences, like economics, develop an interest in topics covered in other social sciences — in this case, criminology: They assume that no rigorous empirical work is being done.

Ask yourself: How broad is this data? How broad are the claims being made about it?

The Times report does explain the data that Fryer and company used in the new study. But it turns out it’s not nearly as broad a sample as the conclusion “these results undercut the idea that the police wield lethal force with racial bias” would suggest:

(Fryer) and a group of student researchers spent about 3,000 hours assembling detailed data from police reports in Houston; Austin, Tex.; Dallas; Los Angeles; Orlando, Fla.; Jacksonville, Fla.; and four other counties in Florida.

That’s 10 police departments in three states, with a majority of them based in major cities. That allowed Fryer and his team to compile a database with a lot of different shootings — about 1,330 over 16 years, from 2000 to 2015 — but not a lot of different police departments or institutional cultures.

When it comes to policing, this is especially important, because so many issues of crime and policing are local. Different cities have different approaches to police-community relations; different tensions; different standards for use of force. (In fact, the cities Fryer and his team worked with are all members of a White House initiative on policing data launched in 2015 — and the kind of department that thinks data collection and transparency are important is likely to have different priorities in other regards than one that isn’t.)

In comparison, the FBI’s Uniform Crime Report database includes records from thousands of police departments around the country. The number of shootings in the Fryer data set, spanning 16 years, is about equivalent to what the Uniform Crime Report compiles in two or three.

More importantly, the UCR report includes not just major cities but small towns and rural areas; not just diverse cities but less diverse ones; not just departments that think carefully about data collection but departments for which it’s just a needed chore to qualify for government funds.

Ask yourself: Is the question the study answers the same one the public is asking?

The most revealing passage in the Times article is probably the one explaining what Fryer and his team didn’t include in their study:

It focused on what happens when police encounters occur, not how often they happen. Racial differences in how often police-civilian interactions occur reflect greater structural problems in society.

In other words, Fryer and company found that there weren’t big racial disparities in how often black and white suspects who’d already been stopped by police were killed. But they deliberately avoided the question of whether black citizens are more likely to be stopped to begin with (they are) and whether they’re more likely to be stopped without cause (yup).

Avoiding those issues makes sense for the question Fryer was trying to answer. He wanted to know what happens between the moment a police officer stops someone and the moment he pulls the trigger — and how those sequences of events vary by race.

But when people talk about racial disparities in police use of force, they’re usually not asking, Is a black American stopped by police treated the same as a white American in the same circumstances? They’re making a broader critique of the “greater structural problems” in society in general and the criminal justice system in particular. They’re saying that black Americans are more likely to get stopped by police, which makes them more likely to get killed.

Eric Garner was killed in 2014 when police tried to arrest him for selling loose cigarettes. Philando Castile had been pulled over 52 times on misdemeanors (including for driving without a muffler and not wearing a seatbelt) before he was shot and killed last week. Michael Brown was stopped by Darren Wilson for walking in the middle of the street.

Maybe it’s possible (maybe) that those encounters would have been just as likely to escalate to the point of lethal force if each of those men had been white — but it kind of misses the point to say that, because if they’d been white, the encounters probably never would have happened.

Controlling for variables is an extremely important thing in social science. It allows you to figure out which factors actually matter and which ones don’t. In this case, Fryer and his team have given us suggestive evidence that among major-city police forces, police in tense situations are not unusually likely to shoot black suspects. They’ve made a valuable addition to the literature. But it’s just that: an addition, not a discovery, and not the last word.