Thursday, August 21, 2014

Reviewers will find more spelling errors in your writing if they think you're black

People would like Thurgood Marshall's writing even better if they thought he was white. John Greim/Lightrocket

You're reading this article. Should it change your opinion of the piece's quality to know that its author was white, black, or of another race? No, if you're evaluating it on its merits.

Yet a new study by the consultancy firm Nextions shows that reviewers of a legal brief did just that.

The idea

In an experimental context, when reviewers were told the author of a legal brief was black they consistently rated identical pieces lower in quality and identified more spelling, grammar, factual, or analytical errors. It's evidence that, even if the days of overt bigotry and explicit discrimination are mostly past, the United States still struggles with a deep problem of implicit racism.

Arin N. Revees, the president of Nextions and the author of the study, argues that the implicit racism happened because reviewers take the racial information she provided as a cue for how they should judge the work. When the author is supposed to be white, reviewers excused errors as out of haste or inexperience. They commented that the author "has potential" and was "generally a good writer but needs to work on" some skills. When the author is supposed to be black, those same errors became evidence of incompetence. A reviewer said he "can't believe he [the author] went to NYU," and others said he "needs lots of work" and was "average at best."

The evidence for

Nextions recruited five lawyers at five different law firms to co-write a research memo about trade secrets at Internet start-ups. Then they inserted several errors into the memos — errors in spelling, grammar, legal terminology, fact, and analysis. Last, they created two different headers for the memo. Both identified the author as "Thomas Meyer," a third-year associate with a degree from the New York University School of Law, a top-ranked school. Yet in one version of the header, Thomas Meyer was identified as "Caucasian"; in another, he was identified as "African American."

They sent these otherwise-identical memos to 60 different partners at law firms who had agreed to review the pieces and identify errors. The reviewers, though, weren't told that the experiment was about race — they were just asked for their opinion on the quality of the memo.

They judged the memo far more harshly when they were told the author was black than when they were told the author was white. On a scale of one to five, reviewers gave the black Thomas Meyer a 3.2 and the white Thomas Meyer a 4.1.

More troublingly, not only was the overall score lower, but reviewers seemed to find more errors when they believed the author was black. Out of seven spelling and grammar errors in the text, they found 2.9 of them on average in white Thomas Meyer's memo and 5.8 of them on average in black Thomas Meyer's memo. Out of six technical writing errors, reviewers found an average of 4.1 in the white version and 4.9 in the black version. Out of the five factual errors, they found an average of 3.2 in the white version and 3.9 in the black version.

There was no apparent effect from the race of the reviewer, and the sample of reviewers included a mix of races.

Possible grounds for doubt

There are two serious shortcomings in the Nextions study. It should be taken as suggestive, but not conclusive, evidence of the power of implicit racism. Those two flaws are that the sample size was puny and, more dangerously, the study design makes it impossible to control for the harshness of the reviewer.

Only 60 reviewers participated, and only 53 actually sent back their reviews. That's not enough for strong conclusions.

What was more problematic was that each reviewer only looked at one memo. This is a problem because some reviewers will be harsher in general, and a better study would control for differences in the reviewer. The Nextions report assumes that all reviewers are equivalent -- but, with such a small sample size, this is a very risky assumption. If the reviewers that received memos from black Thomas Meyer were just harsher overall, this study would conclude that the blackness of Thomas Meyer, and not the harshness of the reviewer, was at play. The way to solve this problem would be to have each reviewer look at more than one memo.

Log In Sign Up

Log In Sign Up

Please choose a new Vox username and password

As part of the new Vox launch, prior users will need to choose a permanent username, along with a new password.

Your username will be used to login to Vox going forward.

I already have a Vox Media account!

Verify Vox Media account

Please login to your Vox Media account. This account will be linked to your previously existing Eater account.

Please choose a new Vox username and password

As part of the new Vox launch, prior MT authors will need to choose a new username and password.

Your username will be used to login to Vox going forward.

Forgot password?

We'll email you a reset link.

If you signed up using a 3rd party account like Facebook or Twitter, please login with it instead.

Forgot password?

Try another email?

Almost done,

By becoming a registered user, you are also agreeing to our Terms and confirming that you have read our Privacy Policy.



Choose an available username to complete sign up.

In order to provide our users with a better overall experience, we ask for more information from Facebook when using it to login so that we can learn more about our audience and provide you with the best possible experience. We do not store specific user data and the sharing of it is not required to login with Facebook.