As he promoted his new book, Sen. Jeff Flake (R-AZ) made news last week by calling for more civility from politicians and especially President Trump. He said one of the major concerns with the declining civility in Washington is the example it sets for the public in general. As he put it on Face the Nation Sunday, “We have to model behavior that we'd be proud that our kids are watching.”
But to what extent does the tone in Washington actually appear to affect how the public discusses politics? In a new project, we have been working to track and understand incivility by examining the extent to which users post offensive comments on Reddit. Reddit has received a great deal of attention from political analysts in recent years due to the fact that it has a robust offering of political discussion boards. In fact, Reddit has become an important enough player in online political discussions that President Obama and a host of 2016 presidential candidates took to the forum to engage with its users.
We examined every political comment posted on Reddit from January 2015 to January 2017, more than 39 million comments in total. Because it would be impossible to code millions of comments by hand, we instead developed an algorithm to accurately identify “offensive” comments. Among other things, the algorithm tagged posts as offensive when they used profanity, compared someone to Hitler or a pedophile, engaged in name calling, and so on.
We tested the accuracy of our algorithm on a data set of 14,500 tweets that were coded by humans, and our automated approach correctly classified 90 percent of the tweets based on whether or not human coders would have found them offensive. In testing, we have found that when the classifier does make errors, it is more likely to underestimate the presence of offensive speech. Thus, our results can be considered conservative estimates of the presence of offensive speech on Reddit.
Posts became more offensive during the general election campaign
The figure below shows the percentage of Reddit comments that were classified as “offensive” during each week of 2015 and 2016. The blue dots represent political comments posted to Reddit, while the red dots are from a sample of nonpolitical comments. The first thing to note is that roughly 8 percent of nonpolitical comments are classified as offensive, and that this level of offensiveness is remarkably consistent over the entire period that we examine.
By contrast, the offensiveness of political comments posted to Reddit varied much more over the two-year period. Interestingly, political comments became somewhat less offensive from early 2015 through the presidential primary period of 2016. It seems the focus on the primary competitions softened the tone of discourse, since intraparty battles are generally less hostile than cross-partisan ones. The one major departure from this trend happened during the week when Trump announced his candidacy for president in a speech that featured a highly offensive categorization of Mexican immigrants.
While political comments on Reddit were no more offensive than nonpolitical Reddit comments during most of the presidential primary period, this clearly changed once the general election campaign kicked off in full. Indeed, we see that the share of offensive comments spiked during the party conventions and continued to increase as the general election campaign was in full swing from August through November 2016. In early October, we see a large single-week spike in offensive comments; this coincides with the week during which the Access Hollywood tapes were released. During the week of and following Election Day, we see similar levels of offensiveness — during these two weeks, nearly one of every 10 political comments posted on Reddit was classified as offensive.
Offensive posts are more popular
Perhaps just as striking as the increase in offensiveness during the general election campaign is the fact that in the two months following the election, offensive speech on political Reddit remained quite high. In fact, after the election, political comments on Reddit were about 20 percent more likely to be offensive than those posted about other topics.
Why are offensive comments so common on Reddit? One reason suggested by our data is that such comments are typically more popular among the Reddit community. To gauge popularity, we calculated the net score for each comment, which is simply the number of “upvotes” that a comment received minus the number of “downvotes.” Offensive comments posted to Reddit receive an average score of 8.9 versus 6.7 for non-offensive comments. Thus, the Reddit community provides substantial positive reinforcement when its users post offensive content.
How important are these patterns we’ve uncovered? After all, Reddit is a forum that attracts just a small proportion of the population who are not very representative of Americans in general. According to a Pew survey, just 4 percent of Americans use Reddit. Reddit users are more likely to be younger, male, and more liberal in their political leanings compared to the average American.
Nevertheless, even with these caveats, our findings have important implications for our politics. Research has shown that contentious political discussions tend to alienate moderates from the political process and make it so that people view opposing viewpoints as illegitimate. Millions of Americans are exposed to political discussions on Reddit every week, and while these individuals are not representative of ordinary Americans, many are political moderates. In fact, the same Pew survey found that people who report learning about political news on Reddit are just as likely to identify as political moderates as the general population.
What’s most telling is that our research shows that the tone of discussions on Reddit seems to respond to how our politicians are behaving. Thus, Flake appears to be correct when he worries about the effect that incivility in Washington may be having on political discourse more generally.
Rishab Nithyanand is a postdoctoral researcher in the College of Information and Computer Science at the University of Massachusetts Amherst. He obtained his PhD from Stony Brook University in 2017.
Brian Schaffner is a professor of political science at the University of Massachusetts Amherst. Find him on Twitter @b_schaffner.
Phillipa Gill is an assistant professor of computer science at the University of Massachusetts Amherst. Find her on Twitter @phillipa_gill.