:format(webp)/cdn.vox-cdn.com/uploads/chorus_image/image/36179960/shutterstock_156882656.0.jpg)
Last month, Facebook got in hot water when one of its data scientists published a paper revealing the company had attempted to alter the emotional content of hundreds of thousands of people's news feeds, just to see how they'd react. For many, the experiment crossed an ethical line, while others, like Vox's Nilay Patel, argued that screwing with your emotions is Facebook's entire business model anyway (personally, I just thought the study's methodology sucked).
But Facebook is hardly the only web company to use its power over what users see for the sake of research. In a blog post Monday (thanks to UNC sociologist Zeynep Tufekci for the link), OkCupid's Christian Rudder laid out three separate experiments the company has run on users.
First, delete all the photos
The first experiment — "Love is Blind Day" on January 15, 2013 — removed photos from everyone's account for seven hours, and found that while usage plummeted, responses to initial messages soared, conversations "went deeper," other contact information was exchanged more quickly. "In short, OkCupid worked better," Rudder concluded. Surveys of women who actually went on dates with men they contacted during the "blind date" period found that they reported having a good time regardless of how much more attractive they were than the men (as rated by other OkCupid users):
Chart by OkCupid
The trouble is that when photos went back up, about 2,200 people's conversations, which started in the blind period, simply ended. In the end, looks still mattered.
Then, delete all the text
The second experiment revived an early feature of OkCupid wherein users could rate each other on both looks and personality; currently, users can only give each other a single rating. OkCupid found that, under the prior system, the two ratings were extremely well correlated, but it wanted to see if that remained true under experimental conditions. To that end, it took a sample of users and only showed them the profile text of other users half the time, and then compared those users' ratings of the text-less profiles to ratings from users who saw the profile text. If they correlated well, then the natural conclusion is that ratings, even with text present, are a de facto attractiveness rating, rather than a personality rating. And sure enough, the ratings with and without text were very similar, so it's no wonder that OkCupid uses the current single-rating as an attractiveness rating in thing like the first experiment:
Chart by OkCupid
Third, mix up all the match scores
The third experiment is probably the most troubling. In it, users were told they had different compatibility scores (on a scale of 0 to 100) than they in fact did; OkCupid then measured what this did both to likelihood of sending messages, and to the likelihood of initial messages turning into conversations. As you'd expect, users told they were 90 percent matches when they were in fact 30 percent matches were likelier to send messages to each other, with odds increasing from 14.2 percent to 16.9 percent. But they were also likelier to keep talking once contact was established — less likely than if they were true matches according to the site's algorithm, but more likely than if they had known their true score:
Chart by OkCupid
Were they justified?
What are we to make of manipulations like these? The findings are at least interesting. The third experiment appears to confirm that OkCupid's scores at least measure some aspect of true compatibility. If they didn't, you'd expect pairs who scored 90 percent and knew it to have conversations at the same rate as pairs were merely told they scored 90 percent, and the fact that the former rate is higher suggests some value in the metric. The first two experiments' findings are somewhat less surprising — ATTN: PEOPLE ARE SHALLOW, MUST CREDIT VOX — but there's value in confirming obvious things.
The question is whether the potential damage of the interventions justifies the lack of informed consent outside of OkCupid's normal terms and conditions. The first experiment is the least troubling on these grounds, since everyone was informed the change was coming, rather than it being rolled out surreptitiously, which enabled people who really did not want a picture-less OkCupid to just not log on in that period (as many users opted to do).
The third experiment is the really concerning one. Users were told their true match scores after the experiment was concluded. What if an actual couple, whose relationship began when they thought they were a 90 percent match, were then informed they were a 30 percent match? It's hardly the biggest emotional blow in the world, and if the relationship was really working, it'd make sense to shrug off the new information. But users can be forgiven for finding the prospect of OkCupid dropping bombs like that on relationships still in their honeymoon phase a tad disturbing.
Update: Tim Carmody's piece on why we're less upset about OkCupid's experiments than Facebook's is excellent, though I'm not sure I agree with that premise; the differential in outrage might be entirely attributable to the fact that Facebook has more users by a couple orders of magnitude.
