Sunday, August 3, 2014

On "experimenting on your users"

OKCupid recently ran an experiment on its users to determine whether its reported match percentage with another user had an effect on messaging behavior, and people are angry.

The premise of that anger is that it means something to be a 90% match or a 30% match with another OKCupid user; if you lie to a user and tell them that this percentage is something different from what the algorithm currently says it is, that's wrong. But what if that number is meaningless? What if the secret sauce is not about the compatibility metrics that OKCupid calculates between any two users?

If the secret sauce to get two people to have a successful relationship is actually to tell them that they're an algorithmic match, the matching system is useless. OKCupid should then consider dropping their algorithms that attempt to create matches. Indeed, OKCupid's data show that the odds among real 90% matches of a single message turning into a conversation are 20%, versus 17% among the 30% matches that thought they were 90% matches. So the real matching adds just 18% to the likelihood you'll have a real conversation. That's not nothing, but it's not huge. If the "X% match" number is close to meaningless, it isn't really lying to alter it. In fact, the greatest lie is to present it as meaningful. Presumably, OKCupid believes that this difference in likelihood is worth it, or it believes that it can improve it.

OKCupid's job is to help its users find other users that they'll be happy with, whatever their romantic goals. OKCupid can measure some of the behaviors that indicate that - in this case, messaging. It's in the interest of users that OKCupid figure out how to give them the best matches. Messaging may not be the best way to measure that (how about surveying them a few weeks later?), but it's real behavioral data. OKCupid may be irresponsible about the way they selected users for this, and maybe they should tell users that by joining the site, they're consenting to X% of their actions being subject to experimental alteration. But the idea that they're just jerks overlooks the fact they're trying to better understand how to give their users better matches. And that's their job.