I spent the past three years working on interesting problems at a great company. I worked with smart people, learned a ton about using data to make decisions and uncover knowledge, and built the soft skills necessary to get things done in a range of situations. I also lived very, very comfortably. I bought dinner without worrying about whether I could afford it. I visited my family in New York as often as I wanted, working remotely whenever it was most convenient. I had the space and resources to fix fitness issues that had plagued me my whole life. Life was good.
But my work became unfulfilling. I was optimizing at the margins for people and organizations who are already very successful, trying to find the insights to improve outcomes by a few percentage points at a time. At my company's scale, that was well worth my salary. But diminishing returns now meant that it was no longer worth the opportunity cost for me. Since my first year teaching, I knew that my goal in my career was to find the point of greatest leverage to improve outcomes for my students - broadly, those affected by the achievement gap. Becoming a strong analyst gave me skills to increase that leverage, but I'd grown my skill set enough in three years that it was time to bring it back to the work I really care about.
So I quit my comfortable job and found a teaching job. My departure from my job for the classroom has been greeted with a pretty consistent pattern: first surprise and confusion, and then expressions of admiration at my "noble" career choice. While I like a compliment as much as the next person, I hope that we'll someday live in a world where we don't look at highly skilled teachers as people who gave up bigger careers to teach. In this work, we need all the dedicated and growth-oriented hands we can get. Our students deserve teachers that work in a competitive labor market, funded by the true incremental value that great teachers can provide and staffed by people who become educators because it's both the thing they can do best and a job that they love.
This time, I'm working for a school with the stated and substantially fulfilled mission of setting its urban students on a path to and through college. At my old school, I worked with a handful of amazing educators that shared this goal; at my new school, it's a culture that penetrates everything that every member of our staff does. I will face new challenges and new failures every day, but I'll gain new points of leverage. In the near term, I'll work to make my Biology students stronger scientific thinkers, writers, and doers. In the long term, I hope that my experience in this school will equip me to work in school leadership that brings these outcomes to a larger group of students.
Tuesday, August 5, 2014
Sunday, August 3, 2014
On "experimenting on your users"
OKCupid recently ran an experiment on its users to determine whether its reported match percentage with another user had an effect on messaging behavior, and people are angry.
The premise of that anger is that it means something to be a 90% match or a 30% match with another OKCupid user; if you lie to a user and tell them that this percentage is something different from what the algorithm currently says it is, that's wrong. But what if that number is meaningless? What if the secret sauce is not about the compatibility metrics that OKCupid calculates between any two users?
If the secret sauce to get two people to have a successful relationship is actually to tell them that they're an algorithmic match, the matching system is useless. OKCupid should then consider dropping their algorithms that attempt to create matches. Indeed, OKCupid's data show that the odds among real 90% matches of a single message turning into a conversation are 20%, versus 17% among the 30% matches that thought they were 90% matches. So the real matching adds just 18% to the likelihood you'll have a real conversation. That's not nothing, but it's not huge. If the "X% match" number is close to meaningless, it isn't really lying to alter it. In fact, the greatest lie is to present it as meaningful. Presumably, OKCupid believes that this difference in likelihood is worth it, or it believes that it can improve it.
OKCupid's job is to help its users find other users that they'll be happy with, whatever their romantic goals. OKCupid can measure some of the behaviors that indicate that - in this case, messaging. It's in the interest of users that OKCupid figure out how to give them the best matches. Messaging may not be the best way to measure that (how about surveying them a few weeks later?), but it's real behavioral data. OKCupid may be irresponsible about the way they selected users for this, and maybe they should tell users that by joining the site, they're consenting to X% of their actions being subject to experimental alteration. But the idea that they're just jerks overlooks the fact they're trying to better understand how to give their users better matches. And that's their job.
The premise of that anger is that it means something to be a 90% match or a 30% match with another OKCupid user; if you lie to a user and tell them that this percentage is something different from what the algorithm currently says it is, that's wrong. But what if that number is meaningless? What if the secret sauce is not about the compatibility metrics that OKCupid calculates between any two users?
If the secret sauce to get two people to have a successful relationship is actually to tell them that they're an algorithmic match, the matching system is useless. OKCupid should then consider dropping their algorithms that attempt to create matches. Indeed, OKCupid's data show that the odds among real 90% matches of a single message turning into a conversation are 20%, versus 17% among the 30% matches that thought they were 90% matches. So the real matching adds just 18% to the likelihood you'll have a real conversation. That's not nothing, but it's not huge. If the "X% match" number is close to meaningless, it isn't really lying to alter it. In fact, the greatest lie is to present it as meaningful. Presumably, OKCupid believes that this difference in likelihood is worth it, or it believes that it can improve it.
OKCupid's job is to help its users find other users that they'll be happy with, whatever their romantic goals. OKCupid can measure some of the behaviors that indicate that - in this case, messaging. It's in the interest of users that OKCupid figure out how to give them the best matches. Messaging may not be the best way to measure that (how about surveying them a few weeks later?), but it's real behavioral data. OKCupid may be irresponsible about the way they selected users for this, and maybe they should tell users that by joining the site, they're consenting to X% of their actions being subject to experimental alteration. But the idea that they're just jerks overlooks the fact they're trying to better understand how to give their users better matches. And that's their job.
Subscribe to:
Posts (Atom)