16
Vent: My A/B test disaster that killed conversion for 2 days
I was running a split test on our landing page headline in Austin last month. I set up the wrong variant as the winner by accident and it tanked our signups by 30 percent for 48 hours. Caught it when my coworker mentioned the form looked weird on her phone. Felt stupid but at least the data from the failure taught me which headline really worked. Has anyone else had a mistaken campaign setting blow up in their face?
3 comments
Log in to join the discussion
Log In3 Comments
ward.jamie6d ago
Read a case study from some SaaS company that had the exact same thing happen. They lost like 40 percent of their trial signups for three days before anyone noticed the test was flipped.
2
keith_bennett6d ago
You said "I set up the wrong variant as the winner by accident" but that's not really how A/B testing works. You can't just "set" a winner manually unless you're using some really basic tool that lets you override the results. Most platforms like Optimizely or Google Optimize automatically show the winning variant based on traffic. You probably just paused the wrong one or switched the URLs in the redirect. Either way, losing 30 percent for two days stings. I did something similar once where I accidently gave the "control" the same URL as the variant, so everyone saw the new headline for a full weekend. At least you walked away with good data, that's more than most people get from a blown test.
0