6
Why does nobody talk about how A/B testing can fail silently?
I ran an A/B test on a landing page last week for a client in Austin. Changed the headline and button color, thought it was all good. After 4 days I checked the results and the sample size was tiny because my tracking code was broken from the start. Wasted $200 on ad spend for nothing. Has anyone else had tracking issues mess up their tests like that?
3 comments
Log in to join the discussion
Log In3 Comments
hannah3201mo ago
Tracking code issues are brutal but there's a way bigger silent killer that nobody talks about. I worked on a split test for a SaaS company back in 2021 and we ran it for 3 weeks with a solid sample size. The problem was our landing page loaded at 4.5 seconds for mobile users and the control page was at 2 seconds. A/B testing platforms don't flag page speed differences unless you specifically monitor for them. So we thought the new design was performing worse when really it was just slower on mobile. That hidden variable completely changed the results without any warning.
8
blake7921mo ago
and nobody checks page speed unless a developer specifically tells them to, which is almost never.
7