Loading…
11
The week our billing system broke and I learned how fragile our whole ops stack really is
Last Tuesday morning I got a Slack from our head of CS saying 14 customers could not log in because their trials had expired 3 days early. Turns out a dev had pushed a small change to our Stripe webhook handler on Friday afternoon and nobody caught it. I spent the next 6 hours pulling logs, comparing timestamps, and manually extending trials for every affected account. The worst part was we had no alert set up for failed webhook deliveries, so the issue sat there all weekend while people quietly churned. By Wednesday we had 3 cancellation emails and a very angry call with a $40k ARR client. I ended up building a simple daily check that pings our webhook endpoint and throws a Slack alert if the count drops below a certain number. It is crazy that we run a whole SaaS on top of this stuff and one line of code almost cost us a quarter of revenue. Has anyone else had a billing bug sneak through because monitoring was basically just vibes and hope?
1 comments
Log in to join the discussion
Log In1 Comment
nelson.vera21d ago
Who hasn't had a billing bug slip through because your monitoring was a guy named Dave who checks a dashboard on Mondays? Friday afternoon deploys to anything touching money should honestly require a blood oath and two witnesses. The real kicker is always that the system works fine until it doesn't, and then you find out your entire safety net was hope and a prayer. At least you got a daily check out of it, which is more than most teams do before the fire. Just wait until that check itself silently breaks someday and nobody notices for a month.
7