Loading…
22
Our billing webhook fired 4,000 times in one night and I found out at 2am
I was half asleep in my kitchen in Denver when PagerDuty lit up my phone, so I just unplugged the webhook endpoint and let the queue pile up until morning. Anyone else ever fix a live incident by just turning the thing off until sunrise?
1 comments
Log in to join the discussion
Log In1 Comment
rileygarcia28d ago
Back in 2021 we had a Stripe endpoint go crazy over a bad deploy and I disabled it for six hours while the queue backed up. The tricky part was replaying everything in the right order afterward, so I ended up sorting the backlog by timestamp before letting it process. Turning it off saved us from double charging anyone, but plan the replay before you flip it back on.
2