Loading…
31
Pro tip: Our Stripe webhook retry storm ate 14k duplicate events in one night
Last Wednesday our billing webhooks started failing around 2am because a downstream API we call was timing out, and Stripe's retry logic kept hammering us for about 4 hours. We ended up with 14k duplicate events and roughly 300 customers got double usage alerts before anyone noticed. What fixed it was moving our webhook handler to a queue with idempotency keys stored in Redis for 24 hours, so repeats just get dropped instead of reprocessed. Cost us a weekend of cleanup but the queue change took an afternoon. Has anyone handled retry storms from a different angle, like capping retries on your side?
0 comments
Log in to join the discussion
Log In0 Comments
No comments yet
Be the first to share your thoughts on this discussion.