Loading…
15
Our webhook retry logic dropped 1,200 billing events during the Tuesday deploy and I'm still not sure if manual backfill or a dead letter queue is the right fix
Last Tuesday around 2am our deploy pushed a bad change to the retry logic and around 1,200 billing events never got delivered to the customer workspaces. We caught it at 9am when a CS rep in Austin flagged that three accounts were missing usage data. I spent most of the day replaying events from the raw logs and got maybe 80% back, but the rest are gone unless we go back to the source system. Part of me thinks we should just build a dead letter queue and move on, but the other part thinks we need manual backfill plus better alerting on delivery failures. What do you all do for webhook reliability, and do you trust a DLQ enough to skip the manual checks?
0 comments
Log in to join the discussion
Log In0 Comments
No comments yet
Be the first to share your thoughts on this discussion.