Our entire data sync went down for 12 hours because someone changed a single API permission in Salesforce and didn't tell us.

4 comments

4 Comments

Shouldn't your system have checks for that kind of change? A single permission shouldn't take down everything for half a day. Maybe the real issue is a lack of built-in alerts.

john_fisher2mo agoTop Commenter

We added alerts after our last outage, fixed it fast.

jakewhite2mo ago

Seriously, that's the whole point. Alerts are just noise if the system lets you break it that easy in the first place. You need to stop the bad change from going through, not just yell about it after. Building in those blocks is basic stuff, not some fancy feature.

juliarodriguez16d ago

Wait, a SINGLE permission change took your whole sync down for TWELVE hours? That's insane, I can't believe there wasn't a backup or a way to roll it back instantly. You guys must have been freaking out the whole time.