We’re back. Sorry for the interruption. Follow ups to come.
Mar 16, 02:59 UTC
We figured it out. A database failover at an inconvenient moment caused a bad propagation of credentials which didn’t present itself until the AMI reboot.
We’re partially back and think we’ll be back up shortly.
Mar 16, 02:52 UTC
We’ve determined that our we’re having issues connecting to our follower database and investigating.
Rolling back the AMI change had no effect.
Mar 16, 02:43 UTC
Still working. We’re not sure what the origin of the issue is. We’re working to revert all the changes leading up to the issue.
Mar 16, 02:18 UTC
Still working. Good news, we’ve tried a lot of things that haven’t worked so we must be getting close to finding the thing that does.
Mar 16, 01:47 UTC
Still investigating. We updated the AMIs for our PGBouncer instance and believe the issue is related. We’re running down that lead now.
Mar 16, 01:36 UTC
We made a database proxy configuration change that we believe is related to the issue and are working a reversion.
Mar 16, 01:27 UTC
The dashboard and API are currently inaccessible. We are investigating the issue.
Mar 16, 01:21 UTC