We’re back. Sorry for the interruption. Follow ups to come.
Posted Mar 16, 2023 - 02:59 UTC
Update
We figured it out. A database failover at an inconvenient moment caused a bad propagation of credentials which didn’t present itself until the AMI reboot.
We’re partially back and think we’ll be back up shortly.
Posted Mar 16, 2023 - 02:52 UTC
Update
We’ve determined that our we’re having issues connecting to our follower database and investigating.
Rolling back the AMI change had no effect.
Posted Mar 16, 2023 - 02:43 UTC
Update
Still working. We’re not sure what the origin of the issue is. We’re working to revert all the changes leading up to the issue.
Posted Mar 16, 2023 - 02:18 UTC
Update
Still working. Good news, we’ve tried a lot of things that haven’t worked so we must be getting close to finding the thing that does.
Posted Mar 16, 2023 - 01:47 UTC
Update
Still investigating. We updated the AMIs for our PGBouncer instance and believe the issue is related. We’re running down that lead now.
Posted Mar 16, 2023 - 01:36 UTC
Update
We made a database proxy configuration change that we believe is related to the issue and are working a reversion.
Posted Mar 16, 2023 - 01:27 UTC
Investigating
The dashboard and API are currently inaccessible. We are investigating the issue.