The maintenance is now finished. Our intention was to perform a minor upgrade to Aurora PostgreSQL 14.10 and upgrade the AWS instance type we use for our main database cluster. The version upgrade went as expected, but we ran into unexpected errors when trying to perform a failover to the new instance type.
During the maintenance period there were three periods of elevated error rates: * ~5:45 UTC - there were about 4 minutes of elevated error rates while the upgrade was being performed. This was estimated to be about 30 seconds during our trial run, but ended up being much longer for the real upgrade. * ~6:00 UTC - there were about 2 minutes of elevated error rates from an unrelated database performance issue where where writes are blocked on IO:XactSync. We are working with AWS to find the root cause of this and believe it is unrelated to the maintenance. * ~7:25 - ~7:50 UTC - once the database was stable we ended up in a situation where a pgbouncer cluster for read queries was still pointing to an incorrect instance. This resulted in a more severe outage where about 30% of posted receipts were failing and a 1.5% error rate across all endpoints.
All purchases that failed during the outage have been captured and are being replayed.
Posted Apr 10, 2024 - 07:30 UTC
Update
Scheduled maintenance is still in progress. We will provide updates as necessary.
Posted Apr 10, 2024 - 06:20 UTC
Update
For integration events, there was a brief unintended pause in dispatching from ~5 AM UTC to ~6 AM UTC. The issue has been fixed and all queued events are currently going out.
Posted Apr 10, 2024 - 06:03 UTC
In progress
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Posted Apr 10, 2024 - 05:30 UTC
Scheduled
We will be performing maintenance tasks on our database. We expect a brief service interruption (<1min) but we will enable RC fortress so app purchase flow is not affected
Posted Apr 04, 2024 - 12:10 UTC
This scheduled maintenance affected: API Uptime, Scheduled Data Exports, Event Dispatching and Dashboard (Customer Lists, Overview, Charts).