New database is deployed and seems to have alleviated the issue. Marking resolved for now but we're still watching closely. Will be working on deeper performance and monitoring enhancements over the next few days to get ahead of it the next time.
Posted Sep 02, 2021 - 16:25 UTC
Update
We've identified a few potential causes that will take time to address, in the mean time we're throwing money at the problem by upgrading our DB.
We'll see if this new capacity solves the degradation temporarily as we work on some code changes to address the underlying performance issues.
Posted Sep 02, 2021 - 16:02 UTC
Identified
We are currently working on fixing this issue, we'll post updates as the situation evolves.