Completed -
The overview tiles, charts, and subscriber lists features have caught up with the latency. All the systems are now in operational state. We are marking the incident as completed but will continue monitoring all the systems.
Nov 23, 19:51 UTC
Update -
We are continuing to verify the maintenance items.
Nov 23, 15:12 UTC
Verifying -
Everything is running smoothly. We are still waiting for replication to our analytical databases to catch up for the overview tiles, charts, and subscriber lists. However, everything is functioning normally.
Additionally, we are replaying requests that failed to ensure all traffic is captured.
Nov 23, 15:11 UTC
Update -
Things are looking much better now - integration events and our SDK are now fully back to normal.
The overview tiles, charts, and subscriber lists features are functioning, but they are currently lagging behind. Once their data is up to date we will post another update.
Nov 23, 14:04 UTC
Update -
Integration event dispatching has resumed.
Nov 23, 13:52 UTC
Update -
Service reliability has mostly recovered. We are still monitoring things before giving an all clear.
For apps using the RevenueCat SDK, things should be operating normally now.
Nov 23, 12:13 UTC
Update -
We think we have found the root cause and are currently attempting to restore service.
For apps using RevenueCat, cached entitlements are fine (so subscribers have access to their content), but purchases that go through on Google / Apple currently don’t immediately gain access.
Those purchases will process when the database catches up. Subscribers trying to update information in-app will experience the same delayed processing. Lastly, until the performance issue is resolved your RC dashboard will be unavailable.
Suffice to say that we’re doing our utmost to get performance back to the expected level, and will do (and publish) a full retro on the incident once it resolves.
Nov 23, 11:41 UTC
Update -
We are still working on mitigating the performance issues with the new database, performance is still impacted.
Nov 23, 10:22 UTC
Update -
Extreme DB load in the new database is causing overload in the API.
Nov 23, 09:57 UTC
Update -
We have enabled maintenance mode. RevenueCat dashboard is unavailable, meaning you will not be able to create or update any apps, integrations or settings.
Nov 23, 09:07 UTC
In progress -
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Nov 23, 07:54 UTC
Scheduled -
On November 23th, we will be performing a major version upgrade on our main database cluster.
During this time, the RevenueCat dashboard will be unavailable, meaning you will not be able to create or update any apps, integrations or settings. You should also expect delivery delays in all integrations, including webhooks. Note that integration timestamps, such as purchase time, will still be accurate but the delivery time will be delayed.
Additionally, during the maintenance period, sending receipts to RevenueCat will temporarily fail for about 1-2 minutes. We will be capturing all traffic and replaying it afterwards so there will be no data loss. Existing installs of apps using RevenueCat will have no impact in terms of accessing/unlocking entitlements.
FAQ
Q: How long will receipt processing be unavailable?
A: Only 1-2 minutes. Any receipts received during this short time will be replayed, so no data loss is expected.
Q: Will RevenueCat's API be down?
A: No, only endpoints that cause updates to subscribers (such as POST /receipts) will be unavailable for 1-2 minutes. Other endpoints that simply read data (such as GET /subscribers) will be operational.
Q: Will existing customers of my app be affected?
A: Only new customers who attempt to make purchases during the 1-2 minutes that receipt processing is unavailable will have their purchases delayed. Customer who take actions that cause updates in RevenueCat (such as updating their email in subscriber attributes) will also be affected during the 1-2 minutes.
Nov 23, 07:54 UTC
