Resolved -
All metrics have been stable for some time now. Last seen data is being served as normal and all service is fully restored. This concludes the incident. Thank you for your patience and understanding.
Mar 18, 05:24 UTC
Update -
Last seen data is now back to normal. Updates are being stored, and the information is read back for the appropriate requests. We are doing some final checks and monitoring before declaring the incident as resolved.
Mar 18, 04:41 UTC
Update -
Engineers are still in the process of reading last seen data from the new cluster, but last seen data is now being written.
Mar 18, 04:06 UTC
Update -
The new cluster is online; engineers are slowly ramping up traffic to it and verifying correctness and load.
Mar 17, 23:35 UTC
Monitoring -
Unfortunately in the process of trying to recover last seen data during the previous incident, the cluster was discovered not to be able to handle the load.
Engineers have worked to bring last seen data into a brand new cluster with increased capacity that will be sufficiently stable for the longer term, and will soon be able to begin using it for last seen data reads and writes.
Mar 17, 23:18 UTC
Identified -
We are currently working to make the cluster that holds last seen data fully operational.
Mar 17, 22:15 UTC
Update -
We are continuing to investigate this issue.
Mar 17, 21:01 UTC
Investigating -
Last seen data may be missing from some responses and outdated in others. Our engineers are working on the cluster that stores this data to recover this functionality fully.
Mar 17, 19:51 UTC