All metrics have been stable for some time now. Last seen data is being served as normal and all service is fully restored. This concludes the incident. Thank you for your patience and understanding.
Posted Mar 18, 2024 - 05:24 UTC
Update
Last seen data is now back to normal. Updates are being stored, and the information is read back for the appropriate requests. We are doing some final checks and monitoring before declaring the incident as resolved.
Posted Mar 18, 2024 - 04:41 UTC
Update
Engineers are still in the process of reading last seen data from the new cluster, but last seen data is now being written.
Posted Mar 18, 2024 - 04:06 UTC
Update
The new cluster is online; engineers are slowly ramping up traffic to it and verifying correctness and load.
Posted Mar 17, 2024 - 23:35 UTC
Monitoring
Unfortunately in the process of trying to recover last seen data during the previous incident, the cluster was discovered not to be able to handle the load.
Engineers have worked to bring last seen data into a brand new cluster with increased capacity that will be sufficiently stable for the longer term, and will soon be able to begin using it for last seen data reads and writes.
Posted Mar 17, 2024 - 23:18 UTC
Identified
We are currently working to make the cluster that holds last seen data fully operational.
Posted Mar 17, 2024 - 22:15 UTC
Update
We are continuing to investigate this issue.
Posted Mar 17, 2024 - 21:01 UTC
Investigating
Last seen data may be missing from some responses and outdated in others. Our engineers are working on the cluster that stores this data to recover this functionality fully.