At 2019-10-21 01:38 UTC the docs.rs website went down because no available disk space was left on the server hosting the application. Crate builds were failing since 2019-10-20 00:55 UTC due to the same reason.
docs.rs needs to store the built documentation on the filesystem before
uploading it to the database, and it does so in the
/opt/docs-rs-prefix/documentations directory. docs.rs never cleared that
directory though, so over time it started to increase its size until it caused
this outage. Code to periodically purge temporary directories was present, but
it was never configured to purge the one which caused the outage.
As the directory doesn’t contain any persistent data we cleared it and the web server was restarted. Once we were confident the situation was resolved all the crates that failed due to the outage were queued for a rebuild.
The increased disk usage was gradual over weeks, slowly reaching 100% and causing the outage. While monitoring systems were in place and recorded graphs of the increase, no alert was configured so nobody noticed the problem. We need to add alerts when disk usage reaches 90%, so the problem can be investigated and dealt with on time.
Crates started to fail to build a day earlier, and close to no builds were successfully completed since then. We need to setup alerts when most of the builds are failing: as we don’t have the necessary metrics at the moment to reliably alert we'll have to add extra instrumentation as well.
Our response was slower due to issues with our on-call rotation for the service. The primary contacts don’t have the level of access required to increase the disk space of the instance (the temporary fix that was investigated at first but discarded after the discovery nobody awake could do it), and the backup contacts don’t have any production access or expertise on docs.rs.
Unless otherwise noted all events happened on 2019-10-21, and all times are in UTC.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。