TL;DR
Around 15:00 UTC we learned that all images were broken on Buttondown. This included viewing images within the Editor, as well as seeing past images in the Archives.
Buttondown uses Cloudflare to host and optimize our images and assets. Due to an outage with their tooling, images were not loading correctly on Buttondown.
How did we detect the issue?
Unfortunately, we didn't have an automated system monitoring images. We learned about this issue from user reports, which is unacceptable.
How did we mitigate the issue?
At 16:00 UTC the team bypassed our image optimizer on Cloudflare, and pointed it to a new worker. This restored image visibility, although they were no longer optimized.
We checked in with our customers that reported the issues to confirm that they were seeing the fix. We also continued to monitor until Cloudflare marked their outage as resolved at 21:34 UTC.
On June 11th at 23:15 UTC we reverted back to the image optimizer.
How will we prevent this from happening again?
- We've added alerting for our image optimizer tool so we can get advance warning when errors and outages occur.
- We're also adding tooling so that we can actively monitor metrics from the image optimizer.