Yesterday Ubuntu released a security update for nginx, the well-known & widely used web server & reverse proxy, as part of their "security" update channel. All machines that have automatic security updates enabled (usually the "unattended upgrades" mechanism) dutifully installed the update over the last 24h.

Today we arrived at $DAYJOB to a lot of our customers having opened support tickets as their reverse proxies didn't work properly anymore. Judging by the bug report on Launchpad[1] server fleets all over the world face the same issue after upgrading.

Turns out they broke the ABI with their backport of the security fix into the packaged versions. Now that wouldn't be bad if they had also re-built all packages using said ABI & including versions of those packages in the same security update, but they didn't realize they needed to (= didn't realize they broke the ABI) in the first place, so they didn't do that. Hence tons of servers trying to run a mixture of now incompatible nginx ABIs. As of 1 hour ago Ubuntu has acknowledged this, bumped the criticality to "critical", and will address the issue by first reverting the change, releasing another security update (a "regression" update), and then figuring out how to implement the fix without breaking the ABI.

I'm not making a judgement here about static vs. dynamic linking as both prioritize different properties and concerns which simply cannot be fulfilled all at the same time. Would this particular issue have happened with static linking? Obviously not. Would have static linking helped in any way, shape or form with any of the user usual security updates that distros push out daily and which do NOT require re-building the whole world? Also obviously no.

[1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/2155992