When you've got background processes competing with foreground processes, the correct answer is priority scheduling. Trying to solve these problems above the block layer is probably doomed to failure.

Unfortunately, block layer scheduling is a mess - there's a cgroup interface, which doesn't work for in-kernel users like background writeback, and there's a bio interface which has never as far as I can tell actually worked. And there's two (!) competing block IO schedulers - the real one, and blk-writeback-throttling, which is really a scheduler in disguise. The writeback throttling scheduler has the right design for something that wants to limit load to the device to keep latencies low (so fsync is fast), but it has no priority interface (and the interface it does have is one of the more insane things in the block layer).

On top of that, if the device is doing writeback caching, the signal you get for doing wbt-throttle style latency limiting is crap. I've already moved all internal writes in bcachefs to REQ_FUA to avoid causing queuing delays for foreground writes, but writeback probably needs to be FUA too.