As always, Jake did a great writeup, but this is a complex topic. A couple of small clarifications/corrections:

> In the last few months, Layton began, some new ioctl() commands were added...

You don't use ioctl() to set the io_cache_read/write modes. You just write() to them. Unfortunately, they take an integer, and not a string, so you have to peek at the source code to figure out the right values.

> The basic problem is that immediately performing the write from the thread that receives the client request blocks the thread until the write completes,

Not technically true. With the old DONTCACHE code, writes are written to the pagecache, and then the kernel immediately did a filemap_fdatawrite_range() on the range written. That submits them to the device, but doesn't wait for the I/O to complete. With multiple writing threads, they all compete to submit to the device queues, and that makes them perform terribly in aggregate.

With the new model, we just kick the flusher thread and have it do a proportional amount of writeback to what was written. The writers can get back to doing other stuff more quickly, and the writeback side of things is handled by the dedicated thread (or eventually, a set of threads), so there is less contention.