> .... but he thinks there needs to be a way for the virtual filesystem (VFS) layer to slow down writers. Neil Brown had told Lever that there already was a mechanism to do that.....

The VFS (or maybe the MM) estimates the throughput of each "backing device" and throttles writes when they risk using more than a fair share of the dirty threshold for that one backing device (slower devices get a smaller share). This, as Jan mentioned, works nicely for local applications.

It works for nfsd threads too in that they will be throttled, but they aren't the source of the problem, they are the messenger. If the NFS client keeps sending WRITE requests, more and more nfsd threads will be throttled trying to write, until we run out of threads. With the latest kernels we set a high limit and dynamically grow the thread pool to this limit, so this can be lots of threads each with (e.g.) 1MB of data being written. Once we run out of threads the receive-queue on the socket fills up, and if the client used "nconnect" there might be 16 sockets each of which will have a full receive queue. Then the xmit queues on the client will fill up. Then the applications on the client will start being throttled....

Or maybe the client-side applications will be throttled earlier. Data that hasn't been acknowledged as "safe" the server is accounted as dirty data on the client and should throttle the client application based on measured throughput.

So if the server has a larger dirty-threshold than the sum of the dirty thresholds for all the active clients, then the client applications should get throttled before the server hits the threshold and starts throttling nfsd threads. But if the clients have a larger total dirty threshold, then they will swamp the server's dirty pool and I guess that creates the situation that Chuck sees as problematic.

This feels a lot like "buffer bloat" which affected wifi and similar network links until it was fixed. As I understand it, the various buffers were much bigger than the link could handle in a "reasonable" time, much like the situation where the clients have more total buffer space than the server and generate more load than it can handle. The solution for buffer bloat was (I think) to better estimate end-to-end throughput and limit the total outstanding queue size. Possibly a similar solution would help NFS.

The current throttling is based on a number of bytes or fraction of available memory size. I have occasionally wondered if it would be better to limit it based on time-to-flush. As a throughput estimate is already available, we could limit the number of dirty pages for each "bdi" to the amount of memory which can be flushed in N seconds. When I first looked into the dirty threshold many years ago it was because NFS clients would take a very long to time to fsync a file, and it needs to do this to respond to "stat" - because to get an accurate mtime, it needs to flush pending writes, then ask the server what the timestamp is. So "ls -l" would take a very long time. The work-around was to reduce the dirty-threshold substantially. I really wanted to set a "maximum time that stat() is allowed to take" but I had to set a "maximum number of dirty bytes". If throttling was time-to-flush based, I would have been able to have more meaningful control...