NFS async thread tuning (Managing NFS and NIS, 2nd Edition)

Book Home

Previous

Next

18.5. NFS async thread tuning

Early NFS client implementations provided biod user-level daemons in order to add concurrency to NFS operations. In such implementations, a client process performing an I/O operation on a file hands the request to the biod daemon, and proceeds with its work without blocking. The process doesn't have to wait for the I/O request to be sent and acknowledged by the server, because the biod daemon is responsible for issuing the appropriate NFS operation request to the server and to wait for its response. When the response is received, the biod daemon is free to handle a new I/O request. The idea is to have as many concurrent outstanding NFS operations as the server can handle at once, in order to accelerate I/O handling. Once all biod daemons are busy handling I/O requests, the client-side process generating the requests has to directly contact the NFS server and block awaiting its response.

For example, a file read request generated by the client-side process is handed to one biod daemon, and the rest of the biod daemons are asked to perform read-ahead operations on the same file. The idea is to anticipate the next move of the client-side application, by assuming that it is interested in sequentially reading the file. The NFS client hopes to avoid having to contact the NFS server on the next I/O request by the application, by having the next chunk of data already available.

Solaris, as well as other modern Unix kernels support multiple threads of execution without the need of a user context. Solaris has no biod daemons, instead it uses kernel threads to implement read-ahead and write-behind, achieving the same increased read and write throughput.

The number of read-aheads performed once the Solaris client detects a sequential read pattern is specified by the kernel tunable variables nfs_nra for NFS Version 2 and nfs3_nra for NFS Version 3. Solaris sets both values to four read-aheads by default. Depending on your file access patterns, network bandwidth, and hardware capabilities, you may need to modify the number of read-aheads to achieve optimal use of your resources. For example, you may find that this value needs to be increased on Gigabit Ethernet, but decreased over ISDN. To reduce the number of read-aheads over a low bandwidth connection, you can add the following lines to /etc/system on the NFS client and reboot the system:

set nfs:nfs_nra=2
set nfs:nfs3_nra=1

When running over a high bandwidth network, make sure not to set these values too high above their default, not only will sequential read performance not improve, but the increased memory used by the NFS async threads will ultimately degrade overall performance of the system.

If nfs3_nra is set to four, and if you have two processes reading two separate files concurrently over NFSVersion 3, the system by default will generate four read-aheads triggered by the read request of the first process, and four more read-aheads triggered by the read request of the second process for a total of eight concurrent read-aheads. The maximum number of concurrent read-aheads for the entire system is limited by the number of NFS async threads available. The kernel tunables nfs_max_threads and nfs3_max_threads control the maximum number of active NFS async threads active at once per filesystem.

By default, a Solaris client uses eight NFS async threads per NFS filesystem. To drop the number of NFS async threads to two, add the following lines to /etc/system on the NFS client and reboot the system:

set nfs:nfs_max_threads=2
set nfs:nfs3_max_threads=2

After rebooting, you will have reduced the amount of NFS read-ahead and write-behind performed by the client. Note that simply decreasing the number of kernel threads may produce an effect similar to that of eliminating them completely, so be conservative.

Be careful when server performance is a problem, since increasing NFS async threads on the client machines beyond their default usually makes the server performance problems worse. The NFS async threads impose an implicit limit on the number of NFS requests requiring disk I/O that may be outstanding from any client at any time. Each NFS async thread has at most one NFS request outstanding at any time, and if you increase the number of NFS async threads, you allow each client to send more disk-bound requests at once, further loading the network and the servers.

Decreasing the number of NFS async threads doesn't always improve performance either, and usually reduces NFS filesystem throughput. You must have some small degree of NFS request multithreading on the NFS client to maintain the illusion of having filesystem on local disks. Reducing or eliminating the number of NFS async threads effectively throttles the filesystem throughput of the NFS client -- diminishing or eliminating the amount of read-ahead and write-behind done.

In some cases, you may want to reduce write-behind client requests because the network interface of the NFS server cannot handle that many NFS write requests at once, such as when you have the NFS client and NFS server on opposite sides of a 56-kbs connection. In these radical cases, adequate performance can be achieved by reducing the number of NFS async threads. Normally, an NFS async thread does write-behind caching to improve NFS performance, and running multiple NFS async threads allows a single process to have several write requests outstanding at once. If you are running eight NFS async threads on an NFS client, then the client will generate eight NFS write requests at once when it is performing a sequential write to a large file. The eight requests are handled by the NFS async threads. In contrast to the biod mechanism, when a Solaris process issues a new write requests while all the NFS async threads are blocked waiting for a reply from the server, the write request is queued in the kernel and the requesting process returns successfully without blocking. The requesting process does not issue an RPC to the NFS server itself, only the NFS async threads do. When an NFS async thread RPC call completes, it proceeds to grab the next request from the queue and sends a new RPC to the server.

It may be necessary to reduce the number of NFS requests if a server cannot keep pace with the incoming NFS write requests. Reducing the number of NFS async threads accomplishes this; the kernel RPC mechanism continues to work without the async threads, albeit less efficiently.