nhfsstone Command in Linux
The nhfsstone command is a specialized tool designed for benchmarking NFS (Network File System) performance by generating artificial loads with a specific mix of NFS operations. Its main function is to report the server's average response time in milliseconds per call and the load in terms of calls per second.
The program dynamically adjusts its calling patterns based on the client's kernel NFS statistics and the elapsed time, allowing load generation over a specified duration or a set number of NFS calls. However, since nhfsstone relies on kernel NFS statistics to track its progress, it cannot be used to measure the performance of non-NFS filesystems.
The tool manipulates files and directories to generate specific NFS operations in response to various system calls. It employs several techniques informed by the implementation details of the NFS client-side reference port. For instance, it uses long file names to bypass the kernel name lookup cache, ensuring that a stat(2) system call triggers an NFS lookup operation.
The mix of NFS operations can be customized using a mix file, which is the output of the nfsstat(8C) command. The mix percentages are based on the number of NFS calls rather than the percentages printed by nfsstat. Operations with a 0% mix will never be invoked by nhfsstone.
Despite a real server load mix potentially having nonzero calls for operations that show as 0% in the mix file, nhfsstone assumes these will have an insignificant effect on server response. Typically, nhfsstone should be provided with two or more test directories to use, defaulting to the current directory if none are specified.
To realistically simulate typical server loads, these test directories should be on different disks and partitions on the server. Each nhfsstone process looks for a directory <dir>/testdir<n>, where <n> ranges from 0 to nprocs - 1. If a directory already exists, it checks for the required test files; otherwise, it creates and populates the directory.
Table of Contents
Here is a comprehensive guide to the options available with the nhfsstone command −
- Syntax of nhfsstone Command
- nhfsstone Command Options
- Examples of nhfsstone Command in Linux
- Understanding NHFSSTONE Error Messages
- Potential Issues and Bugs
Syntax of nhfsstone Command
The following is the general syntax for the nhfsstone Command −
nhfsstone [options] <directories>
Where −
- [options] − Various command-line options to configure the behavior of nhfsstone.
- <directories> − One or more directories to use for the test. If no directories are specified, the current directory is used by default.
nhfsstone Command Options
The nhfsstone command offers a variety of options to customize its behavior when generating artificial NFS loads. Here's a detailed explanation of each option −
| Options | Description |
|---|---|
| -v | This option enables verbose output, providing more information about what nhfsstone is doing. |
| -t secs | Sets calls based on the given running time (in seconds) and the load. This option lets you specify the duration of the test in seconds. |
| -c calls | Specifies the total number of NFS calls to generate. The default value is 5000 calls. |
| -l load | Sets the load to generate in NFS calls per second. The default value is 30 calls per second. |
| -p nprocs | Determines the number of load-generating subprocesses to fork. The default is 7. This option helps to maximize the load generated by a single machine. On less powerful client machines (e.g., those with slow CPUs or limited memory), you might want to use fewer processes to avoid swapping. |
| -m mixfile |
Specifies the mix of NFS operations to generate using a mix file. The mix file format is the same as the output of the nfsstat(8C) program. You can create a mix file on a server by typing nfsstat -s > mixfile. The default mix of operations is: null 0%, getattr 13%, setattr 1%, root 0%, lookup 34%, readlink 8%, read 22%, wrcache 0%, write 15%, create 2%, remove 1%, rename 0%, link 0%, symlink 0%, mkdir 0%, rmdir 0%, readdir 3%, fsstat 1%. |
| -2 | -3 | -4 | Specifies the NFS version, similar to the nfsstat (8C) command. The default is NFSv3. The nhfsstone command uses the specified NFS statistics from /proc/net/rpc/nfs. |
Examples of nhfsstone Command in Linux
In this section, we'll look at various practical examples that should give you a comprehensive understanding of how to use the nhfsstone command effectively.
Basic Execution with Default Parameters
To establish a baseline for the NFS server's performance, you can simply run the following command −
sudo nhfsstone -d /mnt/nfs/testdir
This runs nhfsstone on the /mnt/nfs/testdir directory with default settings, giving you an overview of the server's performance under standard load conditions.
Specify the Number of NFS Calls
To increase the number of NFS calls and measure the server's stability over more extensive tests, you can use the following command −
sudo nhfsstone -d /mnt/nfs/testdir -c 1000
Simulate Multiple Clients
To simulate the behavior of multiple clients accessing the NFS server concurrently, you can run the following command −
sudo nhfsstone -d /mnt/nfs/testdir -p 8
This command uses 8 generator processes to simulate concurrent access, providing insights into how the server handles multiple simultaneous requests.
Run Tests on Multiple Exported Directories
To distribute the load across multiple NFS directories for a realistic access pattern, you can run the following command −
sudo nhfsstone -d /mnt/nfs/testdir1 -d /mnt/nfs/testdir2
This test accesses files in /mnt/nfs/testdir1 and /mnt/nfs/testdir2, simulating a workload across multiple exported directories.
Specify a Fixed Test Duration
To run the test for a fixed duration instead of a specific number of calls, use the following command −
sudo nhfsstone -d /mnt/nfs/testdir -t 60
This command runs the test for 60 seconds, helping you observe server performance over a specific time window.
Monitor Network Statistics During Tests
To check for network error and collision rates while running the test, you can use the following command −
sudo nhfsstone -d /mnt/nfs/testdir & netstat -i
The netstat -i command provides real-time insights into the network interface's performance during the nhfsstone test
Test Performance with a Custom NFS Operation Mix
To evaluate how different operation types (reads, writes, and attribute operations) affect server performance, you can simply use the following command −
sudo nhfsstone -d /mnt/nfs/testdir --read-percentage 50 --write-percentage 40 --attr-percentage 10
This test configures 50% of operations as reads, 40% as writes, and 10% as attribute-related operations, simulating a specific workload profile.
Generate High Load to Stress-Test the Server
To stress-test the server with a very high load, you can use the following command −
sudo nhfsstone -d /mnt/nfs/testdir -c 10000 -p 16
This test runs 10,000 calls with 16 processes, pushing the server to its limits and revealing its performance under maximum load.
Compare Results Across Server Configurations
To compare server performance before and after making changes to the server hardware or configuration, follow these steps −
Step 1 − Before the server upgrade, run the following command and save the results −
sudo nhfsstone -d /mnt/nfs/testdir -c 1000 > results_before.txt
Step 2 − After the server upgrade, run the same test and save the new results −
sudo nhfsstone -d /mnt/nfs/testdir -c 1000 > results_after.txt
Step 3 − Compare the results Use the diff command to compare the performance results −
sudo diff results_before.txt results_after.txto
This approach highlights any improvements or regressions in server performance due to the configuration changes.
Understanding NHFSSTONE Error Messages
Error handling is crucial when running benchmarks. Here's an overview and explanation of the error messages you might encounter when using the nhfsstone command −
| Tag | Description |
|---|---|
| illegal calls value | The calls argument following the -c flag on the command line is not a positive number. Ensure that you provide a valid positive number for the calls argument. |
| illegal load value | The load argument following the -l flag on the command line is not a positive number. Make sure you specify a valid positive number for the load argument. |
| illegal time value | The time argument following the -t flag on the command line is not a positive number. Provide a valid positive number for the time argument. |
| bad mix file | The mixfile file argument following the -m flag on the command line could not be accessed. Check if the mix file exists and has the correct permissions. |
| can't find current directory | The parent process couldn't find the pathname of the current directory. This usually indicates a permission problem. Ensure that the current directory is accessible and has the appropriate permissions. |
| can't fork | The parent couldn't fork the child processes. This typically results from a lack of resources, such as memory or swap space. Ensure your system has sufficient resources available. |
| can't open log file, can't stat log, can't truncate log, can't write sync file, can't write log, can't read log | A problem occurred during the creation, truncation, reading, or writing of the synchronization log file. The parent process creates the log file in /tmp and uses it to synchronize and communicate with its children. Verify that /tmp is writable and has sufficient space. |
| can't open test directory, can't create test directory, can't cd to test directory, wrong permissions on test dir, can't stat testfile, wrong permissions on testfile, can't create rename file, can't create subdir | A child process encountered issues while creating or checking the contents of its test directory. This is usually due to permission problems (e.g., the test directory was created by a different user) or a full filesystem. Verify directory permissions and ensure sufficient disk space. |
| bad mix format: unexpected EOF after 'nfs:', can't find 'calls' value, unexpected EOF after 'calls', can't find %d op values, unexpected EOF | A problem occurred while parsing the mix file. The expected format of the file is the same as the output of the nfsstat(8C) command when run with the -s option. Ensure the mix file is correctly formatted. |
| op failed: <operation> | One of the internal pseudo-NFS operations failed. The name of the operation (e.g., read, write, lookup) will be printed along with an indication of the nature of the failure. |
| select failed | The select system call returned an unexpected error. This could be due to resource limitations or other system-level issues. |
Potential Issues and Bugs
While nhfsstone is a powerful tool, certain pitfalls should be considered −
- Running on non-NFS filesystems − When running nhfsstone on a non-NFS filesystem, the program might run indefinitely because it relies on kernel NFS statistics to determine when enough calls have been made. Since nhfsstone uses NFS-specific statistics to monitor progress, it cannot correctly gauge completion on non-NFS filesystems, leading to potential infinite loops.
- High number of file descriptors − nhfsstone uses a large number of file descriptors. As a result, the kernel on the client may need to be reconfigured to increase the number of available file table entries. You might need to adjust the system's file descriptor limits to accommodate the high number of file handles nhfsstone requires during its operations.
- Handling SIGUSR1 Signal − Shell scripts that use nhfsstone must catch and ignore the SIGUSR1 signal. This signal is used to synchronize the test processes, and if not caught, the shell running the script will be terminated. The SIGUSR1 signal is used for inter-process communication within nhfsstone. If a shell script doesn't handle this signal, it might inadvertently kill the shell session, causing the script to fail.
Conclusion
The nhfsstone command is a valuable tool for benchmarking NFS servers. It was originally available via anonymous FTP from bugs.cs.wisc.edu and through Legato Systems, Inc. (IETF).
However, nhfsstone is considered outdated and may not be compatible with modern systems. As an alternative, you might consider using nfswatch, a tool that allows you to monitor NFS traffic in real-time.
By following the outlined practices and being mindful of the limitations, you can utilize nhfsstone or its alternatives to achieve meaningful insights into your NFS server's performance.