~/ linuxnetwork.net

An In - Depth Guide to `iostat`: Monitoring System I/O Performance

In the world of system administration and performance tuning, having a clear understanding of Input/Output (I/O) performance is crucial. I/O operations involve reading from and writing to storage devices such as hard drives, solid - state drives (SSDs), and network - attached storage. Poor I/O performance can lead to slow application response times, decreased system throughput, and overall user dissatisfaction.

One of the most powerful and widely used tools for monitoring I/O performance in Linux and other Unix - like systems is iostat. iostat is part of the sysstat package, which provides a set of utilities for monitoring system performance. This blog post will take you through a comprehensive exploration of iostat, including its basic functionality, advanced usage, common practices, and best practices.

Table of Contents#

  1. Installation of iostat
  2. Basic Usage of iostat
  3. Understanding iostat Output
    • CPU Statistics
    • Device Statistics
    • Extended Device Statistics (-x)
  4. Advanced Usage
    • Specifying Intervals and Counts
    • Filtering Devices
    • Filtering and Display Options
    • JSON Output
  5. Common Practices
  6. Best Practices
  7. Conclusion
  8. References

Installation of iostat#

iostat is part of the sysstat package. The installation process may vary depending on your Linux distribution:

Ubuntu/Debian#

sudo apt-get update
sudo apt-get install sysstat

CentOS/RHEL#

sudo yum install sysstat

Fedora#

sudo dnf install sysstat

Basic Usage of iostat#

The simplest way to use iostat is to run it without any arguments:

iostat

This command will display a summary of CPU and I/O device statistics since the system was last booted.

Understanding iostat Output#

CPU Statistics#

The CPU section of the iostat output provides information about the overall CPU utilization. Here are some of the key columns:

  • %user: The percentage of CPU time spent in user mode.
  • %nice: The percentage of CPU time spent in user mode with low priority (nice).
  • %system: The percentage of CPU time spent in kernel mode.
  • %iowait: The percentage of CPU time spent waiting for I/O operations to complete. High %iowait values can indicate I/O bottlenecks.
  • %steal: The percentage of CPU time stolen from a virtual machine by the hypervisor.
  • %idle: The percentage of CPU time spent idle.

Device Statistics#

The device section of the output displays information about each I/O device. Some important columns include:

  • Device: The name of the device, such as sda or nvme0n1.
  • tps: Transfers per second. A transfer is an I/O request to the device.
  • kB_read/s: The number of kilobytes read from the device per second.
  • kB_wrtn/s: The number of kilobytes written to the device per second.
  • kB_read: The total number of kilobytes read from the device.
  • kB_wrtn: The total number of kilobytes written to the device.

Extended Device Statistics (-x)#

When using the -x flag, iostat displays additional columns that provide deeper insight into device performance:

  • r/s, w/s: Read and write requests completed per second (after merges).
  • rrqm/s, wrqm/s: Read and write requests merged per second before being sent to the device. Higher merge rates indicate the kernel is combining adjacent I/O requests, which improves efficiency.
  • areq-sz: The average size (in kibibytes) of I/O requests issued to the device. (In older versions, this was called avgrq-sz and expressed in sectors.)
  • aqu-sz: The average queue length of requests issued to the device. (In older versions, this was called avgqu-sz.) A consistently high value may indicate device saturation.
  • await: The average time (in milliseconds) for I/O requests to be served, including queue time and service time. High await values suggest I/O bottlenecks.
  • r_await, w_await: Average wait times for read and write requests separately.
  • %util: Percentage of elapsed time during which I/O requests were issued to the device. Values near 100% generally indicate device saturation for devices serving requests serially. Note: For devices serving requests in parallel, such as RAID arrays and modern NVMe SSDs, %util does not accurately reflect their performance limits — a device at 100% %util may still have significant headroom.

Advanced Usage#

Specifying Intervals and Counts#

To continuously monitor I/O performance, you can specify an interval and the number of reports. For example, to display reports every 5 seconds, with a total of 3 reports:

iostat 5 3

Filtering Devices#

If you only want to monitor a specific device, you can specify the device name as an argument. For example, to monitor the sda device:

iostat sda

Filtering and Display Options#

You can control which sections of the report are displayed using these flags:

  • -c: Display only CPU statistics.
  • -d: Display only device statistics.
  • -x: Show extended device statistics (includes additional columns described above).

For example, to get extended device statistics without CPU information:

iostat -xd

To display statistics in human-readable units (e.g., 1.0k, 1.2M) with the device name on the right side:

iostat -h

JSON Output#

For integration with monitoring systems or scripts, iostat can output statistics in JSON format using the -o JSON flag:

iostat -o JSON

Common Practices#

  • Regular Monitoring: Regularly run iostat to establish a baseline of normal I/O performance. This baseline can be used to detect abnormal behavior in the future.
  • Monitoring During Peak Load: Run iostat during peak usage hours to identify potential I/O bottlenecks that may not be apparent during normal operation.
  • Comparing Before and After Changes: If you make any changes to the system, such as adding new storage devices or changing I/O scheduling algorithms, run iostat before and after the changes to measure the impact.

Best Practices#

  • Use in Conjunction with Other Tools: iostat provides valuable I/O information, but it should be used in conjunction with other performance monitoring tools such as top, vmstat, and iotop for a more comprehensive view of system performance.
  • Analyze Trends: Instead of focusing on individual data points, look for trends in the iostat output over time. This can help you identify long - term issues and plan for future capacity upgrades.
  • Understand Your Workload: Different workloads have different I/O characteristics. For example, a database server may have a high read - to - write ratio, while a file server may have a more balanced I/O pattern. Understanding your workload can help you interpret the iostat output more effectively.
  • Suppress Inactive Devices: Use the -z flag to hide devices with no activity during the sample period, making the output easier to read on systems with many block devices.
  • Skip the Boot Report: When monitoring at intervals, use -y to omit the initial report covering the time since boot, which can skew your analysis with long-running averages.
  • Use Extended Statistics for Diagnosis: The default output is useful for a quick overview, but -x provides the columns (await, %util, aqu-sz) needed for meaningful performance diagnosis.

Conclusion#

iostat is a powerful and versatile tool for monitoring system I/O performance. By understanding its basic and advanced usage, as well as following common and best practices, you can effectively diagnose and troubleshoot I/O - related issues in your system. Whether you are a system administrator, a developer, or a performance analyst, iostat should be part of your toolkit for maintaining a high - performing system.

References#

2026-06