Observation and diagnostics · Intermediate

Reading disk I/O with iostat

Read device request rates, delay and activity over matched windows and relate them to application latency and storage layers.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Separate device activity from application delay
  2. Collect interval reports
  3. Combine latency, queueing and throughput
  4. Interpret virtual and parallel devices carefully
  5. Ask Logeia about this articleCheck your understanding

Separate device activity from application delay

iostat is a sysstat tool for observing Linux device I/O activity over time. It helps identify which devices are receiving work and how request rates, throughput and waiting change during a slowdown. It does not directly measure the application’s response time. Page cache, filesystem behavior and network or virtual storage layers can separate application reads from physical device operations. Establish which layer a reported device represents before correlating it with a database volume or an application directory that happens to be slow.

Collect interval reports

For example, iostat -y -xz 1 5 requests extended reports at one-second intervals and suppresses the initial since-boot report. Check option support and field names for the installed sysstat version. Fields such as r/s, w/s, rkB/s and wkB/s describe rates, not raw cumulative counts. The command is an observation example; no measured output is asserted here. Resolve device mapping first, especially with logical volumes and virtual disks, so that statistics from an upper logical device are not mistaken for independent work on another physical device.

sh
iostat -y -xz 1 5

Combine latency, queueing and throughput

Await represents average request completion delay including time queued and serviced; it is not simply the time spent accessing the storage medium. Compare it with queue measures such as aqu-sz and the request rate. If similar request volume accompanies increasing queue depth and delay, the device path is a plausible investigation target. That relationship is an interpretation aid rather than proof of hardware failure. Average latency can also conceal a small fraction of very long requests, so an application’s tail latency can change without a proportionally dramatic change in the device-wide average.

Interpret virtual and parallel devices carefully

Do not interpret high %util as a universal proof that every storage device has reached its complete capacity. Parallel SSDs and virtualized paths can behave differently depending on request sizes and concurrent operations. Conversely, a low average does not establish that every individual request was prompt. Align application latency, CPU and memory observations with the device reports, and use more focused tracing where necessary. Running a synthetic load generator is a separate experiment with its own operational impact; an observational iostat session does not by itself authorize saturating a shared production volume.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Understand the systems behind the web · STEP 14

Choose a learning path