fio Benchmarking Cookbook: Stop Using dd, Start Measuring What Matters

dd if=/dev/zero of=/tmp/test bs=1M count=1000 tells you almost nothing useful about your storage. It measures sequential write throughput of zero-filled data to a single-threaded userspace buffered I/O path, on a filesystem that may have deduplicated or compressed it away entirely, in a cache-warm scenario that won’t represent any real workload. If this is how you’ve been benchmarking storage, this post is for you.

fio — the Flexible I/O Tester by Jens Axboe (the Linux block layer maintainer and author of io_uring) — is the tool that replaces dd, bonnie++, iozone, and your homemade Bash scripts. It can simulate virtually any I/O pattern with precision: sequential or random, reads or writes or mixes, direct or buffered, synchronous or asynchronous, with any block size, queue depth, and concurrency. This post is a practical cookbook: the job files you actually need, how to read the output, and how to avoid benchmarking fiction.

Why this matters

Storage performance is multi-dimensional. A drive that delivers 7 GB/s sequential reads might deliver 50k random 4k write IOPS, or 200k — a 4× difference that means completely different things about the suitability for different workloads. A NAS that looks fast on dd might fall over at queue depth 32 with mixed read/write. The question is never “how fast is this disk?” — it’s “how fast is this disk under the workload I care about?”

And that matters because:

• Database performance is dominated by random 4k–16k sync writes (WAL) and random reads (index lookups).
• Log ingestion is sequential 64k–1M async writes at moderate concurrency.
• ML training is large sequential reads at deep queue depth.
• VM host storage is a chaotic mix at small block sizes.

Measuring one doesn’t predict the others. fio lets you measure each.

Installing and orienting

fio is packaged on every mainstream distro:

1
2
3

dnf install fio        # RHEL/Fedora
apt install fio        # Debian/Ubuntu
zypper install fio     # openSUSE

Check version:

1
2

fio --version
# fio-3.36 (or newer — features change meaningfully between versions)

Modern fio supports io_uring, which matters for high-end NVMe testing. If you’re on fio 3.19+, you have it. If you’re on something ancient, update.

The rest of this post assumes you’re running as root or have permission to read/write raw devices. Always benchmark on a disk or filesystem you can afford to have corrupted — fio can write to raw devices and will happily destroy filesystems if you point it at the wrong path.

The core concepts

A fio run consists of jobs. Each job has an I/O pattern (read/write/rw/randread/randwrite/randrw), a block size, a queue depth, a number of threads/processes, and a target (file or device). You can define jobs on the command line or in a config file.

Key parameters that change the meaning of your results:

• rw: the I/O pattern. read, write, randread, randwrite, rw (50/50 mixed sequential), randrw (50/50 mixed random).
• bs: block size. 4k for IOPS-style tests, 1M for throughput.
• iodepth: how many I/Os can be in flight at once. 1 is synchronous-ish; 32 is typical for async workloads; 128+ stresses high-end NVMe.
• numjobs: how many concurrent threads/processes. Multiplies with iodepth.
• ioengine: how fio submits I/O. libaio (older async), io_uring (modern async), sync (simple), psync (preadv/pwritev), mmap.
• direct=1: bypass the page cache. Always set this for storage benchmarks unless you’re specifically measuring caching.
• size: total I/O per job.
• runtime: cap the benchmark duration.
• time_based: run for runtime regardless of whether size is reached.

The first rule: always use direct=1

Without O_DIRECT, you’re measuring the Linux page cache. The page cache is fast. Your SSD is slower than the page cache. If you don’t bypass it, you’ll measure RAM speed, not disk speed, on reads — and on writes you’ll measure the rate at which dirty pages accumulate, not the rate at which the disk can write them.

Every cookbook recipe below uses --direct=1. If you remove it to “simulate real workloads”, you are measuring nothing useful and worse, you’ll get wildly inconsistent results across runs depending on how warm the cache was.

The exception: you explicitly want to measure the filesystem’s buffered I/O path, e.g., “how fast can an application that doesn’t use O_DIRECT actually push data?” In that case, set direct=0 and ioengine=psync, but recognize that you’re measuring a system, not a disk.

Recipe 1: Baseline random 4k read IOPS

The single most-referenced benchmark number for a drive. This tells you how the drive performs on small-random-read workloads (databases, typical filesystem metadata, tons of small-file web serving).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# randread-4k.fio
[global]
ioengine=io_uring
direct=1
randrepeat=0
norandommap=1
time_based=1
runtime=60
ramp_time=10
group_reporting=1
filename=/dev/nvme0n1

[randread-4k-qd32]
rw=randread
bs=4k
iodepth=32
numjobs=4

Run it: fio randread-4k.fio.

Explanation of every setting:

• ioengine=io_uring: modern async. Falls back to libaio on kernels without io_uring.
• randrepeat=0: don’t pre-seed the RNG with the same value every job (avoids artificially repeated access patterns).
• norandommap=1: don’t track which blocks have been read — saves memory on large devices and lets fio hit the same block more than once in random patterns, which better approximates real workloads.
• time_based=1 + runtime=60: run for 60 seconds regardless of data volume.
• ramp_time=10: ignore the first 10 seconds. Lets caches and queues stabilize.
• group_reporting=1: aggregate multiple jobs into one summary.
• numjobs=4 + iodepth=32: 128 outstanding I/Os total. Enough to saturate most NVMe; not enough to bottleneck the queue.

Output to care about:

  read: IOPS=412k, BW=1610MiB/s (1688MB/s)(94.3GiB/60001msec)
    slat (usec): min=1, max=87, avg= 2.14, stdev= 0.98
    clat (usec): min=15, max=2840, avg=308.47, stdev=112.33
     lat (usec): min=16, max=2841, avg=310.67, stdev=112.42
    clat percentiles (usec):
     |  50.00th=[  293],  95.00th=[  498],  99.00th=[  734],
     | 99.50th=[  832], 99.90th=[ 1074], 99.95th=[ 1188],
     | 99.99th=[ 1549]

Read it like this:

• IOPS: 412,000 random 4k reads/sec. This is your primary number.
• BW: 1.6 GB/s — derived from IOPS × block size.
• slat (submission latency): how long fio spent getting the I/O into the kernel. Tiny on modern kernels.
• clat (completion latency): the real latency number. How long the kernel/device took to complete the I/O.
• lat = slat + clat: total user-visible latency.
• Percentiles: 99.99th = 1.5 ms means 1 in 10,000 I/Os took over 1.5 ms. This is the tail latency number that matters for user-facing systems.

Average latency without tail percentiles is close to useless. A drive that averages 300 µs but has a 99.9th percentile of 50 ms will make databases feel terrible even though the average looks fine.

Recipe 2: Random 4k write IOPS (with sustained sync)

This is the hard test for any drive. Enterprise SSDs maintain steady-state random write performance for hours. Consumer SSDs peak briefly, then collapse when the SLC cache fills, then drop to QLC/TLC write rates. If you’re spec’ing a drive for a write-heavy workload, this is the number that matters.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

# randwrite-4k.fio
[global]
ioengine=io_uring
direct=1
randrepeat=0
norandommap=1
time_based=1
runtime=600
ramp_time=30
group_reporting=1
filename=/dev/nvme0n1
fsync=1

[randwrite-4k-qd32]
rw=randwrite
bs=4k
iodepth=32
numjobs=4

Run for 10 minutes minimum for SSDs. The first minute will show consumer-drive peak numbers; the last minute shows the sustained truth. Watch iostat -xz 1 in another terminal while it runs — you’ll see write bandwidth drop as the drive’s caching tier fills.

fsync=1 forces an fsync after every write, which approximates synchronous write workloads (PostgreSQL WAL). Remove it if you want pure async write throughput.

Warning: this writes over /dev/nvme0n1 entirely. You will destroy the filesystem. Use a scratch device, or write to a file on a filesystem (filename=/mnt/scratch/fio-testfile + size=20G).

Recipe 3: Sequential throughput

For sequential workloads (backups, media, bulk data), large-block throughput is the number:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

# seq-throughput.fio
[global]
ioengine=io_uring
direct=1
time_based=1
runtime=60
ramp_time=5
group_reporting=1
filename=/mnt/scratch/fio-testfile
size=40G

[seq-read-1m-qd16]
rw=read
bs=1M
iodepth=16
numjobs=1
stonewall

[seq-write-1m-qd16]
rw=write
bs=1M
iodepth=16
numjobs=1
stonewall

stonewall makes fio run the jobs sequentially rather than in parallel, so you get a clean read number and a clean write number. Without it, fio runs all jobs concurrently.

Sequential 1M at QD 16 is the workload that advertises the drive’s marketed “up to X GB/s” numbers. If your measurement is dramatically below spec, check PCIe lane width (lspci -vvv | grep -i width), NUMA locality, thermal throttling (nvme smart-log /dev/nvme0), and whether you’re on the expected generation.

Recipe 4: The “real database” test

A more realistic approximation of a PostgreSQL-like workload: random reads from a working set, random writes with fsync for the log, at a read/write ratio typical of OLTP:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

# database-mix.fio
[global]
ioengine=io_uring
direct=1
randrepeat=0
norandommap=1
time_based=1
runtime=300
ramp_time=30
group_reporting=1
directory=/mnt/db-scratch

[random-reads]
rw=randread
bs=8k
iodepth=16
numjobs=8
size=100G

[synced-writes]
rw=randwrite
bs=16k
iodepth=4
numjobs=2
fsync=1
size=20G

Both jobs run concurrently (no stonewall), which simulates database query traffic competing with WAL flush activity. Look at both the IOPS numbers and the write-side latency percentiles — WAL tail latency governs commit latency, and commit latency governs transaction throughput.

Recipe 5: NFS and network filesystem benchmarking

Network filesystems behave differently. Cache behavior, attribute caching, and client-side concurrency all matter. A useful test against an NFS mount:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

# nfs-test.fio
[global]
ioengine=libaio
direct=1
randrepeat=0
time_based=1
runtime=120
ramp_time=10
group_reporting=1
directory=/mnt/nfs

[many-small-files]
rw=randwrite
bs=64k
iodepth=8
numjobs=16
size=100M
nrfiles=50
file_service_type=random

nrfiles=50 tells fio to spread writes across 50 files. file_service_type=random picks files at random (rather than round-robin), which approximates how many small-file workloads actually behave.

For NFS, also test with ioengine=sync and direct=0 — many real NFS clients use the page cache, and you’ll see very different numbers. NFS sync-write performance depends heavily on wsize, rsize, async/sync export options, and whether you have pNFS.

Recipe 6: Saturation testing (what does it take to break this?)

Find the point at which the drive’s latency curve hockey-sticks. This is the real “how much load can this handle” number:

1
2
3
4
5
6
7

for qd in 1 2 4 8 16 32 64 128 256; do
    echo "=== QD=$qd ==="
    fio --name=qd-sweep --filename=/dev/nvme0n1 --rw=randread --bs=4k \
        --direct=1 --ioengine=io_uring --runtime=30 --time_based \
        --iodepth=$qd --numjobs=1 --group_reporting --minimal \
        | awk -F';' '{print "IOPS:", $8, "lat_avg:", $40, "lat_99.9:", $76}'
done

--minimal produces semicolon-separated output that’s easy to parse; the column numbers depend on fio version, so verify with your version. Plot IOPS vs QD and latency vs QD — you’ll see IOPS rise and flatten, latency rise linearly then exponentially. The “knee” is your real working point.

Interpreting results: the numbers that lie

“My drive hits 7 GB/s reads in spec but only 4 GB/s in my test”

Usual causes, in order:

1. PCIe lane mismatch. A Gen 4 x4 drive in a Gen 3 slot caps at ~3.5 GB/s. lspci -vvv shows LnkSta: Speed 8GT/s, Width x4 — 8 GT/s is Gen 3, 16 GT/s is Gen 4, 32 GT/s is Gen 5.
2. Single-thread bottleneck. Consumer drives need QD 16+ with multiple threads to saturate. Up numjobs and iodepth.
3. Not enough CPU. Reading 7 GB/s from NVMe takes real CPU. Pin fio to the NUMA node containing the drive (numactl -N 0 fio ...) and verify CPU isn’t saturated.
4. Filesystem overhead. Test against /dev/nvme0n1 directly to remove the FS; the gap to the filesystem-tested number is your FS overhead.

“Writes are faster than reads”

Usually the page cache (remember: use direct=1). Occasionally the drive’s DRAM cache absorbing writes while reads still hit flash. Sustain the test for 10+ minutes and the number normalizes.

“Results vary wildly run-to-run”

• Thermal throttling on M.2 NVMe without heatsinks.
• Background GC on SSDs — run fstrim first and give the drive idle time between runs.
• Co-tenant noise on cloud VMs. Nothing you can do about this; average over many runs.
• Insufficient ramp_time. Always give fio at least 10 seconds to reach steady state.

The ultimate fio invocation

If you want one general-purpose command that you can memorize, this is close:

1
2
3
4

fio --name=test --filename=/dev/nvme0n1 \
    --rw=randread --bs=4k --direct=1 --ioengine=io_uring \
    --iodepth=32 --numjobs=4 --group_reporting \
    --time_based --runtime=60 --ramp_time=10 --randrepeat=0

Change rw=, bs=, and target. That’s your starter kit.

Output formats for automation

fio --output-format=json produces JSON. Pipe to jq:

1
2

fio job.fio --output-format=json \
    | jq '.jobs[0] | {iops: .read.iops, bw: .read.bw, p99: .read.clat_ns.percentile."99.000000"}'

fio --output-format=terse --minimal produces semicolon-separated fields — good for CSV ingestion but positional and fragile across versions.

For continuous benchmarking, consider fio-plot (a community tool) or feed JSON output into Grafana via a script.

Common mistakes

1. Not using direct=1. The single most common mistake. Measures the page cache.
2. Too-short runs. 10 seconds tells you nothing about sustained performance.
3. Single job, single queue depth. Won’t saturate modern NVMe, produces misleading “slow drive” results.
4. Running on a new drive without preconditioning. First 50% of writes on a fresh SSD are free; after that you’re writing into dirty flash. For steady-state numbers, pre-fill the drive.
5. Comparing dd numbers to fio numbers. Stop doing this.
6. Benchmarking on a file much smaller than the drive. Fits in cache. Use at least 2× RAM size, or test against raw devices.
7. Not pinning NUMA. On multi-socket boxes, cross-NUMA I/O is measurably slower. numactl -N <node> the fio process.
8. Ignoring tail latency. p99 and p99.9 matter more than average for most real workloads.

When fio is not the right tool

• You want to measure application-level performance. Use the application’s own benchmark (pgbench, sysbench, ycsb).
• You want to model bursty traffic. fio can do this with thinktime=, but application-level replay tools are usually easier.
• You want to measure the full stack including network and protocol. For S3/Ceph/object stores, use warp or s3-benchmark.

A two-line summary you can put on a sticky note

Use --direct=1 --ioengine=io_uring --time_based --runtime=60 --ramp_time=10 --group_reporting. Report IOPS and 99.9th percentile latency. Sustain for 10 minutes on SSDs to get steady-state.

That’s fio. Learn the 10 parameters that matter, build a small library of .fio files for your workloads, and you’ll never reach for dd again.