ext4、xfs、zfs、btrfs 在 RAID6 下的性能测试

1. 单盘

1.1 创建文件系统

清理磁盘

格式化磁盘

挂载磁盘

1
2

mkdir -p /mnt/nvme0n1-test
mount /dev/nvme0n1 /mnt/nvme0n1-test

1.2 测试性能

4k 随机写

echo 3 > /proc/sys/vm/drop_caches
fio --name=4k_randwrite --rw=randwrite --bs=4k --ioengine=libaio --direct=1 \
  --numjobs=8 --iodepth=32 --size=20G --time_based --runtime=120 --group_reporting \
  --allow_mounted_write=1 --filename=/mnt/nvme0n1-test/fio-4k-randwrite.dat

Starting 8 processes
Jobs: 8 (f=8): [w(8)][100.0%][w=970MiB/s][w=248k IOPS][eta 00m:00s]
4k_randwrite: (groupid=0, jobs=8): err= 0: pid=1909379
  write: IOPS=235k, BW=919MiB/s (964MB/s)(108GiB/120001msec); 0 zone resets
    slat (usec): min=2, max=979550, avg=17.07, stdev=525.74
    clat (nsec): min=944, max=60651k, avg=1060067.20, stdev=832980.74
     lat (usec): min=20, max=980100, avg=1077.38, stdev=987.11
    clat percentiles (usec):
     |  1.00th=[  343],  5.00th=[  519], 10.00th=[  619], 20.00th=[  668],
     | 30.00th=[  709], 40.00th=[  791], 50.00th=[  922], 60.00th=[ 1057],
     | 70.00th=[ 1205], 80.00th=[ 1401], 90.00th=[ 1614], 95.00th=[ 1745],
     | 99.00th=[ 2376], 99.50th=[ 5014], 99.90th=[11076], 99.95th=[14222],
     | 99.99th=[27395]
   bw (  KiB/s): min=52536, max=1151376, per=100.00%, avg=946201.45, stdev=17317.50, samples=1904
   iops        : min=13134, max=287844, avg=236550.35, stdev=4329.38, samples=1904
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=0.14%
  lat (usec)   : 500=4.28%, 750=31.30%, 1000=20.26%
  lat (msec)   : 2=42.17%, 4=1.22%, 10=0.42%, 20=0.15%, 50=0.02%
  lat (msec)   : 100=0.01%
  cpu          : usr=10.95%, sys=34.24%, ctx=19512693, majf=0, minf=12967
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,28241228,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=919MiB/s (964MB/s), 919MiB/s-919MiB/s (964MB/s-964MB/s), io=108GiB (116GB), run=120001-120001msec

Disk stats (read/write):
  nvme0n1: ios=7795/28219461, merge=0/0, ticks=894/21461568, in_queue=21462462, util=99.74%

4k 随机读

echo 3 > /proc/sys/vm/drop_caches
fio --name=4k_randread --rw=randread --bs=4k --ioengine=libaio --direct=1 \
  --numjobs=8 --iodepth=32 --size=20G --time_based --runtime=120 --group_reporting \
  --filename=/mnt/nvme0n1-test/fio-4k-randread.dat

Starting 8 processes
Jobs: 8 (f=8): [r(8)][100.0%][r=1618MiB/s][r=414k IOPS][eta 00m:00s]
4k_randread: (groupid=0, jobs=8): err= 0: pid=1912551
  read: IOPS=413k, BW=1614MiB/s (1692MB/s)(189GiB/120003msec)
    slat (nsec): min=1918, max=23825k, avg=7052.22, stdev=10803.55
    clat (usec): min=16, max=35556, avg=611.04, stdev=554.43
     lat (usec): min=32, max=35568, avg=618.23, stdev=554.54
    clat percentiles (usec):
     |  1.00th=[   94],  5.00th=[  113], 10.00th=[  141], 20.00th=[  202],
     | 30.00th=[  273], 40.00th=[  351], 50.00th=[  449], 60.00th=[  562],
     | 70.00th=[  717], 80.00th=[  930], 90.00th=[ 1287], 95.00th=[ 1663],
     | 99.00th=[ 2507], 99.50th=[ 2900], 99.90th=[ 3916], 99.95th=[ 4490],
     | 99.99th=[10945]
   bw (  MiB/s): min= 1492, max= 1693, per=100.00%, avg=1615.94, stdev= 3.55, samples=1912
   iops        : min=382186, max=433586, avg=413680.06, stdev=907.93, samples=1912
  lat (usec)   : 20=0.01%, 50=0.01%, 100=2.24%, 250=24.86%, 500=27.75%
  lat (usec)   : 750=17.09%, 1000=10.61%
  lat (msec)   : 2=14.84%, 4=2.53%, 10=0.08%, 20=0.01%, 50=0.01%
  cpu          : usr=16.08%, sys=44.37%, ctx=17601497, majf=0, minf=5047
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=49583832,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=1614MiB/s (1692MB/s), 1614MiB/s-1614MiB/s (1692MB/s-1692MB/s), io=189GiB (203GB), run=120003-120003msec

Disk stats (read/write):
  nvme0n1: ios=49546789/2, merge=0/0, ticks=29364649/0, in_queue=29364648, util=100.00%

4k 随机读写

echo 3 > /proc/sys/vm/drop_caches
fio --name=4k_randrw --rw=randrw --rwmixread=70 --bs=4k --ioengine=libaio --direct=1 \
  --numjobs=8 --iodepth=1 --size=20G --time_based --runtime=120 --group_reporting \
  --allow_mounted_write=1 --filename=/mnt/nvme0n1-test/fio-4k-randrw.dat

Starting 8 processes
Jobs: 8 (f=8): [m(8)][100.0%][r=233MiB/s,w=100MiB/s][r=59.7k,w=25.7k IOPS][eta 00m:00s]
4k_randrw: (groupid=0, jobs=8): err= 0: pid=1917796
  read: IOPS=59.9k, BW=234MiB/s (245MB/s)(27.4GiB/120001msec)
    slat (usec): min=2, max=12515, avg= 7.15, stdev= 9.01
    clat (nsec): min=771, max=24656k, avg=112575.37, stdev=61675.06
     lat (usec): min=15, max=24667, avg=119.89, stdev=62.41
    clat percentiles (usec):
     |  1.00th=[   87],  5.00th=[   90], 10.00th=[   90], 20.00th=[   92],
     | 30.00th=[   93], 40.00th=[   95], 50.00th=[  103], 60.00th=[  109],
     | 70.00th=[  110], 80.00th=[  130], 90.00th=[  157], 95.00th=[  174],
     | 99.00th=[  237], 99.50th=[  265], 99.90th=[  326], 99.95th=[  367],
     | 99.99th=[ 1598]
   bw (  KiB/s): min=225784, max=257424, per=100.00%, avg=239785.33, stdev=532.51, samples=1912
   iops        : min=56446, max=64356, avg=59946.35, stdev=133.12, samples=1912
  write: IOPS=25.7k, BW=100MiB/s (105MB/s)(11.8GiB/120001msec); 0 zone resets
    slat (usec): min=2, max=8215, avg= 7.65, stdev=10.00
    clat (nsec): min=638, max=24452k, avg=18600.38, stdev=60415.22
     lat (usec): min=11, max=24468, avg=26.41, stdev=61.41
    clat percentiles (usec):
     |  1.00th=[   11],  5.00th=[   13], 10.00th=[   15], 20.00th=[   17],
     | 30.00th=[   18], 40.00th=[   18], 50.00th=[   18], 60.00th=[   18],
     | 70.00th=[   19], 80.00th=[   20], 90.00th=[   22], 95.00th=[   25],
     | 99.00th=[   37], 99.50th=[   48], 99.90th=[   95], 99.95th=[  124],
     | 99.99th=[  457]
   bw (  KiB/s): min=93880, max=112776, per=100.00%, avg=102798.74, stdev=411.55, samples=1912
   iops        : min=23470, max=28194, avg=25699.67, stdev=102.88, samples=1912
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.05%, 20=25.62%, 50=4.20%
  lat (usec)   : 100=34.21%, 250=35.41%, 500=0.47%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=4.46%, sys=11.58%, ctx=10291639, majf=0, minf=5604
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=7186561,3080936,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=234MiB/s (245MB/s), 234MiB/s-234MiB/s (245MB/s-245MB/s), io=27.4GiB (29.4GB), run=120001-120001msec
  WRITE: bw=100MiB/s (105MB/s), 100MiB/s-100MiB/s (105MB/s-105MB/s), io=11.8GiB (12.6GB), run=120001-120001msec

Disk stats (read/write):
  nvme0n1: ios=7178343/3077415, merge=0/0, ticks=795079/40898, in_queue=835977, util=99.99%

4M 顺序写

echo 3 > /proc/sys/vm/drop_caches
fio --name=4m_seq_write --rw=write --bs=4M --ioengine=libaio --direct=1 \
  --numjobs=1 --iodepth=8 --size=50G --time_based --runtime=60 --group_reporting \
  --directory=/mnt/nvme0n1-test

Starting 1 process
4m_seq_write: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=976MiB/s][w=244 IOPS][eta 00m:00s]
4m_seq_write: (groupid=0, jobs=1): err= 0: pid=1903815
  write: IOPS=245, BW=983MiB/s (1031MB/s)(57.6GiB/60031msec); 0 zone resets
    slat (usec): min=407, max=8882, avg=968.11, stdev=212.68
    clat (usec): min=4385, max=81306, avg=31579.35, stdev=5785.37
     lat (usec): min=4905, max=82386, avg=32547.98, stdev=5778.14
    clat percentiles (usec):
     |  1.00th=[13698],  5.00th=[24511], 10.00th=[28181], 20.00th=[29754],
     | 30.00th=[30016], 40.00th=[30540], 50.00th=[30802], 60.00th=[31065],
     | 70.00th=[31589], 80.00th=[32375], 90.00th=[39060], 95.00th=[41681],
     | 99.00th=[52167], 99.50th=[56361], 99.90th=[64750], 99.95th=[66847],
     | 99.99th=[72877]
   bw (  KiB/s): min=950272, max=1048576, per=100.00%, avg=1007720.76, stdev=20941.03, samples=119
   iops        : min=  232, max=  256, avg=246.03, stdev= 5.11, samples=119
  lat (msec)   : 10=0.03%, 20=3.25%, 50=95.16%, 100=1.56%
  cpu          : usr=9.38%, sys=14.64%, ctx=14556, majf=0, minf=2374
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,14752,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=983MiB/s (1031MB/s), 983MiB/s-983MiB/s (1031MB/s-1031MB/s), io=57.6GiB (61.9GB), run=60031-60031msec

Disk stats (read/write):
  nvme0n1: ios=0/481501, merge=0/7, ticks=0/14164438, in_queue=14164438, util=99.91%

4M 顺序读

echo 3 > /proc/sys/vm/drop_caches
fio --name=4m_seq_read --rw=read --bs=4M --ioengine=libaio --direct=1 \
  --numjobs=1 --iodepth=8 --size=50G --time_based --runtime=60 --group_reporting \
  --directory=/mnt/nvme0n1-test

Starting 1 process
4m_seq_read: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [R(1)][100.0%][r=2939MiB/s][r=734 IOPS][eta 00m:00s]
4m_seq_read: (groupid=0, jobs=1): err= 0: pid=1907021
  read: IOPS=740, BW=2962MiB/s (3106MB/s)(174GiB/60007msec)
    slat (usec): min=204, max=3437, avg=580.49, stdev=137.23
    clat (usec): min=1369, max=31000, avg=10217.58, stdev=1029.34
     lat (usec): min=1994, max=31582, avg=10798.45, stdev=1016.62
    clat percentiles (usec):
     |  1.00th=[ 7701],  5.00th=[ 8848], 10.00th=[ 9241], 20.00th=[ 9503],
     | 30.00th=[ 9765], 40.00th=[10028], 50.00th=[10159], 60.00th=[10421],
     | 70.00th=[10683], 80.00th=[10814], 90.00th=[11207], 95.00th=[11600],
     | 99.00th=[12518], 99.50th=[13042], 99.90th=[17171], 99.95th=[19530],
     | 99.99th=[29492]
   bw (  MiB/s): min= 2832, max= 3192, per=100.00%, avg=2964.73, stdev=44.43, samples=120
   iops        : min=  708, max=  798, avg=740.97, stdev=11.11, samples=120
  lat (msec)   : 2=0.01%, 4=0.14%, 10=38.39%, 20=61.42%, 50=0.05%
  cpu          : usr=0.35%, sys=44.25%, ctx=40915, majf=0, minf=11582
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=44441,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=2962MiB/s (3106MB/s), 2962MiB/s-2962MiB/s (3106MB/s-3106MB/s), io=174GiB (186GB), run=60007-60007msec

Disk stats (read/write):
  nvme0n1: ios=1505398/10, merge=9/6, ticks=6272903/1, in_queue=6272903, util=99.88%

1.3 清理

1
2

umount /mnt/nvme0n1-test
wipefs -a /dev/nvme0n1

2. ext4

2.1 创建文件系统

创建 RAID6

`1`	`mdadm --create --verbose /dev/md0 --level=6 --raid-devices=4 /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1`

格式化 RAID6

挂载 RAID6

1
2

mkdir -p /mnt/raid6-ext4
mount /dev/md0 /mnt/raid6-ext4

2.2 测试性能

4k 随机写

echo 3 > /proc/sys/vm/drop_caches
fio --name=4k_randwrite --rw=randwrite --bs=4k --ioengine=libaio --direct=1 \
  --numjobs=8 --iodepth=32 --size=20G --time_based --runtime=120 --group_reporting \
  --allow_mounted_write=1 --filename=/mnt/raid6-ext4/fio-4k-randwrite.dat

Starting 8 processes
4k_randwrite: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [w(8)][100.0%][w=158MiB/s][w=40.5k IOPS][eta 00m:00s]
4k_randwrite: (groupid=0, jobs=8): err= 0: pid=1921021
  write: IOPS=34.5k, BW=135MiB/s (141MB/s)(15.8GiB/120005msec); 0 zone resets
    slat (usec): min=3, max=54445, avg=87.25, stdev=330.51
    clat (usec): min=237, max=119183, avg=7338.96, stdev=2989.31
     lat (usec): min=302, max=123776, avg=7426.39, stdev=3053.93
    clat percentiles (usec):
     |  1.00th=[ 5407],  5.00th=[ 5932], 10.00th=[ 6128], 20.00th=[ 6390],
     | 30.00th=[ 6587], 40.00th=[ 6783], 50.00th=[ 6980], 60.00th=[ 7177],
     | 70.00th=[ 7439], 80.00th=[ 7832], 90.00th=[ 8455], 95.00th=[ 8979],
     | 99.00th=[12387], 99.50th=[20055], 99.90th=[55313], 99.95th=[70779],
     | 99.99th=[93848]
   bw (  KiB/s): min=94648, max=166944, per=100.00%, avg=137850.87, stdev=1858.05, samples=1912
   iops        : min=23662, max=41736, avg=34462.36, stdev=464.52, samples=1912
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.04%, 4=0.21%, 10=97.50%, 20=1.75%, 50=0.37%
  lat (msec)   : 100=0.12%, 250=0.01%
  cpu          : usr=1.87%, sys=19.38%, ctx=6049293, majf=0, minf=4725
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,4135584,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=135MiB/s (141MB/s), 135MiB/s-135MiB/s (141MB/s-141MB/s), io=15.8GiB (16.9GB), run=120005-120005msec

Disk stats (read/write):
    md0: ios=0/4411647, merge=0/0, ticks=0/31116932, in_queue=31116932, util=100.00%, aggrios=1053022/3151757, aggrmerge=457303/223146, aggrticks=361746/801096, aggrin_queue=1162842, aggrutil=99.97%
  nvme0n1: ios=1051796/3163738, merge=456583/241374, ticks=333187/790180, in_queue=1123367, util=99.97%
  nvme3n1: ios=1054364/3139558, merge=457404/205297, ticks=374904/795141, in_queue=1170044, util=99.96%
  nvme2n1: ios=1052708/3163485, merge=456232/241732, ticks=393093/833331, in_queue=1226424, util=99.97%
  nvme1n1: ios=1053220/3140249, merge=458995/204183, ticks=345800/785734, in_queue=1131534, util=99.96%

4k 随机读

echo 3 > /proc/sys/vm/drop_caches
fio --name=4k_randread --rw=randread --bs=4k --ioengine=libaio --direct=1 \
  --numjobs=8 --iodepth=32 --size=20G --time_based --runtime=120 --group_reporting \
  --filename=/mnt/raid6-ext4/fio-4k-randread.dat

Starting 8 processes
4k_randread: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [r(8)][100.0%][r=3781MiB/s][r=968k IOPS][eta 00m:00s]
4k_randread: (groupid=0, jobs=8): err= 0: pid=1928379
  read: IOPS=980k, BW=3827MiB/s (4013MB/s)(448GiB/120002msec)
    slat (usec): min=2, max=22326, avg= 5.36, stdev= 9.55
    clat (usec): min=5, max=28319, avg=255.26, stdev=368.93
     lat (usec): min=30, max=28328, avg=260.69, stdev=369.08
    clat percentiles (usec):
     |  1.00th=[   88],  5.00th=[   95], 10.00th=[  104], 20.00th=[  117],
     | 30.00th=[  133], 40.00th=[  149], 50.00th=[  169], 60.00th=[  192],
     | 70.00th=[  225], 80.00th=[  273], 90.00th=[  371], 95.00th=[  553],
     | 99.00th=[ 2311], 99.50th=[ 2737], 99.90th=[ 3294], 99.95th=[ 3490],
     | 99.99th=[ 7701]
   bw (  MiB/s): min= 3432, max= 4118, per=100.00%, avg=3828.99, stdev=12.81, samples=1912
   iops        : min=878655, max=1054446, avg=980221.39, stdev=3279.92, samples=1912
  lat (usec)   : 10=0.01%, 50=0.01%, 100=7.77%, 250=68.39%, 500=18.08%
  lat (usec)   : 750=1.99%, 1000=0.67%
  lat (msec)   : 2=1.64%, 4=1.44%, 10=0.02%, 20=0.01%, 50=0.01%
  cpu          : usr=16.79%, sys=69.27%, ctx=12007646, majf=0, minf=2518
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=117557831,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=3827MiB/s (4013MB/s), 3827MiB/s-3827MiB/s (4013MB/s-4013MB/s), io=448GiB (482GB), run=120002-120002msec

Disk stats (read/write):
    md0: ios=117436690/43522, merge=0/0, ticks=26505304/140684, in_queue=26645988, util=100.00%, aggrios=29450147/46750, aggrmerge=1865650/1395475, aggrticks=6687322/11001, aggrin_queue=6698324, aggrutil=100.00%
  nvme0n1: ios=29452652/53570, merge=1885324/1622929, ticks=7386033/12314, in_queue=7398348, util=100.00%
  nvme3n1: ios=29449336/39735, merge=1843379/1168037, ticks=6111870/9676, in_queue=6121545, util=100.00%
  nvme2n1: ios=29449103/53885, merge=1885349/1622856, ticks=7134533/12459, in_queue=7146992, util=100.00%
  nvme1n1: ios=29449500/39811, merge=1848551/1168078, ticks=6116853/9557, in_queue=6126411, util=100.00%

4k 随机读写

echo 3 > /proc/sys/vm/drop_caches
fio --name=4k_randrw --rw=randrw --rwmixread=70 --bs=4k --ioengine=libaio --direct=1 \
  --numjobs=8 --iodepth=1 --size=20G --time_based --runtime=120 --group_reporting \
  --allow_mounted_write=1 --filename=/mnt/raid6-ext4/fio-4k-randrw.dat

Starting 8 processes
4k_randrw: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=78.6MiB/s,w=33.2MiB/s][r=20.1k,w=8493 IOPS][eta 00m:00s]
4k_randrw: (groupid=0, jobs=8): err= 0: pid=1931989
  read: IOPS=20.0k, BW=78.1MiB/s (81.9MB/s)(9369MiB/120001msec)
    slat (usec): min=2, max=14600, avg= 9.47, stdev=20.28
    clat (nsec): min=707, max=14475k, avg=114057.25, stdev=53174.77
     lat (usec): min=16, max=14603, avg=123.66, stdev=57.02
    clat percentiles (usec):
     |  1.00th=[   87],  5.00th=[   90], 10.00th=[   91], 20.00th=[   93],
     | 30.00th=[   94], 40.00th=[   96], 50.00th=[  101], 60.00th=[  109],
     | 70.00th=[  111], 80.00th=[  124], 90.00th=[  157], 95.00th=[  178],
     | 99.00th=[  273], 99.50th=[  322], 99.90th=[  461], 99.95th=[  537],
     | 99.99th=[ 1139]
   bw (  KiB/s): min=50080, max=106709, per=100.00%, avg=79991.88, stdev=1347.29, samples=1912
   iops        : min=12520, max=26677, avg=19997.68, stdev=336.82, samples=1912
  write: IOPS=8576, BW=33.5MiB/s (35.1MB/s)(4020MiB/120001msec); 0 zone resets
    slat (usec): min=3, max=47580, avg=26.45, stdev=522.23
    clat (usec): min=2, max=42389, avg=611.31, stdev=1909.48
     lat (usec): min=39, max=51574, avg=637.88, stdev=1997.61
    clat percentiles (usec):
     |  1.00th=[  122],  5.00th=[  131], 10.00th=[  141], 20.00th=[  163],
     | 30.00th=[  206], 40.00th=[  255], 50.00th=[  297], 60.00th=[  347],
     | 70.00th=[  400], 80.00th=[  469], 90.00th=[  578], 95.00th=[  979],
     | 99.00th=[10421], 99.50th=[14877], 99.90th=[25297], 99.95th=[28443],
     | 99.99th=[32637]
   bw (  KiB/s): min=22024, max=45560, per=100.00%, avg=34322.94, stdev=565.57, samples=1912
   iops        : min= 5506, max=11390, avg=8580.33, stdev=141.38, samples=1912
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (usec)   : 100=34.73%, 250=45.84%, 500=14.47%, 750=3.19%, 1000=0.27%
  lat (msec)   : 2=0.31%, 4=0.48%, 10=0.39%, 20=0.25%, 50=0.07%
  cpu          : usr=1.75%, sys=5.21%, ctx=3542124, majf=0, minf=5452
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2398355,1029155,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=78.1MiB/s (81.9MB/s), 78.1MiB/s-78.1MiB/s (81.9MB/s-81.9MB/s), io=9369MiB (9824MB), run=120001-120001msec
  WRITE: bw=33.5MiB/s (35.1MB/s), 33.5MiB/s-33.5MiB/s (35.1MB/s-35.1MB/s), io=4020MiB (4215MB), run=120001-120001msec

Disk stats (read/write):
    md0: ios=2397573/1070445, merge=0/0, ticks=277788/754272, in_queue=1032060, util=99.05%, aggrios=1025481/860471, aggrmerge=5183988/2579731, aggrticks=187967/90796, aggrin_queue=278764, aggrutil=99.92%
  nvme0n1: ios=1024560/882495, merge=5193905/3211896, ticks=193732/91576, in_queue=285309, util=99.92%
  nvme3n1: ios=1027163/837628, merge=5172424/1947710, ticks=186838/86010, in_queue=272848, util=99.82%
  nvme2n1: ios=1025610/882143, merge=5193934/3212044, ticks=190461/103978, in_queue=294439, util=99.86%
  nvme1n1: ios=1024594/839619, merge=5175691/1947276, ticks=180838/81623, in_queue=262461, util=99.82%

4M 顺序写

echo 3 > /proc/sys/vm/drop_caches
fio --name=4m_seq_write --rw=write --bs=4M --ioengine=libaio --direct=1 \
  --numjobs=1 --iodepth=8 --size=50G --time_based --runtime=60 --group_reporting \
  --directory=/mnt/raid6-ext4

Starting 1 process
4m_seq_write: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=1057MiB/s][w=264 IOPS][eta 00m:00s]
4m_seq_write: (groupid=0, jobs=1): err= 0: pid=1934647
  write: IOPS=252, BW=1011MiB/s (1061MB/s)(59.3GiB/60009msec); 0 zone resets
    slat (usec): min=696, max=29051, avg=3947.48, stdev=2894.86
    clat (usec): min=7854, max=65807, avg=27672.29, stdev=4476.28
     lat (usec): min=9388, max=73138, avg=31620.58, stdev=5097.91
    clat percentiles (usec):
     |  1.00th=[20841],  5.00th=[22414], 10.00th=[23200], 20.00th=[24249],
     | 30.00th=[25035], 40.00th=[25822], 50.00th=[26608], 60.00th=[27657],
     | 70.00th=[28705], 80.00th=[30540], 90.00th=[33817], 95.00th=[36439],
     | 99.00th=[42730], 99.50th=[45876], 99.90th=[52691], 99.95th=[55313],
     | 99.99th=[65799]
   bw (  KiB/s): min=663552, max=1171456, per=99.96%, avg=1035287.98, stdev=98841.97, samples=119
   iops        : min=  162, max=  286, avg=252.71, stdev=24.14, samples=119
  lat (msec)   : 10=0.01%, 20=0.30%, 50=99.48%, 100=0.21%
  cpu          : usr=8.03%, sys=34.33%, ctx=47249, majf=0, minf=5932
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,15174,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=1011MiB/s (1061MB/s), 1011MiB/s-1011MiB/s (1061MB/s-1061MB/s), io=59.3GiB (63.6GB), run=60009-60009msec

Disk stats (read/write):
    md0: ios=0/485973, merge=0/0, ticks=0/4879688, in_queue=4879688, util=100.00%, aggrios=8249/255090, aggrmerge=160430/7571355, aggrticks=7477/85380, aggrin_queue=92857, aggrutil=99.98%
  nvme0n1: ios=8911/255924, merge=177042/7582089, ticks=8065/100379, in_queue=108443, util=99.98%
  nvme3n1: ios=8025/253914, merge=154163/7558632, ticks=6573/82685, in_queue=89257, util=99.50%
  nvme2n1: ios=7725/255928, merge=147521/7581260, ticks=7598/80680, in_queue=88278, util=99.53%
  nvme1n1: ios=8335/254597, merge=162995/7563440, ticks=7675/77777, in_queue=85452, util=99.56%

4M 顺序读

echo 3 > /proc/sys/vm/drop_caches
fio --name=4m_seq_read --rw=read --bs=4M --ioengine=libaio --direct=1 \
  --numjobs=1 --iodepth=8 --size=50G --time_based --runtime=60 --group_reporting \
  --directory=/mnt/raid6-ext4

Starting 1 process
4m_seq_read: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=0): [f(1)][100.0%][r=9000MiB/s][r=2250 IOPS][eta 00m:00s]
4m_seq_read: (groupid=0, jobs=1): err= 0: pid=1937785
  read: IOPS=2270, BW=9082MiB/s (9523MB/s)(532GiB/60004msec)
    slat (usec): min=238, max=2234, avg=414.98, stdev=62.30
    clat (usec): min=417, max=18489, avg=3106.55, stdev=1225.58
     lat (usec): min=718, max=18927, avg=3521.79, stdev=1234.17
    clat percentiles (usec):
     |  1.00th=[  717],  5.00th=[ 1221], 10.00th=[ 1631], 20.00th=[ 2114],
     | 30.00th=[ 2540], 40.00th=[ 2835], 50.00th=[ 2999], 60.00th=[ 3294],
     | 70.00th=[ 3589], 80.00th=[ 3916], 90.00th=[ 4424], 95.00th=[ 5014],
     | 99.00th=[ 7308], 99.50th=[ 8160], 99.90th=[10028], 99.95th=[10814],
     | 99.99th=[13566]
   bw (  MiB/s): min= 7760, max=10016, per=100.00%, avg=9084.13, stdev=370.03, samples=119
   iops        : min= 1940, max= 2504, avg=2270.90, stdev=92.50, samples=119
  lat (usec)   : 500=0.03%, 750=1.15%, 1000=1.89%
  lat (msec)   : 2=12.70%, 4=66.07%, 10=18.05%, 20=0.11%
  cpu          : usr=0.59%, sys=94.39%, ctx=9051, majf=0, minf=15159
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=136233,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=9082MiB/s (9523MB/s), 9082MiB/s-9082MiB/s (9523MB/s-9523MB/s), io=532GiB (571GB), run=60004-60004msec

Disk stats (read/write):
    md0: ios=5172987/6, merge=0/0, ticks=3219800/0, in_queue=3219800, util=100.00%, aggrios=1879189/13211, aggrmerge=1371752/405448, aggrticks=1002496/2476, aggrin_queue=1004973, aggrutil=100.00%
  nvme0n1: ios=1892021/17620, merge=1371802/540532, ticks=1065268/3056, in_queue=1068323, util=100.00%
  nvme3n1: ios=1908987/8824, merge=1371770/270360, ticks=872630/1670, in_queue=874301, util=100.00%
  nvme2n1: ios=1908990/17589, merge=1371735/540633, ticks=1083970/3423, in_queue=1087393, util=100.00%
  nvme1n1: ios=1806761/8814, merge=1371703/270270, ticks=988118/1757, in_queue=989876, util=100.00%

2.3 清理

1
2

umount /mnt/raid6-ext4
mdadm --stop /dev/md0

3. xfs

3.1 创建文件系统

创建 RAID6

`1`	`mdadm --create --verbose /dev/md0 --level=6 --raid-devices=4 /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1`

格式化 RAID6

挂载 RAID6

1
2

mkdir -p /mnt/raid6-xfs
mount /dev/md0 /mnt/raid6-xfs

3.2 测试性能

4k 随机写

echo 3 > /proc/sys/vm/drop_caches
fio --name=4k_randwrite --rw=randwrite --bs=4k --ioengine=libaio --direct=1 \
  --numjobs=8 --iodepth=32 --size=20G --time_based --runtime=120 --group_reporting \
  --allow_mounted_write=1 --filename=/mnt/raid6-xfs/fio-4k-randwrite.dat

Starting 8 processes
4k_randwrite: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [w(8)][100.0%][w=123MiB/s][w=31.4k IOPS][eta 00m:00s]
4k_randwrite: (groupid=0, jobs=8): err= 0: pid=1940340
  write: IOPS=29.6k, BW=116MiB/s (121MB/s)(13.6GiB/120007msec); 0 zone resets
    slat (usec): min=3, max=26502, avg=20.73, stdev=136.15
    clat (usec): min=152, max=54282, avg=8622.32, stdev=1428.85
     lat (usec): min=165, max=54297, avg=8643.29, stdev=1431.81
    clat percentiles (usec):
     |  1.00th=[ 7046],  5.00th=[ 7504], 10.00th=[ 7635], 20.00th=[ 7898],
     | 30.00th=[ 8029], 40.00th=[ 8160], 50.00th=[ 8356], 60.00th=[ 8586],
     | 70.00th=[ 8717], 80.00th=[ 9110], 90.00th=[ 9765], 95.00th=[10421],
     | 99.00th=[13304], 99.50th=[15270], 99.90th=[25035], 99.95th=[28443],
     | 99.99th=[41157]
   bw (  KiB/s): min=94072, max=137312, per=100.00%, avg=118473.02, stdev=775.95, samples=1912
   iops        : min=23518, max=34328, avg=29617.89, stdev=194.00, samples=1912
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.05%, 10=92.11%, 20=7.58%, 50=0.24%
  lat (msec)   : 100=0.01%
  cpu          : usr=1.93%, sys=6.29%, ctx=4986888, majf=0, minf=3586
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,3553476,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=116MiB/s (121MB/s), 116MiB/s-116MiB/s (121MB/s-121MB/s), io=13.6GiB (14.6GB), run=120007-120007msec

Disk stats (read/write):
    md0: ios=0/3613480, merge=0/0, ticks=0/30817252, in_queue=30817252, util=99.92%, aggrios=917507/2696600, aggrmerge=641848/356680, aggrticks=320494/752749, aggrin_queue=1073244, aggrutil=100.00%
  nvme0n1: ios=916162/2697378, merge=642093/357094, ticks=295017/755565, in_queue=1050583, util=100.00%
  nvme3n1: ios=916874/2697845, merge=641434/356363, ticks=330966/747980, in_queue=1078947, util=100.00%
  nvme2n1: ios=919207/2695099, merge=641483/356687, ticks=349058/753163, in_queue=1102222, util=100.00%
  nvme1n1: ios=917787/2696081, merge=642385/356576, ticks=306936/754291, in_queue=1061227, util=100.00%

4k 随机读

echo 3 > /proc/sys/vm/drop_caches
fio --name=4k_randread --rw=randread --bs=4k --ioengine=libaio --direct=1 \
  --numjobs=8 --iodepth=32 --size=20G --time_based --runtime=120 --group_reporting \
  --filename=/mnt/raid6-xfs/fio-4k-randread.dat

Starting 8 processes
4k_randread: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=0): [f(8)][100.0%][r=3774MiB/s][r=966k IOPS][eta 00m:00s]
4k_randread: (groupid=0, jobs=8): err= 0: pid=1952661
  read: IOPS=1056k, BW=4124MiB/s (4324MB/s)(483GiB/120001msec)
    slat (usec): min=2, max=24986, avg= 6.17, stdev= 9.24
    clat (usec): min=28, max=25523, avg=235.60, stdev=115.81
     lat (usec): min=31, max=25535, avg=241.85, stdev=116.33
    clat percentiles (usec):
     |  1.00th=[  108],  5.00th=[  129], 10.00th=[  143], 20.00th=[  161],
     | 30.00th=[  176], 40.00th=[  192], 50.00th=[  208], 60.00th=[  231],
     | 70.00th=[  258], 80.00th=[  297], 90.00th=[  363], 95.00th=[  433],
     | 99.00th=[  594], 99.50th=[  660], 99.90th=[  824], 99.95th=[  906],
     | 99.99th=[ 1401]
   bw (  MiB/s): min= 3618, max= 4466, per=100.00%, avg=4125.69, stdev=16.94, samples=1912
   iops        : min=926327, max=1143355, avg=1056175.34, stdev=4335.63, samples=1912
  lat (usec)   : 50=0.01%, 100=0.45%, 250=67.12%, 500=29.93%, 750=2.30%
  lat (usec)   : 1000=0.18%
  lat (msec)   : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=17.52%, sys=81.94%, ctx=571508, majf=0, minf=2912
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=126680444,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=4124MiB/s (4324MB/s), 4124MiB/s-4124MiB/s (4324MB/s-4324MB/s), io=483GiB (519GB), run=120001-120001msec

Disk stats (read/write):
    md0: ios=126511644/13, merge=0/0, ticks=21021424/0, in_queue=21021424, util=100.00%, aggrios=31779374/89, aggrmerge=3381181/20, aggrticks=5311275/4, aggrin_queue=5311279, aggrutil=100.00%
  nvme0n1: ios=31779520/91, merge=3381169/26, ticks=5308492/4, in_queue=5308495, util=100.00%
  nvme3n1: ios=31781887/91, merge=3381175/26, ticks=5317892/5, in_queue=5317896, util=100.00%
  nvme2n1: ios=31779391/88, merge=3381176/26, ticks=5316364/5, in_queue=5316369, util=100.00%
  nvme1n1: ios=31776698/87, merge=3381206/2, ticks=5302352/4, in_queue=5302356, util=100.00%

4k 随机读写

echo 3 > /proc/sys/vm/drop_caches
fio --name=4k_randrw --rw=randrw --rwmixread=70 --bs=4k --ioengine=libaio --direct=1 \
  --numjobs=8 --iodepth=1 --size=20G --time_based --runtime=120 --group_reporting \
  --allow_mounted_write=1 --filename=/mnt/raid6-xfs/fio-4k-randrw.dat

Starting 8 processes
4k_randrw: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=30.4MiB/s,w=12.9MiB/s][r=7782,w=3306 IOPS][eta 00m:00s]
4k_randrw: (groupid=0, jobs=8): err= 0: pid=1955381
  read: IOPS=20.5k, BW=80.0MiB/s (83.9MB/s)(9605MiB/120004msec)
    slat (usec): min=3, max=27865, avg= 9.68, stdev=44.57
    clat (nsec): min=915, max=21420k, avg=105529.98, stdev=53965.56
     lat (usec): min=18, max=31988, avg=115.33, stdev=71.25
    clat percentiles (usec):
     |  1.00th=[   87],  5.00th=[   90], 10.00th=[   91], 20.00th=[   92],
     | 30.00th=[   93], 40.00th=[   95], 50.00th=[   96], 60.00th=[  104],
     | 70.00th=[  110], 80.00th=[  111], 90.00th=[  120], 95.00th=[  153],
     | 99.00th=[  210], 99.50th=[  249], 99.90th=[  347], 99.95th=[  429],
     | 99.99th=[  840]
   bw (  KiB/s): min=21200, max=174135, per=100.00%, avg=82182.77, stdev=5469.20, samples=1912
   iops        : min= 5300, max=43532, avg=20545.33, stdev=1367.29, samples=1912
  write: IOPS=8793, BW=34.3MiB/s (36.0MB/s)(4122MiB/120004msec); 0 zone resets
    slat (usec): min=3, max=41563, avg=57.07, stdev=743.53
    clat (nsec): min=888, max=55891k, avg=577249.86, stdev=1959976.86
     lat (usec): min=33, max=55906, avg=634.44, stdev=2107.59
    clat percentiles (usec):
     |  1.00th=[  120],  5.00th=[  128], 10.00th=[  137], 20.00th=[  149],
     | 30.00th=[  161], 40.00th=[  176], 50.00th=[  194], 60.00th=[  219],
     | 70.00th=[  260], 80.00th=[  367], 90.00th=[  478], 95.00th=[  775],
     | 99.00th=[11469], 99.50th=[15533], 99.90th=[21627], 99.95th=[22938],
     | 99.99th=[27919]
   bw (  KiB/s): min= 8872, max=74673, per=100.00%, avg=35269.38, stdev=2348.71, samples=1912
   iops        : min= 2218, max=18667, avg=8816.96, stdev=587.17, samples=1912
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (usec)   : 100=40.08%, 250=50.04%, 500=7.29%, 750=1.04%, 1000=0.17%
  lat (msec)   : 2=0.15%, 4=0.23%, 10=0.61%, 20=0.32%, 50=0.05%
  lat (msec)   : 100=0.01%
  cpu          : usr=1.70%, sys=5.58%, ctx=3937367, majf=0, minf=2155
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2458810,1055200,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=80.0MiB/s (83.9MB/s), 80.0MiB/s-80.0MiB/s (83.9MB/s-83.9MB/s), io=9605MiB (10.1GB), run=120004-120004msec
  WRITE: bw=34.3MiB/s (36.0MB/s), 34.3MiB/s-34.3MiB/s (36.0MB/s-36.0MB/s), io=4122MiB (4322MB), run=120004-120004msec

Disk stats (read/write):
    md0: ios=2458285/1055004, merge=0/0, ticks=254740/643324, in_queue=898064, util=100.00%, aggrios=1078700/791604, aggrmerge=6173324/44, aggrticks=212008/68723, aggrin_queue=280731, aggrutil=98.10%
  nvme0n1: ios=1079069/791504, merge=6173339/30, ticks=207819/64418, in_queue=272237, util=98.06%
  nvme3n1: ios=1077577/792216, merge=6173310/59, ticks=215384/69736, in_queue=285119, util=98.00%
  nvme2n1: ios=1077832/792186, merge=6173310/59, ticks=214590/73950, in_queue=288540, util=98.10%
  nvme1n1: ios=1080322/790511, merge=6173338/31, ticks=210242/66788, in_queue=277029, util=98.07%

4M 顺序写

echo 3 > /proc/sys/vm/drop_caches
fio --name=4m_seq_write --rw=write --bs=4M --ioengine=libaio --direct=1 \
  --numjobs=1 --iodepth=8 --size=50G --time_based --runtime=60 --group_reporting \
  --directory=/mnt/raid6-xfs

Starting 1 process
4m_seq_write: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=964MiB/s][w=241 IOPS][eta 00m:00s]
4m_seq_write: (groupid=0, jobs=1): err= 0: pid=1957297
  write: IOPS=221, BW=886MiB/s (929MB/s)(51.9GiB/60014msec); 0 zone resets
    slat (usec): min=777, max=28964, avg=4508.44, stdev=3157.21
    clat (usec): min=4967, max=81188, avg=31602.09, stdev=4772.63
     lat (usec): min=15779, max=83917, avg=36111.25, stdev=5350.46
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[25297], 10.00th=[26346], 20.00th=[27919],
     | 30.00th=[28967], 40.00th=[29754], 50.00th=[30802], 60.00th=[32113],
     | 70.00th=[33424], 80.00th=[34866], 90.00th=[38011], 95.00th=[40633],
     | 99.00th=[45351], 99.50th=[47973], 99.90th=[56361], 99.95th=[61604],
     | 99.99th=[67634]
   bw (  KiB/s): min=761856, max=1048576, per=99.89%, avg=906108.73, stdev=73554.26, samples=119
   iops        : min=  186, max=  256, avg=221.14, stdev=17.96, samples=119
  lat (msec)   : 10=0.01%, 20=0.05%, 50=99.61%, 100=0.34%
  cpu          : usr=8.66%, sys=35.80%, ctx=42541, majf=0, minf=71654
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.9%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,13291,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=886MiB/s (929MB/s), 886MiB/s-886MiB/s (929MB/s-929MB/s), io=51.9GiB (55.7GB), run=60014-60014msec

Disk stats (read/write):
    md0: ios=0/424172, merge=0/0, ticks=0/4428772, in_queue=4428772, util=99.97%, aggrios=8809/220705, aggrmerge=183902/6590883, aggrticks=9283/67789, aggrin_queue=77072, aggrutil=99.97%
  nvme0n1: ios=8438/221027, merge=175305/6592858, ticks=9150/85042, in_queue=94190, util=99.97%
  nvme3n1: ios=8844/220420, merge=183811/6589689, ticks=9286/63809, in_queue=73094, util=99.23%
  nvme2n1: ios=8946/220571, merge=187729/6588710, ticks=9383/61551, in_queue=70935, util=99.29%
  nvme1n1: ios=9011/220804, merge=188765/6592276, ticks=9313/60755, in_queue=70069, util=99.34%

4M 顺序读

echo 3 > /proc/sys/vm/drop_caches
fio --name=4m_seq_read --rw=read --bs=4M --ioengine=libaio --direct=1 \
  --numjobs=1 --iodepth=8 --size=50G --time_based --runtime=60 --group_reporting \
  --directory=/mnt/raid6-xfs

Starting 1 process
4m_seq_read: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [R(1)][100.0%][r=9590MiB/s][r=2397 IOPS][eta 00m:00s]
4m_seq_read: (groupid=0, jobs=1): err= 0: pid=1959857
  read: IOPS=2384, BW=9539MiB/s (10.0GB/s)(559GiB/60001msec)
    slat (usec): min=227, max=3364, avg=410.08, stdev=56.82
    clat (usec): min=397, max=15775, avg=2942.87, stdev=690.51
     lat (usec): min=741, max=19141, avg=3353.22, stdev=703.77
    clat percentiles (usec):
     |  1.00th=[  881],  5.00th=[ 2409], 10.00th=[ 2704], 20.00th=[ 2737],
     | 30.00th=[ 2769], 40.00th=[ 2802], 50.00th=[ 2835], 60.00th=[ 2868],
     | 70.00th=[ 2900], 80.00th=[ 2966], 90.00th=[ 3359], 95.00th=[ 4080],
     | 99.00th=[ 5932], 99.50th=[ 6783], 99.90th=[ 8586], 99.95th=[ 9110],
     | 99.99th=[10683]
   bw (  MiB/s): min= 8079, max=10256, per=100.00%, avg=9542.85, stdev=380.62, samples=119
   iops        : min= 2019, max= 2564, avg=2385.66, stdev=95.20, samples=119
  lat (usec)   : 500=0.02%, 750=0.57%, 1000=0.61%
  lat (msec)   : 2=1.83%, 4=91.57%, 10=5.39%, 20=0.02%
  cpu          : usr=0.63%, sys=97.74%, ctx=2739, majf=0, minf=8216
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=143082,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=9539MiB/s (10.0GB/s), 9539MiB/s-9539MiB/s (10.0GB/s-10.0GB/s), io=559GiB (600GB), run=60001-60001msec

Disk stats (read/write):
    md0: ios=5447134/5, merge=0/0, ticks=2711068/0, in_queue=2711068, util=100.00%, aggrios=1961115/42, aggrmerge=1873436/9, aggrticks=867493/1, aggrin_queue=867495, aggrutil=100.00%
  nvme0n1: ios=1977580/42, merge=1876309/5, ticks=852352/2, in_queue=852354, util=100.00%
  nvme3n1: ios=1953626/43, merge=1876454/12, ticks=832879/2, in_queue=832881, util=100.00%
  nvme2n1: ios=1977399/42, merge=1864556/10, ticks=912517/1, in_queue=912519, util=100.00%
  nvme1n1: ios=1935855/42, merge=1876425/9, ticks=872226/2, in_queue=872228, util=100.00%

3.3 清理

1
2

umount /mnt/raid6-xfs
mdadm --stop /dev/md0

4. zfs

4.1 创建文件系统

创建 raidz2 池

`1`	`zpool create -o ashift=12 -f tank raidz2 /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1`

挂载存储

`1`	`zfs create -o mountpoint=/mnt/raid6-zfs tank/data`

4.2 测试性能

4k 随机写

echo 3 > /proc/sys/vm/drop_caches
fio --name=4k_randwrite --rw=randwrite --bs=4k --ioengine=libaio --direct=1 \
  --numjobs=8 --iodepth=32 --size=20G --time_based --runtime=120 --group_reporting \
  --allow_mounted_write=1 --filename=/mnt/raid6-zfs/fio-4k-randwrite.dat

Starting 8 processes
4k_randwrite: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [w(8)][100.0%][w=90.9MiB/s][w=23.3k IOPS][eta 00m:00s]
4k_randwrite: (groupid=0, jobs=8): err= 0: pid=1963969
  write: IOPS=23.3k, BW=91.1MiB/s (95.5MB/s)(10.7GiB/120001msec); 0 zone resets
    slat (usec): min=6, max=78665, avg=340.44, stdev=302.79
    clat (usec): min=2, max=115643, avg=10640.06, stdev=3639.21
     lat (usec): min=170, max=115673, avg=10980.75, stdev=3741.09
    clat percentiles (usec):
     |  1.00th=[ 3163],  5.00th=[ 5669], 10.00th=[ 6456], 20.00th=[ 7570],
     | 30.00th=[ 8455], 40.00th=[ 9634], 50.00th=[10683], 60.00th=[11600],
     | 70.00th=[12387], 80.00th=[13042], 90.00th=[14222], 95.00th=[16188],
     | 99.00th=[21365], 99.50th=[23200], 99.90th=[31589], 99.95th=[38011],
     | 99.99th=[63177]
   bw (  KiB/s): min=65471, max=166791, per=100.00%, avg=93304.09, stdev=1815.69, samples=1912
   iops        : min=16367, max=41697, avg=23325.98, stdev=453.93, samples=1912
  lat (usec)   : 4=0.01%, 10=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
  lat (usec)   : 1000=0.05%
  lat (msec)   : 2=0.56%, 4=0.57%, 10=42.33%, 20=54.68%, 50=1.80%
  lat (msec)   : 100=0.01%, 250=0.01%
  cpu          : usr=0.87%, sys=19.72%, ctx=1994732, majf=0, minf=11071
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2797218,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=91.1MiB/s (95.5MB/s), 91.1MiB/s-91.1MiB/s (95.5MB/s-95.5MB/s), io=10.7GiB (11.5GB), run=120001-120001msec

4k 随机读

echo 3 > /proc/sys/vm/drop_caches
fio --name=4k_randread --rw=randread --bs=4k --ioengine=libaio --direct=1 \
  --numjobs=8 --iodepth=32 --size=20G --time_based --runtime=120 --group_reporting \
  --filename=/mnt/raid6-zfs/fio-4k-randread.dat

Starting 8 processes
4k_randread: Laying out IO file (1 file / 20480MiB)
Jobs: 7 (f=0): [f(2),_(1),f(5)][100.0%][r=496MiB/s][r=127k IOPS][eta 00m:00s]
4k_randread: (groupid=0, jobs=8): err= 0: pid=167510
  read: IOPS=135k, BW=527MiB/s (553MB/s)(61.8GiB/120001msec)
    slat (usec): min=3, max=43573, avg=58.05, stdev=79.10
    clat (nsec): min=1544, max=48675k, avg=1838908.19, stdev=499838.11
     lat (usec): min=47, max=48751, avg=1897.03, stdev=509.35
    clat percentiles (usec):
     |  1.00th=[ 1467],  5.00th=[ 1582], 10.00th=[ 1631], 20.00th=[ 1696],
     | 30.00th=[ 1745], 40.00th=[ 1778], 50.00th=[ 1827], 60.00th=[ 1860],
     | 70.00th=[ 1893], 80.00th=[ 1926], 90.00th=[ 2008], 95.00th=[ 2073],
     | 99.00th=[ 2376], 99.50th=[ 2704], 99.90th=[ 7373], 99.95th=[12256],
     | 99.99th=[23987]
   bw (  KiB/s): min=461232, max=652588, per=100.00%, avg=539742.12, stdev=3565.08, samples=1912
   iops        : min=115308, max=163146, avg=134935.13, stdev=891.27, samples=1912
  lat (usec)   : 2=0.01%, 4=0.01%, 50=0.01%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=90.09%, 4=9.73%, 10=0.11%, 20=0.05%, 50=0.02%
  cpu          : usr=2.04%, sys=96.95%, ctx=21818, majf=0, minf=5155
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=16187984,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=527MiB/s (553MB/s), 527MiB/s-527MiB/s (553MB/s-553MB/s), io=61.8GiB (66.3GB), run=120001-120001msec

4k 随机读写

echo 3 > /proc/sys/vm/drop_caches
fio --name=4k_randrw --rw=randrw --rwmixread=70 --bs=4k --ioengine=libaio --direct=1 \
  --numjobs=8 --iodepth=1 --size=20G --time_based --runtime=120 --group_reporting \
  --allow_mounted_write=1 --filename=/mnt/raid6-zfs/fio-4k-randrw.dat

Starting 8 processes
4k_randrw: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=0): [f(8)][100.0%][r=204MiB/s,w=86.9MiB/s][r=52.2k,w=22.2k IOPS][eta 00m:00s]
4k_randrw: (groupid=0, jobs=8): err= 0: pid=327904
  read: IOPS=54.6k, BW=213MiB/s (224MB/s)(25.0GiB/120001msec)
    slat (usec): min=2, max=57794, avg=55.87, stdev=80.66
    clat (nsec): min=391, max=2255.5k, avg=637.85, stdev=1870.43
     lat (usec): min=3, max=57800, avg=56.61, stdev=80.77
    clat percentiles (nsec):
     |  1.00th=[  422],  5.00th=[  446], 10.00th=[  462], 20.00th=[  482],
     | 30.00th=[  502], 40.00th=[  540], 50.00th=[  572], 60.00th=[  604],
     | 70.00th=[  652], 80.00th=[  716], 90.00th=[  852], 95.00th=[ 1032],
     | 99.00th=[ 1480], 99.50th=[ 1688], 99.90th=[ 2832], 99.95th=[ 5600],
     | 99.99th=[18816]
   bw (  KiB/s): min=186400, max=299796, per=100.00%, avg=218550.90, stdev=1930.64, samples=1912
   iops        : min=46600, max=74949, avg=54637.47, stdev=482.66, samples=1912
  write: IOPS=23.4k, BW=91.5MiB/s (95.9MB/s)(10.7GiB/120001msec); 0 zone resets
    slat (usec): min=7, max=59822, avg=205.29, stdev=193.74
    clat (nsec): min=410, max=1320.0k, avg=854.01, stdev=2124.52
     lat (usec): min=8, max=59828, avg=206.37, stdev=194.13
    clat percentiles (nsec):
     |  1.00th=[  498],  5.00th=[  532], 10.00th=[  564], 20.00th=[  612],
     | 30.00th=[  644], 40.00th=[  684], 50.00th=[  724], 60.00th=[  780],
     | 70.00th=[  852], 80.00th=[  980], 90.00th=[ 1240], 95.00th=[ 1544],
     | 99.00th=[ 2448], 99.50th=[ 2864], 99.90th=[ 4704], 99.95th=[ 8640],
     | 99.99th=[22400]
   bw (  KiB/s): min=79988, max=129866, per=100.00%, avg=93686.46, stdev=813.81, samples=1912
   iops        : min=19997, max=32466, avg=23421.44, stdev=203.45, samples=1912
  lat (nsec)   : 500=20.53%, 750=54.78%, 1000=15.12%
  lat (usec)   : 2=8.80%, 4=0.68%, 10=0.06%, 20=0.02%, 50=0.01%
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=1.76%, sys=57.53%, ctx=1716012, majf=0, minf=4905
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=6554099,2809430,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=213MiB/s (224MB/s), 213MiB/s-213MiB/s (224MB/s-224MB/s), io=25.0GiB (26.8GB), run=120001-120001msec
  WRITE: bw=91.5MiB/s (95.9MB/s), 91.5MiB/s-91.5MiB/s (95.9MB/s-95.9MB/s), io=10.7GiB (11.5GB), run=120001-120001msec

4M 顺序写

echo 3 > /proc/sys/vm/drop_caches
fio --name=4m_seq_write --rw=write --bs=4M --ioengine=libaio --direct=1 \
  --numjobs=1 --iodepth=8 --size=50G --time_based --runtime=60 --group_reporting \
  --directory=/mnt/raid6-zfs

Starting 1 process
4m_seq_write: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=2000MiB/s][w=500 IOPS][eta 00m:00s]
4m_seq_write: (groupid=0, jobs=1): err= 0: pid=2579345
  write: IOPS=575, BW=2303MiB/s (2415MB/s)(135GiB/60003msec); 0 zone resets
    slat (usec): min=1293, max=13604, avg=1732.63, stdev=256.27
    clat (usec): min=6, max=44204, avg=12155.63, stdev=1445.06
     lat (usec): min=2129, max=49405, avg=13888.73, stdev=1623.53
    clat percentiles (usec):
     |  1.00th=[10552],  5.00th=[10814], 10.00th=[11076], 20.00th=[11338],
     | 30.00th=[11469], 40.00th=[11600], 50.00th=[11731], 60.00th=[11863],
     | 70.00th=[12125], 80.00th=[12649], 90.00th=[13960], 95.00th=[15008],
     | 99.00th=[17171], 99.50th=[18220], 99.90th=[23725], 99.95th=[25297],
     | 99.99th=[40109]
   bw (  MiB/s): min= 1704, max= 2448, per=100.00%, avg=2305.14, stdev=140.38, samples=119
   iops        : min=  426, max=  612, avg=576.29, stdev=35.09, samples=119
  lat (usec)   : 10=0.01%
  lat (msec)   : 4=0.01%, 10=0.01%, 20=99.75%, 50=0.23%
  cpu          : usr=14.70%, sys=84.39%, ctx=2041, majf=0, minf=24658
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,34554,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=2303MiB/s (2415MB/s), 2303MiB/s-2303MiB/s (2415MB/s-2415MB/s), io=135GiB (145GB), run=60003-60003msec

4M 顺序读

echo 3 > /proc/sys/vm/drop_caches
fio --name=4m_seq_read --rw=read --bs=4M --ioengine=libaio --direct=1 \
  --numjobs=1 --iodepth=8 --size=50G --time_based --runtime=60 --group_reporting \
  --directory=/mnt/raid6-zfs

Starting 1 process
4m_seq_read: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [R(1)][100.0%][r=1846MiB/s][r=461 IOPS][eta 00m:00s]
4m_seq_read: (groupid=0, jobs=1): err= 0: pid=3990788
  read: IOPS=472, BW=1891MiB/s (1983MB/s)(111GiB/60002msec)
    slat (usec): min=1771, max=3993, avg=2111.42, stdev=116.84
    clat (usec): min=3, max=27204, avg=14804.46, stdev=675.49
     lat (usec): min=2042, max=31201, avg=16916.39, stdev=759.75
    clat percentiles (usec):
     |  1.00th=[13698],  5.00th=[13960], 10.00th=[14091], 20.00th=[14222],
     | 30.00th=[14353], 40.00th=[14615], 50.00th=[14746], 60.00th=[14877],
     | 70.00th=[15139], 80.00th=[15270], 90.00th=[15533], 95.00th=[15795],
     | 99.00th=[17171], 99.50th=[17433], 99.90th=[18220], 99.95th=[19006],
     | 99.99th=[23725]
   bw (  MiB/s): min= 1784, max= 1984, per=100.00%, avg=1891.43, stdev=44.34, samples=119
   iops        : min=  446, max=  496, avg=472.86, stdev=11.09, samples=119
  lat (usec)   : 4=0.01%
  lat (msec)   : 4=0.01%, 10=0.01%, 20=99.94%, 50=0.04%
  cpu          : usr=0.16%, sys=99.82%, ctx=170, majf=0, minf=16427
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=28366,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=1891MiB/s (1983MB/s), 1891MiB/s-1891MiB/s (1983MB/s-1983MB/s), io=111GiB (119GB), run=60002-60002msec

4.3 清理

5. btrfs

5.1 创建文件系统

创建 RAID6

`1`	`mdadm --create --verbose /dev/md0 --level=6 --raid-devices=4 /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1`

格式化 RAID6

挂载 RAID6

1
2

mkdir -p /mnt/raid6-btrfs
mount /dev/md0 /mnt/raid6-btrfs

5.2 测试性能

4k 随机写

echo 3 > /proc/sys/vm/drop_caches
fio --name=4k_randwrite --rw=randwrite --bs=4k --ioengine=libaio --direct=1 \
  --numjobs=8 --iodepth=32 --size=20G --time_based --runtime=120 --group_reporting \
  --allow_mounted_write=1 --filename=/mnt/raid6-btrfs/fio-4k-randwrite.dat

Starting 8 processes
4k_randwrite: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [w(8)][100.0%][w=194MiB/s][w=49.7k IOPS][eta 00m:00s]
4k_randwrite: (groupid=0, jobs=8): err= 0: pid=3994479
  write: IOPS=42.2k, BW=165MiB/s (173MB/s)(19.3GiB/120005msec); 0 zone resets
    slat (usec): min=10, max=32078, avg=96.53, stdev=261.42
    clat (usec): min=398, max=46208, avg=5962.46, stdev=1939.07
     lat (usec): min=423, max=46279, avg=6059.17, stdev=1970.81
    clat percentiles (usec):
     |  1.00th=[ 3621],  5.00th=[ 4080], 10.00th=[ 4490], 20.00th=[ 5211],
     | 30.00th=[ 5473], 40.00th=[ 5669], 50.00th=[ 5800], 60.00th=[ 5932],
     | 70.00th=[ 6128], 80.00th=[ 6390], 90.00th=[ 6980], 95.00th=[ 8029],
     | 99.00th=[11076], 99.50th=[15795], 99.90th=[34341], 99.95th=[37487],
     | 99.99th=[41681]
   bw (  KiB/s): min=108776, max=206328, per=99.97%, avg=168894.49, stdev=2337.59, samples=1912
   iops        : min=27194, max=51582, avg=42223.59, stdev=584.40, samples=1912
  lat (usec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.03%, 4=4.00%, 10=94.46%, 20=1.14%, 50=0.36%
  cpu          : usr=2.00%, sys=42.74%, ctx=3123061, majf=0, minf=29643
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,5068581,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=165MiB/s (173MB/s), 165MiB/s-165MiB/s (173MB/s-173MB/s), io=19.3GiB (20.8GB), run=120005-120005msec

4k 随机读

echo 3 > /proc/sys/vm/drop_caches
fio --name=4k_randread --rw=randread --bs=4k --ioengine=libaio --direct=1 \
  --numjobs=8 --iodepth=32 --size=20G --time_based --runtime=120 --group_reporting \
  --filename=/mnt/raid6-btrfs/fio-4k-randread.dat

Starting 8 processes
4k_randread: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=0): [f(8)][100.0%][r=1365MiB/s][r=349k IOPS][eta 00m:00s]
4k_randread: (groupid=0, jobs=8): err= 0: pid=3998132
  read: IOPS=407k, BW=1588MiB/s (1665MB/s)(186GiB/120001msec)
    slat (usec): min=5, max=29813, avg=18.14, stdev=18.26
    clat (usec): min=84, max=31354, avg=610.73, stdev=185.39
     lat (usec): min=98, max=31366, avg=628.97, stdev=187.08
    clat percentiles (usec):
     |  1.00th=[  437],  5.00th=[  482], 10.00th=[  502], 20.00th=[  529],
     | 30.00th=[  562], 40.00th=[  586], 50.00th=[  603], 60.00th=[  627],
     | 70.00th=[  644], 80.00th=[  668], 90.00th=[  709], 95.00th=[  758],
     | 99.00th=[  889], 99.50th=[  988], 99.90th=[ 1401], 99.95th=[ 1991],
     | 99.99th=[ 9110]
   bw (  MiB/s): min= 1327, max= 2076, per=100.00%, avg=1589.36, stdev=18.22, samples=1912
   iops        : min=339942, max=531600, avg=406875.78, stdev=4663.81, samples=1912
  lat (usec)   : 100=0.01%, 250=0.01%, 500=9.69%, 750=84.78%, 1000=5.06%
  lat (msec)   : 2=0.41%, 4=0.03%, 10=0.02%, 20=0.01%, 50=0.01%
  cpu          : usr=6.83%, sys=92.61%, ctx=56035, majf=0, minf=4323
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=48790319,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=1588MiB/s (1665MB/s), 1588MiB/s-1588MiB/s (1665MB/s-1665MB/s), io=186GiB (200GB), run=120001-120001msec

4k 随机读写

echo 3 > /proc/sys/vm/drop_caches
fio --name=4k_randrw --rw=randrw --rwmixread=70 --bs=4k --ioengine=libaio --direct=1 \
  --numjobs=8 --iodepth=1 --size=20G --time_based --runtime=120 --group_reporting \
  --allow_mounted_write=1 --filename=/mnt/raid6-btrfs/fio-4k-randrw.dat

Starting 8 processes
4k_randrw: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=31.9MiB/s,w=13.8MiB/s][r=8174,w=3527 IOPS][eta 00m:00s]
4k_randrw: (groupid=0, jobs=8): err= 0: pid=4001506
  read: IOPS=16.6k, BW=64.9MiB/s (68.0MB/s)(7786MiB/120002msec)
    slat (usec): min=7, max=8169, avg=23.86, stdev=10.77
    clat (nsec): min=838, max=27750k, avg=121119.69, stdev=86765.10
     lat (usec): min=37, max=27786, avg=145.17, stdev=87.95
    clat percentiles (usec):
     |  1.00th=[   95],  5.00th=[   99], 10.00th=[  101], 20.00th=[  103],
     | 30.00th=[  104], 40.00th=[  106], 50.00th=[  111], 60.00th=[  118],
     | 70.00th=[  121], 80.00th=[  125], 90.00th=[  157], 95.00th=[  178],
     | 99.00th=[  255], 99.50th=[  302], 99.90th=[  457], 99.95th=[  594],
     | 99.99th=[ 3326]
   bw (  KiB/s): min=15120, max=157704, per=100.00%, avg=66654.69, stdev=3811.96, samples=1912
   iops        : min= 3780, max=39426, avg=16663.67, stdev=952.99, samples=1912
  write: IOPS=7128, BW=27.8MiB/s (29.2MB/s)(3341MiB/120002msec); 0 zone resets
    slat (usec): min=11, max=46642, avg=89.32, stdev=907.10
    clat (nsec): min=860, max=47070k, avg=685854.72, stdev=2479658.53
     lat (usec): min=40, max=48153, avg=775.36, stdev=2660.01
    clat percentiles (usec):
     |  1.00th=[   33],  5.00th=[   39], 10.00th=[   49], 20.00th=[   72],
     | 30.00th=[  109], 40.00th=[  145], 50.00th=[  176], 60.00th=[  212],
     | 70.00th=[  277], 80.00th=[  375], 90.00th=[  537], 95.00th=[ 2343],
     | 99.00th=[13960], 99.50th=[19268], 99.90th=[26608], 99.95th=[28967],
     | 99.99th=[35914]
   bw (  KiB/s): min= 6720, max=68872, per=100.00%, avg=28606.03, stdev=1627.74, samples=1912
   iops        : min= 1680, max=17218, avg=7151.51, stdev=406.93, samples=1912
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=3.24%
  lat (usec)   : 100=11.10%, 250=74.91%, 500=7.19%, 750=1.34%, 1000=0.21%
  lat (msec)   : 2=0.42%, 4=0.36%, 10=0.69%, 20=0.39%, 50=0.13%
  cpu          : usr=1.64%, sys=10.42%, ctx=3173151, majf=0, minf=8419
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1993297,855419,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=64.9MiB/s (68.0MB/s), 64.9MiB/s-64.9MiB/s (68.0MB/s-68.0MB/s), io=7786MiB (8165MB), run=120002-120002msec
  WRITE: bw=27.8MiB/s (29.2MB/s), 27.8MiB/s-27.8MiB/s (29.2MB/s-29.2MB/s), io=3341MiB (3504MB), run=120002-120002msec

4M 顺序写

echo 3 > /proc/sys/vm/drop_caches
fio --name=4m_seq_write --rw=write --bs=4M --ioengine=libaio --direct=1 \
  --numjobs=1 --iodepth=8 --size=50G --time_based --runtime=60 --group_reporting \
  --directory=/mnt/raid6-btrfs

Starting 1 process
4m_seq_write: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=1041MiB/s][w=260 IOPS][eta 00m:00s]
4m_seq_write: (groupid=0, jobs=1): err= 0: pid=4004221
  write: IOPS=245, BW=982MiB/s (1030MB/s)(57.6GiB/60010msec); 0 zone resets
    slat (usec): min=1387, max=51095, avg=4064.23, stdev=2197.78
    clat (usec): min=7280, max=75665, avg=28498.74, stdev=4611.73
     lat (usec): min=9874, max=87549, avg=32563.74, stdev=5080.03
    clat percentiles (usec):
     |  1.00th=[22414],  5.00th=[23725], 10.00th=[24249], 20.00th=[25297],
     | 30.00th=[26084], 40.00th=[26608], 50.00th=[27395], 60.00th=[28181],
     | 70.00th=[29230], 80.00th=[31065], 90.00th=[34341], 95.00th=[36439],
     | 99.00th=[44827], 99.50th=[50070], 99.90th=[66847], 99.95th=[70779],
     | 99.99th=[73925]
   bw (  KiB/s): min=704512, max=1114112, per=99.94%, avg=1005275.43, stdev=88912.09, samples=119
   iops        : min=  172, max=  272, avg=245.43, stdev=21.71, samples=119
  lat (msec)   : 10=0.01%, 20=0.04%, 50=99.47%, 100=0.47%
  cpu          : usr=7.55%, sys=56.10%, ctx=28659, majf=0, minf=7210
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,14737,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=982MiB/s (1030MB/s), 982MiB/s-982MiB/s (1030MB/s-1030MB/s), io=57.6GiB (61.8GB), run=60010-60010msec

4M 顺序读

echo 3 > /proc/sys/vm/drop_caches
fio --name=4m_seq_read --rw=read --bs=4M --ioengine=libaio --direct=1 \
  --numjobs=1 --iodepth=8 --size=50G --time_based --runtime=60 --group_reporting \
  --directory=/mnt/raid6-btrfs

Starting 1 process
4m_seq_read: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [R(1)][100.0%][r=837MiB/s][r=209 IOPS][eta 00m:00s]
4m_seq_read: (groupid=0, jobs=1): err= 0: pid=4009283
  read: IOPS=216, BW=866MiB/s (908MB/s)(50.8GiB/60004msec)
    slat (usec): min=3892, max=15785, avg=4611.47, stdev=405.68
    clat (usec): min=397, max=54320, avg=32314.95, stdev=2261.59
     lat (usec): min=4833, max=63086, avg=36927.46, stdev=2531.49
    clat percentiles (usec):
     |  1.00th=[28705],  5.00th=[29492], 10.00th=[29754], 20.00th=[30540],
     | 30.00th=[31065], 40.00th=[31589], 50.00th=[32113], 60.00th=[32637],
     | 70.00th=[33162], 80.00th=[33817], 90.00th=[34341], 95.00th=[35390],
     | 99.00th=[40109], 99.50th=[42730], 99.90th=[46924], 99.95th=[50070],
     | 99.99th=[51643]
   bw (  KiB/s): min=712704, max=983040, per=99.99%, avg=886786.23, stdev=44116.29, samples=119
   iops        : min=  174, max=  240, avg=216.50, stdev=10.77, samples=119
  lat (usec)   : 500=0.01%
  lat (msec)   : 10=0.02%, 20=0.02%, 50=99.91%, 100=0.05%
  cpu          : usr=0.17%, sys=99.77%, ctx=260, majf=0, minf=8214
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.9%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=12992,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=866MiB/s (908MB/s), 866MiB/s-866MiB/s (908MB/s-908MB/s), io=50.8GiB (54.5GB), run=60004-60004msec

5.3 清理

`1`	`umount /mnt/raid6-btrfs`

6. 对比

测试结果（Latency 为 fio 报告的 clat avg，CPU 为 usr+sys）。在四盘位 RAID6 软阵列下，各文件系统性能对比。

6.1 4k - 随机写

文件系统	进程数	IOPS	BW(MiB/s)	Latency(us)	CPU(%)
ext4	8	34.5k	135	7339	21.3
xfs	8	29.6k	116	8622	8.2
zfs	8	23.3k	91.1	10640	20.6
btrfs	8	42.2k	165	5962	44.7
单盘	8	235k	919	1060	45.2
理论 RAID6	—	58.8k	230	—	—

4k 随机写时，btrfs 占用更多 CPU，性能最接近理论值

6.2 4k - 随机读

文件系统	进程数	IOPS	BW(MiB/s)	Latency(us)	CPU(%)
ext4	8	980k	3827	255	86.1
xfs	8	1056k	4124	236	99.5
zfs	8	135k	527	1839	99.0
btrfs	8	407k	1588	611	99.4
单盘	8	413k	1614	611	60.5
理论 RAID6	—	1652k	6456	—	—

4k 随机读时，ext4、xfs 性能最接近理论值，CPU 是瓶颈

6.3 4k - 随机读写

70% 随机读，30% 随机写。

文件系统	进程数	IOPS读	IOPS写	BW读(MiB/s)	BW写(MiB/s)	Latency读(us)	Latency写(us)	CPU(%)
ext4	8	20.0k	8.6k	78.1	33.5	114	611	7.0
xfs	8	20.5k	8.8k	80.0	34.3	106	577	7.3
zfs	8	54.6k	23.4k	213	91.5	0.64	0.85	59.3
btrfs	8	16.6k	7.1k	64.9	27.8	121	686	12.1
单盘	8	59.9k	25.7k	234	100	112	18.6	16.0
理论 RAID6	—	240k	6.4k	936	25	—	—	—

4k 随机读写时，zfs 明显占用更多 CPU，性能明显优于其他文件系统

6.4 4M - 顺序写

文件系统	进程数	IOPS	BW(MiB/s)	Latency(ms)	CPU(%)
ext4	1	252	1011	27.7	42.4
xfs	1	221	886	31.6	44.5
zfs	1	575	2303	12.2	99.1
btrfs	1	245	982	28.5	63.7
单盘	1	245	983	31.6	24.0
理论 RAID6	—	490	1966	—	—

4M 顺序写时，zfs 占用更多 CPU，性能最接近理论值

6.5 4M - 顺序读

文件系统	进程数	IOPS	BW(MiB/s)	Latency(us/ms)	CPU(%)
ext4	1	2270	9082	3107us	95.0
xfs	1	2384	9539	2943us	98.4
zfs	1	472	1891	14.8ms	100
btrfs	1	216	866	32.3ms	99.9
单盘	1	740	2962	10.2ms	44.6
理论 RAID6	—	2960	11848	—	—

4M 顺序读时，ext4、xfs 性能最接近理论值，CPU 是瓶颈

7. 结论

RAID6 需要计算大量校验数据，占用更多 CPU。

4k 随机写时，btrfs 占用更多 CPU，性能最接近理论值
4k 随机读时，ext4、xfs 性能最接近理论值，CPU 是瓶颈
4k 随机读写时，zfs 明显占用更多 CPU，性能明显优于其他文件系统
4M 顺序写时，zfs 占用更多 CPU，性能最接近理论值
4M 顺序读时，ext4、xfs 性能最接近理论值，CPU 是瓶颈

选型建议：大文件顺序读、高吞吐读用 xfs 或 ext4；大文件顺序写优先 zfs；小文件随机 IO 用 ext4、btrfs；备份归档用 xfs、zfs。

推荐订阅源

陈少文的网站

1. 单盘

1.1 创建文件系统

1.2 测试性能

1.3 清理

2. ext4

2.1 创建文件系统

2.2 测试性能

2.3 清理

3. xfs

3.1 创建文件系统

3.2 测试性能

3.3 清理

4. zfs

4.1 创建文件系统

4.2 测试性能

4.3 清理

5. btrfs

5.1 创建文件系统

5.2 测试性能

5.3 清理

6. 对比

6.1 4k - 随机写

6.2 4k - 随机读

6.3 4k - 随机读写

6.4 4M - 顺序写

6.5 4M - 顺序读

7. 结论