xfs 在各种 RAID 下的性能测试

1. 磁盘基准测试

1.1 4k 随机写

fio \
  --name=4k_randwrite \
  --rw=randwrite \
  --bs=4k \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=8 \
  --iodepth=32 \
  --size=20G \
  --time_based \
  --runtime=120 \
  --group_reporting \
  --allow_mounted_write=1 \
  --filename=/data1/fio-4k-randwrite.dat

Starting 8 processes
4k_randwrite: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [w(8)][100.0%][r=0KiB/s,w=252MiB/s][r=0,w=64.5k IOPS][eta 00m:00s]
4k_randwrite: (groupid=0, jobs=8): err= 0: pid=31269
  write: IOPS=65.1k, BW=254MiB/s (267MB/s)(29.8GiB/120004msec)
    slat (usec): min=2, max=300387, avg=98.81, stdev=642.76
    clat (nsec): min=514, max=326192k, avg=3833374.99, stdev=5344771.80
     lat (usec): min=15, max=326239, avg=3932.32, stdev=5437.71
    clat percentiles (usec):
     |  1.00th=[  217],  5.00th=[  693], 10.00th=[  857], 20.00th=[ 1139],
     | 30.00th=[ 1418], 40.00th=[ 1811], 50.00th=[ 2376], 60.00th=[ 3130],
     | 70.00th=[ 4015], 80.00th=[ 5145], 90.00th=[ 7308], 95.00th=[11338],
     | 99.00th=[25822], 99.50th=[33817], 99.90th=[56886], 99.95th=[66323],
     | 99.99th=[87557]
   bw (  KiB/s): min= 7398, max=81995, per=12.51%, avg=32563.03, stdev=6393.43, samples=1920
   iops        : min= 1849, max=20498, avg=8140.60, stdev=1598.37, samples=1920
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.05%, 50=0.19%
  lat (usec)   : 100=0.22%, 250=0.72%, 500=1.61%, 750=3.68%, 1000=8.40%
  lat (msec)   : 2=28.96%, 4=26.09%, 10=24.03%, 20=4.27%, 50=1.63%
  lat (msec)   : 100=0.16%, 250=0.01%, 500=0.01%
  cpu          : usr=2.08%, sys=34.78%, ctx=7037215, majf=0, minf=10314
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,7809386,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=254MiB/s (267MB/s), 254MiB/s-254MiB/s (267MB/s-267MB/s), io=29.8GiB (31.0GB), run=120004-120004msec

Disk stats (read/write):
  nvme0n1: ios=0/7820641, merge=0/293550, ticks=0/10477516, in_queue=11478540, util=53.82%

1.2 4k 随机读

fio \
  --name=4k_randread \
  --rw=randread \
  --bs=4k \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=8 \
  --iodepth=32 \
  --size=20G \
  --time_based \
  --runtime=120 \
  --group_reporting \
  --filename=/data1/fio-4k-randread.dat

Starting 8 processes
4k_randread: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [r(8)][100.0%][r=1140MiB/s,w=0KiB/s][r=292k,w=0 IOPS][eta 00m:00s]
4k_randread: (groupid=0, jobs=8): err= 0: pid=58294
   read: IOPS=294k, BW=1147MiB/s (1202MB/s)(134GiB/120006msec)
    slat (nsec): min=1697, max=23878k, avg=5790.82, stdev=16187.41
    clat (usec): min=2, max=26723, avg=864.94, stdev=780.57
     lat (usec): min=43, max=26728, avg=870.84, stdev=780.71
    clat percentiles (usec):
     |  1.00th=[  105],  5.00th=[  135], 10.00th=[  182], 20.00th=[  273],
     | 30.00th=[  375], 40.00th=[  494], 50.00th=[  635], 60.00th=[  799],
     | 70.00th=[ 1012], 80.00th=[ 1319], 90.00th=[ 1844], 95.00th=[ 2376],
     | 99.00th=[ 3687], 99.50th=[ 4293], 99.90th=[ 5866], 99.95th=[ 6652],
     | 99.99th=[ 9241]
   bw (  KiB/s): min=130360, max=164784, per=12.52%, avg=146966.37, stdev=2595.35, samples=1918
   iops        : min=32590, max=41196, avg=36741.37, stdev=648.83, samples=1918
  lat (usec)   : 4=0.01%, 20=0.01%, 50=0.01%, 100=0.28%, 250=17.26%
  lat (usec)   : 500=22.94%, 750=16.82%, 1000=12.10%
  lat (msec)   : 2=22.44%, 4=7.48%, 10=0.68%, 20=0.01%, 50=0.01%
  cpu          : usr=10.07%, sys=29.47%, ctx=19757756, majf=0, minf=8027
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwt: total=35228529,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=1147MiB/s (1202MB/s), 1147MiB/s-1147MiB/s (1202MB/s-1202MB/s), io=134GiB (144GB), run=120006-120006msec

Disk stats (read/write):
  nvme0n1: ios=35183593/4, merge=0/1, ticks=29535284/0, in_queue=33076828, util=100.00%

1.3 4k 随机读写

fio \
  --name=4k_randrw \
  --rw=randrw \
  --rwmixread=70 \
  --bs=4k \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=8 \
  --iodepth=1 \
  --size=20G \
  --time_based \
  --runtime=120 \
  --group_reporting \
  --allow_mounted_write=1 \
  --filename=/data1/fio-4k-randrw.dat

Starting 8 processes
4k_randrw: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=178MiB/s,w=76.4MiB/s][r=45.7k,w=19.6k IOPS][eta 00m:00s]
4k_randrw: (groupid=0, jobs=8): err= 0: pid=39459
   read: IOPS=45.0k, BW=180MiB/s (188MB/s)(21.0GiB/120001msec)
    slat (usec): min=2, max=11092, avg= 6.59, stdev=23.78
    clat (nsec): min=995, max=23512k, avg=155824.83, stdev=117179.25
     lat (usec): min=15, max=23538, avg=162.52, stdev=119.69
    clat percentiles (usec):
     |  1.00th=[   87],  5.00th=[   92], 10.00th=[   98], 20.00th=[  103],
     | 30.00th=[  109], 40.00th=[  118], 50.00th=[  135], 60.00th=[  151],
     | 70.00th=[  165], 80.00th=[  184], 90.00th=[  231], 95.00th=[  273],
     | 99.00th=[  424], 99.50th=[  742], 99.90th=[ 1663], 99.95th=[ 2008],
     | 99.99th=[ 3490]
   bw (  KiB/s): min=15960, max=24321, per=12.52%, avg=23011.22, stdev=652.38, samples=1916
   iops        : min= 3990, max= 6080, avg=5752.59, stdev=163.06, samples=1916
  write: IOPS=19.7k, BW=76.0MiB/s (80.7MB/s)(9236MiB/120001msec)
    slat (usec): min=2, max=7925, avg= 8.38, stdev=23.90
    clat (nsec): min=516, max=11228k, avg=13944.32, stdev=29810.73
     lat (usec): min=11, max=11286, avg=22.44, stdev=38.63
    clat percentiles (usec):
     |  1.00th=[   11],  5.00th=[   11], 10.00th=[   11], 20.00th=[   12],
     | 30.00th=[   12], 40.00th=[   13], 50.00th=[   13], 60.00th=[   14],
     | 70.00th=[   14], 80.00th=[   15], 90.00th=[   16], 95.00th=[   19],
     | 99.00th=[   34], 99.50th=[   48], 99.90th=[   96], 99.95th=[  131],
     | 99.99th=[  955]
   bw (  KiB/s): min= 7080, max=11076, per=12.52%, avg=9865.67, stdev=419.59, samples=1916
   iops        : min= 1770, max= 2769, avg=2466.18, stdev=104.96, samples=1916
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.19%, 20=28.70%, 50=0.96%
  lat (usec)   : 100=10.01%, 250=55.07%, 500=4.51%, 750=0.18%, 1000=0.12%
  lat (msec)   : 2=0.20%, 4=0.03%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=2.69%, sys=8.01%, ctx=8499979, majf=0, minf=1178
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=5514868,2364483,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=180MiB/s (188MB/s), 180MiB/s-180MiB/s (188MB/s-188MB/s), io=21.0GiB (22.6GB), run=120001-120001msec
  WRITE: bw=76.0MiB/s (80.7MB/s), 76.0MiB/s-76.0MiB/s (80.7MB/s-80.7MB/s), io=9236MiB (9685MB), run=120001-120001msec

Disk stats (read/write):
  nvme0n1: ios=5513305/2363838, merge=0/23, ticks=834508/27104, in_queue=739972, util=100.00%

1.4 4M 顺序写

fio \
  --name=4m_seq_write \
  --rw=write \
  --bs=4M \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=1 \
  --iodepth=8 \
  --size=50G \
  --time_based \
  --runtime=60 \
  --group_reporting \
  --directory=/data1

Starting 1 process
4m_seq_write: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=921MiB/s][r=0,w=230 IOPS][eta 00m:00s]
4m_seq_write: (groupid=0, jobs=1): err= 0: pid=60739
  write: IOPS=237, BW=950MiB/s (996MB/s)(55.7GiB/60038msec)
    slat (usec): min=342, max=15127, avg=747.13, stdev=315.68
    clat (msec): min=3, max=228, avg=32.93, stdev=13.08
     lat (msec): min=4, max=229, avg=33.68, stdev=13.07
    clat percentiles (msec):
     |  1.00th=[   14],  5.00th=[   17], 10.00th=[   18], 20.00th=[   21],
     | 30.00th=[   25], 40.00th=[   29], 50.00th=[   32], 60.00th=[   34],
     | 70.00th=[   39], 80.00th=[   44], 90.00th=[   51], 95.00th=[   57],
     | 99.00th=[   70], 99.50th=[   74], 99.90th=[   80], 99.95th=[   85],
     | 99.99th=[  228]
   bw (  KiB/s): min=925696, max=1007616, per=100.00%, avg=972891.75, stdev=19806.08, samples=120
   iops        : min=  226, max=  246, avg=237.47, stdev= 4.82, samples=120
  lat (msec)   : 4=0.01%, 10=0.03%, 20=16.83%, 50=72.57%, 100=10.54%
  lat (msec)   : 250=0.03%
  cpu          : usr=8.25%, sys=9.91%, ctx=13396, majf=0, minf=16435
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,14256,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=950MiB/s (996MB/s), 950MiB/s-950MiB/s (996MB/s-996MB/s), io=55.7GiB (59.8GB), run=60038-60038msec

Disk stats (read/write):
  nvme0n1: ios=0/454702, merge=0/834, ticks=0/12245780, in_queue=12288772, util=100.00%

4M 顺序写多进程和单进程数据差不多。

1.5 4M 顺序读

fio \
  --name=4m_seq_read \
  --rw=read \
  --bs=4M \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=1 \
  --iodepth=8 \
  --size=50G \
  --time_based \
  --runtime=60 \
  --group_reporting \
  --directory=/data1

Starting 1 process
4m_seq_read: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [R(1)][100.0%][r=2919MiB/s,w=0KiB/s][r=729,w=0 IOPS][eta 00m:00s]
4m_seq_read: (groupid=0, jobs=1): err= 0: pid=39149
   read: IOPS=722, BW=2890MiB/s (3031MB/s)(169GiB/60008msec)
    slat (usec): min=170, max=6405, avg=388.71, stdev=172.35
    clat (usec): min=2039, max=35296, avg=10676.32, stdev=1174.56
     lat (usec): min=3152, max=40049, avg=11065.58, stdev=1163.86
    clat percentiles (usec):
     |  1.00th=[ 7635],  5.00th=[ 8717], 10.00th=[ 9110], 20.00th=[ 9765],
     | 30.00th=[10159], 40.00th=[10552], 50.00th=[10814], 60.00th=[11076],
     | 70.00th=[11338], 80.00th=[11600], 90.00th=[11994], 95.00th=[12256],
     | 99.00th=[13042], 99.50th=[13435], 99.90th=[15533], 99.95th=[16188],
     | 99.99th=[28443]
   bw (  MiB/s): min= 2720, max= 3160, per=100.00%, avg=2890.76, stdev=149.46, samples=120
   iops        : min=  680, max=  790, avg=722.66, stdev=37.37, samples=120
  lat (msec)   : 4=0.03%, 10=26.26%, 20=73.69%, 50=0.02%
  cpu          : usr=0.49%, sys=28.86%, ctx=41669, majf=0, minf=24632
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=43363,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=2890MiB/s (3031MB/s), 2890MiB/s-2890MiB/s (3031MB/s-3031MB/s), io=169GiB (182GB), run=60008-60008msec

Disk stats (read/write):
  nvme0n1: ios=1382412/5, merge=0/1, ticks=7162440/0, in_queue=7266616, util=100.00%

4M 顺序读多进程和单进程数据差不多。

2. RAID 0

2.1 创建 RAID 0

查看磁盘

`1`	`lsblk -d -o NAME,SIZE,TYPE \| grep nvme`

nvme0n1 745.2G disk
nvme1n1 745.2G disk
nvme2n1 745.2G disk
nvme3n1 745.2G disk
nvme4n1 745.2G disk
nvme5n1 745.2G disk

创建 RAID

`1`	`mdadm --create --verbose /dev/md0 --level=0 --raid-devices=5 /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1`

这里的 level=0 表示 RAID 0，raid-devices=5 表示使用 5 块盘。

查看 RAID 状态

`1`	`mdadm --detail /dev/md0`

等盘 State : clean 再操作。

格式化 RAID

挂载 RAID

1
2

mkdir -p /mnt/mdadm
mount /dev/md0 /mnt/mdadm

查看磁盘大小

1
2

Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        3.7T  3.8G  3.7T   1% /mnt/mdadm

2.2 4k 随机写

fio \
  --name=4k_randwrite \
  --rw=randwrite \
  --bs=4k \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=8 \
  --iodepth=32 \
  --size=20G \
  --time_based \
  --runtime=120 \
  --group_reporting \
  --allow_mounted_write=1 \
  --filename=/mnt/mdadm/fio-4k-randwrite.dat

Starting 8 processes
4k_randwrite: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [w(8)][100.0%][r=0KiB/s,w=471MiB/s][r=0,w=121k IOPS][eta 00m:00s]
4k_randwrite: (groupid=0, jobs=8): err= 0: pid=21458
  write: IOPS=87.9k, BW=343MiB/s (360MB/s)(40.2GiB/120001msec)
    slat (usec): min=3, max=480801, avg=87.68, stdev=558.72
    clat (nsec): min=1023, max=495513k, avg=2824145.32, stdev=4122691.21
     lat (usec): min=15, max=495562, avg=2911.98, stdev=4197.27
    clat percentiles (usec):
     |  1.00th=[  537],  5.00th=[  799], 10.00th=[  914], 20.00th=[ 1090],
     | 30.00th=[ 1254], 40.00th=[ 1434], 50.00th=[ 1663], 60.00th=[ 1975],
     | 70.00th=[ 2507], 80.00th=[ 3556], 90.00th=[ 5997], 95.00th=[ 8979],
     | 99.00th=[17171], 99.50th=[21103], 99.90th=[30802], 99.95th=[34866],
     | 99.99th=[48497]
   bw (  KiB/s): min= 2076, max=104449, per=12.52%, avg=43999.32, stdev=19351.22, samples=1920
   iops        : min=  519, max=26112, avg=10999.54, stdev=4837.80, samples=1920
  lat (usec)   : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
  lat (usec)   : 250=0.04%, 500=0.74%, 750=2.84%, 1000=11.08%
  lat (msec)   : 2=45.76%, 4=22.23%, 10=13.25%, 20=3.45%, 50=0.60%
  lat (msec)   : 100=0.01%, 250=0.01%, 500=0.01%
  cpu          : usr=2.51%, sys=47.82%, ctx=8084183, majf=0, minf=2705
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,10544663,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=343MiB/s (360MB/s), 343MiB/s-343MiB/s (360MB/s-360MB/s), io=40.2GiB (43.2GB), run=120001-120001msec

Disk stats (read/write):
    md0: ios=1/10532623, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/2110560, aggrmerge=0/0, aggrticks=0/17402, aggrin_queue=472, aggrutil=0.41%
  nvme0n1: ios=0/2111164, merge=0/1, ticks=0/18048, in_queue=700, util=0.41%
  nvme3n1: ios=0/2109897, merge=0/0, ticks=0/17404, in_queue=448, util=0.31%
  nvme2n1: ios=0/2108603, merge=0/0, ticks=0/17316, in_queue=412, util=0.30%
  nvme1n1: ios=1/2114000, merge=0/0, ticks=0/17700, in_queue=380, util=0.29%
  nvme4n1: ios=0/2109140, merge=0/0, ticks=0/16544, in_queue=424, util=0.24%

2.3 4k 随机读

fio \
  --name=4k_randread \
  --rw=randread \
  --bs=4k \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=8 \
  --iodepth=32 \
  --size=20G \
  --time_based \
  --runtime=120 \
  --group_reporting \
  --filename=/mnt/mdadm/fio-4k-randread.dat

Starting 8 processes
4k_randread: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [r(8)][100.0%][r=3783MiB/s,w=0KiB/s][r=969k,w=0 IOPS][eta 00m:00s]
4k_randread: (groupid=0, jobs=8): err= 0: pid=49074
   read: IOPS=1009k, BW=3940MiB/s (4132MB/s)(462GiB/120002msec)
    slat (nsec): min=1916, max=23433k, avg=6686.82, stdev=13633.31
    clat (usec): min=34, max=23910, avg=246.46, stdev=104.87
     lat (usec): min=38, max=23922, avg=253.20, stdev=106.05
    clat percentiles (usec):
     |  1.00th=[  153],  5.00th=[  172], 10.00th=[  182], 20.00th=[  194],
     | 30.00th=[  206], 40.00th=[  217], 50.00th=[  229], 60.00th=[  245],
     | 70.00th=[  265], 80.00th=[  289], 90.00th=[  334], 95.00th=[  375],
     | 99.00th=[  474], 99.50th=[  519], 99.90th=[  652], 99.95th=[  742],
     | 99.99th=[ 1876]
   bw (  KiB/s): min=323448, max=598204, per=12.51%, avg=504829.01, stdev=36370.66, samples=1920
   iops        : min=80862, max=149551, avg=126207.05, stdev=9092.64, samples=1920
  lat (usec)   : 50=0.01%, 100=0.01%, 250=62.79%, 500=36.53%, 750=0.62%
  lat (usec)   : 1000=0.03%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=16.66%, sys=82.78%, ctx=96372, majf=0, minf=5346
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwt: total=121052850,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=3940MiB/s (4132MB/s), 3940MiB/s-3940MiB/s (4132MB/s-4132MB/s), io=462GiB (496GB), run=120002-120002msec

Disk stats (read/write):
    md0: ios=120867592/15, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=24210570/2, aggrmerge=0/0, aggrticks=3085131/0, aggrin_queue=3179908, aggrutil=100.00%
  nvme0n1: ios=24210257/8, merge=0/2, ticks=3083560/0, in_queue=3146512, util=100.00%
  nvme3n1: ios=24209120/0, merge=0/0, ticks=3085116/0, in_queue=3163832, util=100.00%
  nvme2n1: ios=24210502/0, merge=0/0, ticks=3088560/0, in_queue=3243228, util=100.00%
  nvme1n1: ios=24211505/2, merge=0/0, ticks=3076108/0, in_queue=3173424, util=100.00%
  nvme4n1: ios=24211466/3, merge=0/0, ticks=3092312/0, in_queue=3172548, util=100.00%

2.4 4k 随机读写

fio \
  --name=4k_randrw \
  --rw=randrw \
  --rwmixread=70 \
  --bs=4k \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=8 \
  --iodepth=1 \
  --size=20G \
  --time_based \
  --runtime=120 \
  --group_reporting \
  --allow_mounted_write=1 \
  --filename=/mnt/mdadm/fio-4k-randrw.dat

Starting 8 processes
4k_randrw: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=246MiB/s,w=105MiB/s][r=63.1k,w=26.0k IOPS][eta 00m:00s]
4k_randrw: (groupid=0, jobs=8): err= 0: pid=42379
   read: IOPS=62.6k, BW=245MiB/s (257MB/s)(28.7GiB/120001msec)
    slat (usec): min=2, max=15986, avg= 8.75, stdev=43.09
    clat (nsec): min=725, max=20233k, avg=100358.72, stdev=59825.88
     lat (usec): min=14, max=23745, avg=109.26, stdev=75.95
    clat percentiles (usec):
     |  1.00th=[   86],  5.00th=[   87], 10.00th=[   88], 20.00th=[   89],
     | 30.00th=[   90], 40.00th=[   92], 50.00th=[   94], 60.00th=[   99],
     | 70.00th=[  105], 80.00th=[  106], 90.00th=[  113], 95.00th=[  127],
     | 99.00th=[  172], 99.50th=[  188], 99.90th=[  302], 99.95th=[  519],
     | 99.99th=[ 2966]
   bw (  KiB/s): min=10072, max=33010, per=12.51%, avg=31349.65, stdev=1699.73, samples=1912
   iops        : min= 2518, max= 8252, avg=7837.21, stdev=424.93, samples=1912
  write: IOPS=26.9k, BW=105MiB/s (110MB/s)(12.3GiB/120001msec)
    slat (usec): min=3, max=20195, avg=13.70, stdev=51.79
    clat (nsec): min=572, max=307733k, avg=22953.37, stdev=287025.71
     lat (usec): min=13, max=307770, avg=36.82, stdev=291.95
    clat percentiles (usec):
     |  1.00th=[   13],  5.00th=[   16], 10.00th=[   16], 20.00th=[   18],
     | 30.00th=[   18], 40.00th=[   19], 50.00th=[   20], 60.00th=[   20],
     | 70.00th=[   21], 80.00th=[   23], 90.00th=[   25], 95.00th=[   30],
     | 99.00th=[   56], 99.50th=[   75], 99.90th=[  289], 99.95th=[  930],
     | 99.99th=[ 4359]
   bw (  KiB/s): min= 4224, max=15014, per=12.51%, avg=13438.51, stdev=799.40, samples=1912
   iops        : min= 1056, max= 3753, avg=3359.39, stdev=199.83, samples=1912
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.04%, 4=0.01%, 10=0.01%, 20=18.55%, 50=11.01%
  lat (usec)   : 100=42.95%, 250=27.27%, 500=0.10%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.01%, 250=0.01%, 500=0.01%
  cpu          : usr=4.46%, sys=15.06%, ctx=11607725, majf=0, minf=2890
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=7517356,3222298,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=245MiB/s (257MB/s), 245MiB/s-245MiB/s (257MB/s-257MB/s), io=28.7GiB (30.8GB), run=120001-120001msec
  WRITE: bw=105MiB/s (110MB/s), 105MiB/s-105MiB/s (110MB/s-110MB/s), io=12.3GiB (13.2GB), run=120001-120001msec

Disk stats (read/write):
    md0: ios=7510767/3219671, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=1503471/644462, aggrmerge=0/0, aggrticks=145287/6216, aggrin_queue=69136, aggrutil=41.33%
  nvme0n1: ios=1502724/643760, merge=0/3, ticks=144384/6196, in_queue=67528, util=40.15%
  nvme3n1: ios=1501741/644320, merge=0/1, ticks=144768/6540, in_queue=69136, util=41.33%
  nvme2n1: ios=1506705/644691, merge=0/0, ticks=146568/6288, in_queue=70480, util=41.25%
  nvme1n1: ios=1503224/645075, merge=0/0, ticks=145016/6036, in_queue=69064, util=41.23%
  nvme4n1: ios=1502962/644466, merge=0/0, ticks=145700/6020, in_queue=69476, util=40.81%

2.5 4M 顺序写

fio \
  --name=4m_seq_write \
  --rw=write \
  --bs=4M \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=1 \
  --iodepth=8 \
  --size=50G \
  --time_based \
  --runtime=60 \
  --group_reporting \
  --directory=/mnt/mdadm

Starting 1 process
4m_seq_write: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=4844MiB/s][r=0,w=1211 IOPS][eta 00m:00s]
4m_seq_write: (groupid=0, jobs=1): err= 0: pid=27148
  write: IOPS=1203, BW=4815MiB/s (5049MB/s)(282GiB/60001msec)
    slat (usec): min=287, max=7225, avg=681.36, stdev=196.26
    clat (usec): min=411, max=33510, avg=5961.80, stdev=1525.37
     lat (usec): min=921, max=37964, avg=6643.65, stdev=1539.31
    clat percentiles (usec):
     |  1.00th=[ 4047],  5.00th=[ 4621], 10.00th=[ 4948], 20.00th=[ 5276],
     | 30.00th=[ 5473], 40.00th=[ 5669], 50.00th=[ 5735], 60.00th=[ 5866],
     | 70.00th=[ 5997], 80.00th=[ 6194], 90.00th=[ 6652], 95.00th=[ 7767],
     | 99.00th=[14091], 99.50th=[15270], 99.90th=[17957], 99.95th=[24249],
     | 99.99th=[30016]
   bw (  MiB/s): min= 4552, max= 5058, per=100.00%, avg=4818.99, stdev=119.09, samples=120
   iops        : min= 1138, max= 1264, avg=1204.55, stdev=29.75, samples=120
  lat (usec)   : 500=0.01%, 750=0.01%
  lat (msec)   : 2=0.01%, 4=0.74%, 10=96.68%, 20=2.50%, 50=0.06%
  cpu          : usr=41.67%, sys=39.12%, ctx=46501, majf=0, minf=8232
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,72222,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=4815MiB/s (5049MB/s), 4815MiB/s-4815MiB/s (5049MB/s-5049MB/s), io=282GiB (303GB), run=60001-60001msec

Disk stats (read/write):
    md0: ios=0/2305321, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/462222, aggrmerge=0/7, aggrticks=0/809132, aggrin_queue=763212, aggrutil=78.44%
  nvme0n1: ios=0/462225, merge=0/6, ticks=0/723764, in_queue=678840, util=74.29%
  nvme3n1: ios=0/462220, merge=0/6, ticks=0/867552, in_queue=821964, util=78.06%
  nvme2n1: ios=0/462220, merge=0/6, ticks=0/652296, in_queue=604092, util=72.10%
  nvme1n1: ios=0/462225, merge=0/6, ticks=0/1077684, in_queue=1036168, util=78.44%
  nvme4n1: ios=0/462223, merge=0/11, ticks=0/724364, in_queue=675000, util=73.74%

2.6 4M 顺序读

fio \
  --name=4m_seq_read \
  --rw=read \
  --bs=4M \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=1 \
  --iodepth=8 \
  --size=50G \
  --time_based \
  --runtime=60 \
  --group_reporting \
  --directory=/mnt/mdadm

Jobs: 1 (f=1): [R(1)][100.0%][r=13.4GiB/s,w=0KiB/s][r=3430,w=0 IOPS][eta 00m:00s]
4m_seq_read: (groupid=0, jobs=1): err= 0: pid=17975
   read: IOPS=3385, BW=13.2GiB/s (14.2GB/s)(794GiB/60002msec)
    slat (usec): min=157, max=25717, avg=287.34, stdev=116.19
    clat (usec): min=543, max=28567, avg=2073.83, stdev=510.64
     lat (usec): min=743, max=29028, avg=2361.44, stdev=559.20
    clat percentiles (usec):
     |  1.00th=[ 1336],  5.00th=[ 1516], 10.00th=[ 1614], 20.00th=[ 1713],
     | 30.00th=[ 1795], 40.00th=[ 1893], 50.00th=[ 1975], 60.00th=[ 2057],
     | 70.00th=[ 2180], 80.00th=[ 2343], 90.00th=[ 2704], 95.00th=[ 3032],
     | 99.00th=[ 3720], 99.50th=[ 3982], 99.90th=[ 5080], 99.95th=[ 5735],
     | 99.99th=[ 7504]
   bw (  MiB/s): min=11046, max=15038, per=100.00%, avg=13554.51, stdev=812.98, samples=120
   iops        : min= 2761, max= 3759, avg=3388.38, stdev=203.25, samples=120
  lat (usec)   : 750=0.01%, 1000=0.04%
  lat (msec)   : 2=52.20%, 4=47.29%, 10=0.46%, 20=0.01%, 50=0.01%
  cpu          : usr=0.81%, sys=97.04%, ctx=9999, majf=0, minf=8211
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=203138,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=13.2GiB/s (14.2GB/s), 13.2GiB/s-13.2GiB/s (14.2GB/s-14.2GB/s), io=794GiB (852GB), run=60002-60002msec

Disk stats (read/write):
    md0: ios=7321063/16, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=1467827/2, aggrmerge=9/1, aggrticks=1054376/0, aggrin_queue=962353, aggrutil=99.57%
  nvme0n1: ios=1467825/6, merge=0/4, ticks=1057288/0, in_queue=966596, util=99.57%
  nvme3n1: ios=1467745/2, merge=16/1, ticks=1063604/0, in_queue=971024, util=99.50%
  nvme2n1: ios=1467840/0, merge=0/0, ticks=1050392/0, in_queue=958220, util=99.53%
  nvme1n1: ios=1467885/2, merge=16/0, ticks=1042768/0, in_queue=951248, util=99.53%
  nvme4n1: ios=1467840/1, merge=16/0, ticks=1057832/0, in_queue=964680, util=99.50%

3. RAID 1

3.1 创建 RAID 1

查看磁盘

`1`	`lsblk -d -o NAME,SIZE,TYPE \| grep nvme`

nvme0n1 745.2G disk
nvme1n1 745.2G disk
nvme2n1 745.2G disk
nvme3n1 745.2G disk
nvme4n1 745.2G disk
nvme5n1 745.2G disk

停掉 md0

1
2

umount /mnt/mdadm
mdadm --stop /dev/md0

创建 RAID

`1`	`mdadm --create --verbose /dev/md0 --level=1 --raid-devices=5 /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1`

这里的 level=1 表示 RAID 1，raid-devices=5 表示使用 5 块盘。

查看 RAID 状态

`1`	`mdadm --detail /dev/md0`

等盘 State : clean 再操作。

格式化 RAID

挂载 RAID

1
2

mkdir -p /mnt/mdadm
mount /dev/md0 /mnt/mdadm

查看磁盘大小

1
2

Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        745G  5.3G  740G   1% /mnt/mdadm

3.2 4k 随机写

fio \
  --name=4k_randwrite \
  --rw=randwrite \
  --bs=4k \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=8 \
  --iodepth=32 \
  --size=20G \
  --time_based \
  --runtime=120 \
  --group_reporting \
  --allow_mounted_write=1 \
  --filename=/mnt/mdadm/fio-4k-randwrite.dat

Starting 8 processes
4k_randwrite: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [w(8)][100.0%][w=667MiB/s][w=171k IOPS][eta 00m:00s]
4k_randwrite: (groupid=0, jobs=8): err= 0: pid=2652550
  write: IOPS=82.4k, BW=322MiB/s (338MB/s)(37.7GiB/120002msec); 0 zone resets
    slat (usec): min=9, max=97750, avg=89.03, stdev=302.57
    clat (nsec): min=1173, max=113435k, avg=3013092.85, stdev=2295088.76
     lat (usec): min=31, max=113694, avg=3102.50, stdev=2341.60
    clat percentiles (usec):
     |  1.00th=[  955],  5.00th=[ 1254], 10.00th=[ 1418], 20.00th=[ 1647],
     | 30.00th=[ 1860], 40.00th=[ 2089], 50.00th=[ 2376], 60.00th=[ 2737],
     | 70.00th=[ 3261], 80.00th=[ 4015], 90.00th=[ 5211], 95.00th=[ 6456],
     | 99.00th=[11207], 99.50th=[15008], 99.90th=[23462], 99.95th=[28705],
     | 99.99th=[48497]
   bw (  KiB/s): min=115696, max=692616, per=99.75%, avg=328859.77, stdev=18064.76, samples=1912
   iops        : min=28924, max=173154, avg=82214.49, stdev=4516.17, samples=1912
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.02%, 750=0.20%, 1000=1.02%
  lat (msec)   : 2=35.14%, 4=43.29%, 10=18.99%, 20=1.14%, 50=0.18%
  lat (msec)   : 100=0.01%, 250=0.01%
  cpu          : usr=5.14%, sys=25.04%, ctx=8249593, majf=0, minf=8356
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,9890927,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=322MiB/s (338MB/s), 322MiB/s-322MiB/s (338MB/s-338MB/s), io=37.7GiB (40.5GB), run=120002-120002msec

Disk stats (read/write):
    md0: ios=0/10261391, merge=0/0, ticks=0/3865644, in_queue=3865644, util=98.87%, aggrios=37784/10387166, aggrmerge=37454/191430, aggrticks=5046/1061219, aggrin_queue=1066265, aggrutil=99.44%
  nvme0n1: ios=188924/10236122, merge=187272/41518, ticks=25231/402344, in_queue=427575, util=99.41%
  nvme3n1: ios=0/10425016, merge=0/228820, ticks=0/990071, in_queue=990071, util=99.43%
  nvme2n1: ios=0/10425311, merge=0/228525, ticks=0/1238220, in_queue=1238219, util=99.43%
  nvme1n1: ios=0/10425217, merge=0/228619, ticks=0/1453918, in_queue=1453919, util=99.42%
  nvme4n1: ios=0/10424164, merge=0/229672, ticks=0/1221545, in_queue=1221545, util=99.44%

3.3 4k 随机读

fio \
  --name=4k_randread \
  --rw=randread \
  --bs=4k \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=8 \
  --iodepth=32 \
  --size=20G \
  --time_based \
  --runtime=120 \
  --group_reporting \
  --filename=/mnt/mdadm/fio-4k-randread.dat

Starting 8 processes
4k_randread: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=0): [f(8)][100.0%][r=3149MiB/s][r=806k IOPS][eta 00m:00s]
4k_randread: (groupid=0, jobs=8): err= 0: pid=2683004
  read: IOPS=734k, BW=2869MiB/s (3008MB/s)(336GiB/120001msec)
    slat (usec): min=2, max=9386, avg= 7.80, stdev= 5.01
    clat (usec): min=30, max=11243, avg=339.88, stdev=237.70
     lat (usec): min=34, max=11250, avg=347.77, stdev=237.83
    clat percentiles (usec):
     |  1.00th=[  112],  5.00th=[  174], 10.00th=[  200], 20.00th=[  235],
     | 30.00th=[  255], 40.00th=[  273], 50.00th=[  289], 60.00th=[  310],
     | 70.00th=[  338], 80.00th=[  375], 90.00th=[  457], 95.00th=[  652],
     | 99.00th=[ 1483], 99.50th=[ 1811], 99.90th=[ 2606], 99.95th=[ 2999],
     | 99.99th=[ 3949]
   bw (  MiB/s): min= 1674, max= 4300, per=100.00%, avg=2869.67, stdev=81.34, samples=1912
   iops        : min=428666, max=1100852, avg=734634.40, stdev=20824.14, samples=1912
  lat (usec)   : 50=0.01%, 100=0.53%, 250=26.95%, 500=64.53%, 750=3.90%
  lat (usec)   : 1000=1.54%
  lat (msec)   : 2=2.19%, 4=0.33%, 10=0.01%, 20=0.01%
  cpu          : usr=19.84%, sys=70.85%, ctx=4603551, majf=0, minf=7479
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=88136288,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=2869MiB/s (3008MB/s), 2869MiB/s-2869MiB/s (3008MB/s-3008MB/s), io=336GiB (361GB), run=120001-120001msec

Disk stats (read/write):
    md0: ios=87946164/6, merge=0/0, ticks=17355860/0, in_queue=17355860, util=100.00%, aggrios=17665861/154492, aggrmerge=37864/151457, aggrticks=3493488/18341, aggrin_queue=3511830, aggrutil=100.00%
  nvme0n1: ios=30093595/79, merge=189323/0, ticks=8458371/1, in_queue=8458373, util=100.00%
  nvme3n1: ios=14502719/193095, merge=0/189324, ticks=2242312/21979, in_queue=2264292, util=87.26%
  nvme2n1: ios=14586400/193093, merge=0/189323, ticks=2262548/21228, in_queue=2283777, util=87.26%
  nvme1n1: ios=14741596/193106, merge=0/189321, ticks=2284985/22290, in_queue=2307275, util=87.26%
  nvme4n1: ios=14404995/193090, merge=0/189320, ticks=2219228/26209, in_queue=2245436, util=87.26%

3.4 4k 随机读写

fio \
  --name=4k_randrw \
  --rw=randrw \
  --rwmixread=70 \
  --bs=4k \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=8 \
  --iodepth=1 \
  --size=20G \
  --time_based \
  --runtime=120 \
  --group_reporting \
  --allow_mounted_write=1 \
  --filename=/mnt/mdadm/fio-4k-randrw.dat

Starting 8 processes
4k_randrw: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=247MiB/s,w=105MiB/s][r=63.2k,w=26.9k IOPS][eta 00m:00s]
4k_randrw: (groupid=0, jobs=8): err= 0: pid=2707685
  read: IOPS=57.6k, BW=225MiB/s (236MB/s)(26.4GiB/120001msec)
    slat (usec): min=3, max=8035, avg= 7.60, stdev=10.20
    clat (nsec): min=920, max=16185k, avg=107158.69, stdev=52548.88
     lat (usec): min=16, max=16192, avg=114.84, stdev=53.93
    clat percentiles (usec):
     |  1.00th=[   83],  5.00th=[   84], 10.00th=[   85], 20.00th=[   87],
     | 30.00th=[   88], 40.00th=[   90], 50.00th=[   97], 60.00th=[  102],
     | 70.00th=[  105], 80.00th=[  128], 90.00th=[  149], 95.00th=[  163],
     | 99.00th=[  229], 99.50th=[  258], 99.90th=[  302], 99.95th=[  396],
     | 99.99th=[ 2311]
   bw (  KiB/s): min=179614, max=263334, per=100.00%, avg=230318.28, stdev=3146.19, samples=1912
   iops        : min=44903, max=65833, avg=57579.30, stdev=786.56, samples=1912
  write: IOPS=24.7k, BW=96.4MiB/s (101MB/s)(11.3GiB/120001msec); 0 zone resets
    slat (usec): min=8, max=6734, avg=21.99, stdev=16.46
    clat (nsec): min=679, max=22236k, avg=29493.50, stdev=96588.38
     lat (usec): min=21, max=22264, avg=51.86, stdev=97.88
    clat percentiles (nsec):
     |  1.00th=[   1256],  5.00th=[   5216], 10.00th=[  13120],
     | 20.00th=[  14912], 30.00th=[  15808], 40.00th=[  16768],
     | 50.00th=[  17536], 60.00th=[  18560], 70.00th=[  20096],
     | 80.00th=[  23424], 90.00th=[  45824], 95.00th=[  92672],
     | 99.00th=[ 264192], 99.50th=[ 358400], 99.90th=[ 577536],
     | 99.95th=[ 765952], 99.99th=[4046848]
   bw (  KiB/s): min=75695, max=117144, per=100.00%, avg=98725.13, stdev=1379.97, samples=1912
   iops        : min=18923, max=29286, avg=24680.95, stdev=345.02, samples=1912
  lat (nsec)   : 750=0.01%, 1000=0.08%
  lat (usec)   : 2=1.09%, 4=0.25%, 10=1.13%, 20=18.19%, 50=6.41%
  lat (usec)   : 100=38.67%, 250=33.44%, 500=0.67%, 750=0.04%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=4.14%, sys=14.80%, ctx=9901027, majf=0, minf=7895
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=6908303,2961139,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=225MiB/s (236MB/s), 225MiB/s-225MiB/s (236MB/s-236MB/s), io=26.4GiB (28.3GB), run=120001-120001msec
  WRITE: bw=96.4MiB/s (101MB/s), 96.4MiB/s-96.4MiB/s (101MB/s-101MB/s), io=11.3GiB (12.1GB), run=120001-120001msec

Disk stats (read/write):
    md0: ios=6907167/2960677, merge=0/0, ticks=712932/107392, in_queue=820324, util=100.00%, aggrios=1422070/3123074, aggrmerge=40251/161015, aggrticks=150936/102939, aggrin_queue=253875, aggrutil=100.00%
  nvme0n1: ios=2815994/2961442, merge=201255/4, ticks=284862/50206, in_queue=335068, util=100.00%
  nvme3n1: ios=885144/3163482, merge=0/201268, ticks=96499/112731, in_queue=209229, util=100.00%
  nvme2n1: ios=1124500/3163482, merge=0/201268, ticks=123379/122800, in_queue=246179, util=100.00%
  nvme1n1: ios=1496981/3163482, merge=0/201268, ticks=164100/104848, in_queue=268948, util=100.00%
  nvme4n1: ios=787733/3163482, merge=0/201268, ticks=85842/124112, in_queue=209954, util=100.00%

3.5 4M 顺序写

fio \
  --name=4m_seq_write \
  --rw=write \
  --bs=4M \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=1 \
  --iodepth=8 \
  --size=50G \
  --time_based \
  --runtime=60 \
  --group_reporting \
  --directory=/mnt/mdadm

Starting 1 process
4m_seq_write: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=977MiB/s][w=244 IOPS][eta 00m:00s]
4m_seq_write: (groupid=0, jobs=1): err= 0: pid=2739437
  write: IOPS=242, BW=969MiB/s (1016MB/s)(56.8GiB/60029msec); 0 zone resets
    slat (usec): min=1240, max=40022, avg=3601.34, stdev=6282.34
    clat (usec): min=6126, max=79244, avg=29402.88, stdev=15252.97
     lat (usec): min=11895, max=81106, avg=33005.52, stdev=14984.86
    clat percentiles (usec):
     |  1.00th=[11338],  5.00th=[12256], 10.00th=[12911], 20.00th=[14353],
     | 30.00th=[16057], 40.00th=[18220], 50.00th=[22152], 60.00th=[36963],
     | 70.00th=[43779], 80.00th=[46924], 90.00th=[49546], 95.00th=[52167],
     | 99.00th=[56886], 99.50th=[58983], 99.90th=[66847], 99.95th=[70779],
     | 99.99th=[76022]
   bw (  KiB/s): min=925696, max=1048576, per=100.00%, avg=993005.50, stdev=32797.75, samples=119
   iops        : min=  226, max=  256, avg=242.43, stdev= 8.01, samples=119
  lat (msec)   : 10=0.01%, 20=45.48%, 50=45.13%, 100=9.38%
  cpu          : usr=8.75%, sys=40.43%, ctx=9015, majf=0, minf=9275
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,14543,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=969MiB/s (1016MB/s), 969MiB/s-969MiB/s (1016MB/s-1016MB/s), io=56.8GiB (61.0GB), run=60029-60029msec

Disk stats (read/write):
    md0: ios=0/466632, merge=0/0, ticks=0/9388192, in_queue=9388192, util=100.00%, aggrios=2530/478947, aggrmerge=2509/10047, aggrticks=738/6505043, aggrin_queue=6505781, aggrutil=99.54%
  nvme0n1: ios=12650/468830, merge=12549/7, ticks=3691/5552350, in_queue=5556041, util=98.32%
  nvme3n1: ios=0/481465, merge=0/12557, ticks=0/6885075, in_queue=6885075, util=99.54%
  nvme2n1: ios=0/481480, merge=0/12557, ticks=0/6705007, in_queue=6705007, util=99.31%
  nvme1n1: ios=0/481480, merge=0/12558, ticks=0/6724055, in_queue=6724055, util=99.37%
  nvme4n1: ios=0/481481, merge=0/12558, ticks=0/6658729, in_queue=6658729, util=99.28%

3.6 4M 顺序读

fio \
  --name=4m_seq_read \
  --rw=read \
  --bs=4M \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=1 \
  --iodepth=8 \
  --size=50G \
  --time_based \
  --runtime=60 \
  --group_reporting \
  --directory=/mnt/mdadm

Starting 1 process
4m_seq_read: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [R(1)][100.0%][r=2991MiB/s][r=747 IOPS][eta 00m:00s]
4m_seq_read: (groupid=0, jobs=1): err= 0: pid=2764172
  read: IOPS=762, BW=3049MiB/s (3197MB/s)(179GiB/60005msec)
    slat (usec): min=249, max=3163, avg=530.66, stdev=184.39
    clat (usec): min=768, max=21504, avg=9960.66, stdev=1419.33
     lat (usec): min=1591, max=21939, avg=10491.80, stdev=1394.86
    clat percentiles (usec):
     |  1.00th=[ 3326],  5.00th=[ 8029], 10.00th=[ 8848], 20.00th=[ 9372],
     | 30.00th=[ 9765], 40.00th=[ 9896], 50.00th=[10159], 60.00th=[10290],
     | 70.00th=[10552], 80.00th=[10814], 90.00th=[11207], 95.00th=[11600],
     | 99.00th=[12518], 99.50th=[13042], 99.90th=[14222], 99.95th=[15008],
     | 99.99th=[18482]
   bw (  MiB/s): min= 2810, max= 3944, per=100.00%, avg=3050.60, stdev=212.52, samples=119
   iops        : min=  702, max=  986, avg=762.57, stdev=53.14, samples=119
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.39%, 4=0.92%, 10=42.17%, 20=56.51%, 50=0.01%
  cpu          : usr=0.54%, sys=41.41%, ctx=41948, majf=0, minf=8209
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=45733,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=3049MiB/s (3197MB/s), 3049MiB/s-3049MiB/s (3197MB/s-3197MB/s), io=179GiB (192GB), run=60005-60005msec

Disk stats (read/write):
    md0: ios=1489199/2, merge=0/0, ticks=10292636/0, in_queue=10292636, util=100.00%, aggrios=299771/9965, aggrmerge=2993/9768, aggrticks=2063822/2188, aggrin_queue=2066011, aggrutil=100.00%
  nvme0n1: ios=1438808/39, merge=14867/0, ticks=9987926/12, in_queue=9987937, util=100.00%
  nvme3n1: ios=0/12445, merge=0/12211, ticks=0/2339, in_queue=2339, util=59.79%
  nvme2n1: ios=1041/12437, merge=2/12212, ticks=5788/2668, in_queue=8456, util=59.73%
  nvme1n1: ios=58368/12458, merge=95/12211, ticks=322095/3827, in_queue=325922, util=60.20%
  nvme4n1: ios=640/12446, merge=2/12210, ticks=3305/2097, in_queue=5402, util=60.14%

3.7 4M 多进程读

fio \
  --name=4m_seq_read \
  --rw=read \
  --bs=4M \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=4 \
  --iodepth=8 \
  --size=50G \
  --time_based \
  --runtime=60 \
  --group_reporting \
  --directory=/mnt/mdadm

Starting 4 processes
4m_seq_read: Laying out IO file (1 file / 51200MiB)
4m_seq_read: Laying out IO file (1 file / 51200MiB)
4m_seq_read: Laying out IO file (1 file / 51200MiB)
Jobs: 4 (f=4): [R(4)][100.0%][r=6671MiB/s][r=1667 IOPS][eta 00m:00s]
4m_seq_read: (groupid=0, jobs=4): err= 0: pid=2820380
  read: IOPS=1729, BW=6919MiB/s (7255MB/s)(406GiB/60024msec)
    slat (usec): min=193, max=3597, avg=509.02, stdev=173.82
    clat (usec): min=728, max=135913, avg=17982.17, stdev=12712.06
     lat (usec): min=1474, max=136601, avg=18491.66, stdev=12703.65
    clat percentiles (usec):
     |  1.00th=[  1778],  5.00th=[  4490], 10.00th=[  6915], 20.00th=[  9110],
     | 30.00th=[  9765], 40.00th=[ 10290], 50.00th=[ 11207], 60.00th=[ 16319],
     | 70.00th=[ 22414], 80.00th=[ 30016], 90.00th=[ 36963], 95.00th=[ 42730],
     | 99.00th=[ 54789], 99.50th=[ 60556], 99.90th=[ 73925], 99.95th=[ 80217],
     | 99.99th=[105382]
   bw (  MiB/s): min= 2728, max=11256, per=100.00%, avg=6927.51, stdev=618.57, samples=476
   iops        : min=  682, max= 2814, avg=1731.83, stdev=154.64, samples=476
  lat (usec)   : 750=0.01%, 1000=0.02%
  lat (msec)   : 2=1.33%, 4=2.84%, 10=29.61%, 20=32.01%, 50=32.26%
  lat (msec)   : 100=1.91%, 250=0.01%
  cpu          : usr=0.33%, sys=22.52%, ctx=90244, majf=0, minf=38564
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=103825,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=6919MiB/s (7255MB/s), 6919MiB/s-6919MiB/s (7255MB/s-7255MB/s), io=406GiB (435GB), run=60024-60024msec

Disk stats (read/write):
    md0: ios=3589357/2, merge=0/0, ticks=50536872/24, in_queue=50536896, util=100.00%, aggrios=720567/7054, aggrmerge=1793/6945, aggrticks=10133686/1979, aggrin_queue=10135665, aggrutil=100.00%
  nvme0n1: ios=1595779/42, merge=8724/0, ticks=37877166/106, in_queue=37877272, util=100.00%
  nvme3n1: ios=137295/8808, merge=23/8683, ticks=811084/1485, in_queue=812569, util=55.37%
  nvme2n1: ios=781918/8808, merge=105/8681, ticks=5042680/2424, in_queue=5045103, util=77.49%
  nvme1n1: ios=953641/8814, merge=104/8682, ticks=6080949/3472, in_queue=6084421, util=85.15%
  nvme4n1: ios=134204/8798, merge=9/8682, ticks=856553/2409, in_queue=858961, util=54.78%

RAID 1 下多线程比单线程，大文件读性能提升明显。

4. RAID 5

4.1 创建 RAID 5

查看磁盘

`1`	`lsblk -d -o NAME,SIZE,TYPE \| grep nvme`

nvme0n1 745.2G disk
nvme1n1 745.2G disk
nvme2n1 745.2G disk
nvme3n1 745.2G disk
nvme4n1 745.2G disk
nvme5n1 745.2G disk

停掉 md0

1
2

umount /mnt/mdadm
mdadm --stop /dev/md0

创建 RAID

`1`	`mdadm --create --verbose /dev/md0 --level=5 --raid-devices=5 /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1`

查看 RAID 状态

`1`	`mdadm --detail /dev/md0`

等盘 State : clean 再操作。

格式化 RAID

挂载 RAID

1
2

mkdir -p /mnt/mdadm
mount /dev/md0 /mnt/mdadm

查看磁盘大小

1
2

Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        3.0T  3.1G  3.0T   1% /mnt/mdadm

4.2 4k 随机写

fio \
  --name=4k_randwrite \
  --rw=randwrite \
  --bs=4k \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=8 \
  --iodepth=32 \
  --size=20G \
  --time_based \
  --runtime=120 \
  --group_reporting \
  --allow_mounted_write=1 \
  --filename=/mnt/mdadm/fio-4k-randwrite.dat

Starting 8 processes
4k_randwrite: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [w(8)][100.0%][r=0KiB/s,w=128MiB/s][r=0,w=32.7k IOPS][eta 00m:00s]
4k_randwrite: (groupid=0, jobs=8): err= 0: pid=51252
  write: IOPS=29.4k, BW=115MiB/s (120MB/s)(13.4GiB/120011msec)
    slat (usec): min=4, max=9439.1k, avg=28.79, stdev=7258.31
    clat (usec): min=161, max=16550k, avg=8685.16, stdev=66489.70
     lat (usec): min=184, max=16550k, avg=8714.07, stdev=66884.47
    clat percentiles (msec):
     |  1.00th=[    5],  5.00th=[    6], 10.00th=[    6], 20.00th=[    7],
     | 30.00th=[    7], 40.00th=[    8], 50.00th=[    8], 60.00th=[    9],
     | 70.00th=[    9], 80.00th=[   10], 90.00th=[   11], 95.00th=[   12],
     | 99.00th=[   14], 99.50th=[   18], 99.90th=[   55], 99.95th=[   66],
     | 99.99th=[  584]
   bw (  KiB/s): min= 6236, max=27078, per=12.81%, avg=15042.38, stdev=2559.85, samples=1876
   iops        : min= 1559, max= 6769, avg=3760.36, stdev=639.97, samples=1876
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.02%, 4=0.22%, 10=85.41%, 20=13.89%, 50=0.30%
  lat (msec)   : 100=0.12%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2000=0.01%, >=2000=0.01%
  cpu          : usr=1.14%, sys=8.79%, ctx=2975367, majf=0, minf=5335
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,3523217,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=115MiB/s (120MB/s), 115MiB/s-115MiB/s (120MB/s-120MB/s), io=13.4GiB (14.4GB), run=120011-120011msec

Disk stats (read/write):
    md0: ios=1/3524290, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=1431863/1419187, aggrmerge=437534/125725, aggrticks=207352/66853, aggrin_queue=100468, aggrutil=31.43%
  nvme0n1: ios=1438642/1412948, merge=545900/19325, ticks=206960/65304, in_queue=99152, util=30.91%
  nvme3n1: ios=1436346/1410694, merge=545590/19081, ticks=208604/64388, in_queue=98924, util=31.31%
  nvme2n1: ios=1437739/1411528, merge=548042/19440, ticks=207920/65316, in_queue=99248, util=31.07%
  nvme1n1: ios=1438882/1412818, merge=548132/19412, ticks=208776/65200, in_queue=100252, util=31.43%
  nvme4n1: ios=1407707/1447950, merge=6/551371, ticks=204500/74060, in_queue=104764, util=30.68%

4.3 4k 随机读

fio \
  --name=4k_randread \
  --rw=randread \
  --bs=4k \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=8 \
  --iodepth=32 \
  --size=20G \
  --time_based \
  --runtime=120 \
  --group_reporting \
  --filename=/mnt/mdadm/fio-4k-randread.dat

Starting 8 processes
4k_randread: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [r(8)][39.7%][r=457MiB/s][r=117k IOPS][eta 01m:13s]
Jobs: 8 (f=8): [r(8)][100.0%][r=430MiB/s][r=110k IOPS][eta 00m:00s]
4k_randread: (groupid=0, jobs=8): err= 0: pid=217473
  read: IOPS=112k, BW=437MiB/s (458MB/s)(51.2GiB/120003msec)
    slat (usec): min=2, max=17923, avg= 9.85, stdev=18.68
    clat (usec): min=185, max=6460.9k, avg=2278.74, stdev=1852.82
     lat (usec): min=197, max=6460.9k, avg=2288.70, stdev=1852.94
    clat percentiles (usec):
     |  1.00th=[ 1778],  5.00th=[ 1909], 10.00th=[ 1975], 20.00th=[ 2057],
     | 30.00th=[ 2114], 40.00th=[ 2180], 50.00th=[ 2245], 60.00th=[ 2278],
     | 70.00th=[ 2343], 80.00th=[ 2442], 90.00th=[ 2573], 95.00th=[ 2704],
     | 99.00th=[ 3326], 99.50th=[ 3818], 99.90th=[ 6652], 99.95th=[12125],
     | 99.99th=[23987]
   bw (  KiB/s): min=392690, max=505688, per=100.00%, avg=447417.35, stdev=2872.96, samples=1912
   iops        : min=98172, max=126422, avg=111854.24, stdev=718.25, samples=1912
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.04%
  lat (msec)   : 2=12.52%, 4=87.02%, 10=0.34%, 20=0.04%, 50=0.02%
  lat (msec)   : >=2000=0.01%
  cpu          : usr=4.92%, sys=17.20%, ctx=8592874, majf=0, minf=19666
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=13411036,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=437MiB/s (458MB/s), 437MiB/s-437MiB/s (458MB/s-458MB/s), io=51.2GiB (54.9GB), run=120003-120003msec

Disk stats (read/write):
    md0: ios=13409323/13, merge=0/0, ticks=30340800/82756, in_queue=30423556, util=100.00%, aggrios=2705594/6089, aggrmerge=675495/168948, aggrticks=807940/2953, aggrin_queue=810893, aggrutil=100.00%
  nvme0n1: ios=2710474/77, merge=845726/0, ticks=781487/2, in_queue=781490, util=99.94%
  nvme3n1: ios=2712213/73, merge=842766/21, ticks=803642/3, in_queue=803645, util=99.94%
  nvme2n1: ios=2710710/73, merge=843898/21, ticks=845539/4, in_queue=845542, util=99.95%
  nvme1n1: ios=2715919/76, merge=845043/2, ticks=793915/5, in_queue=793921, util=99.94%
  nvme4n1: ios=2678658/30147, merge=42/844698, ticks=815117/14751, in_queue=829868, util=100.00%

4.4 4k 随机读写

fio \
  --name=4k_randrw \
  --rw=randrw \
  --rwmixread=70 \
  --bs=4k \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=8 \
  --iodepth=1 \
  --size=20G \
  --time_based \
  --runtime=120 \
  --group_reporting \
  --allow_mounted_write=1 \
  --filename=/mnt/mdadm/fio-4k-randrw.dat

Starting 8 processes
4k_randrw: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=87.4MiB/s,w=37.9MiB/s][r=22.4k,w=9693 IOPS][eta 00m:00s]
4k_randrw: (groupid=0, jobs=8): err= 0: pid=7507
   read: IOPS=17.1k, BW=66.9MiB/s (70.2MB/s)(8031MiB/120002msec)
    slat (usec): min=3, max=62397, avg=12.54, stdev=221.94
    clat (nsec): min=882, max=777393k, avg=251848.42, stdev=1742140.37
     lat (usec): min=10, max=777402, avg=264.55, stdev=1759.85
    clat percentiles (usec):
     |  1.00th=[  101],  5.00th=[  112], 10.00th=[  122], 20.00th=[  139],
     | 30.00th=[  157], 40.00th=[  178], 50.00th=[  196], 60.00th=[  219],
     | 70.00th=[  247], 80.00th=[  285], 90.00th=[  355], 95.00th=[  449],
     | 99.00th=[ 1045], 99.50th=[ 1680], 99.90th=[ 4686], 99.95th=[ 6718],
     | 99.99th=[21365]
   bw (  KiB/s): min= 2621, max=18076, per=12.62%, avg=8648.18, stdev=2379.10, samples=1904
   iops        : min=  655, max= 4519, avg=2161.83, stdev=594.79, samples=1904
  write: IOPS=7353, BW=28.7MiB/s (30.1MB/s)(3447MiB/120002msec)
    slat (usec): min=5, max=62333, avg=17.07, stdev=197.56
    clat (nsec): min=993, max=778193k, avg=448717.35, stdev=2296947.52
     lat (usec): min=37, max=778226, avg=465.97, stdev=2308.84
    clat percentiles (usec):
     |  1.00th=[  133],  5.00th=[  155], 10.00th=[  178], 20.00th=[  227],
     | 30.00th=[  265], 40.00th=[  306], 50.00th=[  347], 60.00th=[  388],
     | 70.00th=[  441], 80.00th=[  510], 90.00th=[  644], 95.00th=[  840],
     | 99.00th=[ 2180], 99.50th=[ 3523], 99.90th=[ 9241], 99.95th=[13435],
     | 99.99th=[39060]
   bw (  KiB/s): min= 1058, max= 7776, per=12.62%, avg=3711.53, stdev=1019.45, samples=1904
   iops        : min=  264, max= 1944, avg=927.59, stdev=254.87, samples=1904
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (usec)   : 100=0.57%, 250=56.65%, 500=33.73%, 750=5.85%, 1000=1.37%
  lat (msec)   : 2=1.19%, 4=0.40%, 10=0.17%, 20=0.03%, 50=0.01%
  lat (msec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  cpu          : usr=1.17%, sys=5.26%, ctx=3049332, majf=0, minf=1963
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=2056053,882433,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=66.9MiB/s (70.2MB/s), 66.9MiB/s-66.9MiB/s (70.2MB/s-70.2MB/s), io=8031MiB (8422MB), run=120002-120002msec
  WRITE: bw=28.7MiB/s (30.1MB/s), 28.7MiB/s-28.7MiB/s (30.1MB/s-30.1MB/s), io=3447MiB (3614MB), run=120002-120002msec

Disk stats (read/write):
    md0: ios=2054042/881576, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=1038923/407548, aggrmerge=4483248/1134718, aggrticks=137619/25332, aggrin_queue=42784, aggrutil=29.15%
  nvme0n1: ios=1107610/353099, merge=5604199/57, ticks=143444/11540, in_queue=37508, util=21.53%
  nvme3n1: ios=1107306/352408, merge=5603885/1, ticks=145064/11652, in_queue=37192, util=21.47%
  nvme2n1: ios=1107900/353371, merge=5604316/53, ticks=144164/11984, in_queue=37304, util=21.78%
  nvme1n1: ios=1107403/353443, merge=5603844/0, ticks=144516/11524, in_queue=36816, util=21.47%
  nvme4n1: ios=764398/625423, merge=0/5673479, ticks=110908/79960, in_queue=65104, util=29.15%

4.5 4M 顺序写

fio \
  --name=4m_seq_write \
  --rw=write \
  --bs=4M \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=1 \
  --iodepth=8 \
  --size=50G \
  --time_based \
  --runtime=60 \
  --group_reporting \
  --directory=/mnt/mdadm

Starting 1 process
4m_seq_write: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=905MiB/s][r=0,w=226 IOPS][eta 00m:00s]
4m_seq_write: (groupid=0, jobs=1): err= 0: pid=35663
  write: IOPS=233, BW=932MiB/s (978MB/s)(54.6GiB/60014msec)
    slat (usec): min=915, max=46600, avg=1949.51, stdev=657.29
    clat (msec): min=14, max=490, avg=32.34, stdev=13.20
     lat (msec): min=17, max=492, avg=34.29, stdev=13.21
    clat percentiles (msec):
     |  1.00th=[   22],  5.00th=[   24], 10.00th=[   26], 20.00th=[   28],
     | 30.00th=[   29], 40.00th=[   30], 50.00th=[   31], 60.00th=[   32],
     | 70.00th=[   34], 80.00th=[   36], 90.00th=[   40], 95.00th=[   44],
     | 99.00th=[   59], 99.50th=[   74], 99.90th=[  153], 99.95th=[  477],
     | 99.99th=[  489]
   bw (  KiB/s): min=90112, max=1223054, per=100.00%, avg=955317.09, stdev=107989.13, samples=120
   iops        : min=   22, max=  298, avg=233.02, stdev=26.34, samples=120
  lat (msec)   : 20=0.50%, 50=97.37%, 100=1.95%, 250=0.12%, 500=0.06%
  cpu          : usr=9.74%, sys=35.69%, ctx=8041, majf=0, minf=16447
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.9%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,13988,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=932MiB/s (978MB/s), 932MiB/s-932MiB/s (978MB/s-978MB/s), io=54.6GiB (58.7GB), run=60014-60014msec

Disk stats (read/write):
    md0: ios=0/446196, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=3056/113219, aggrmerge=90237/3484367, aggrticks=4905/76840, aggrin_queue=13444, aggrutil=12.88%
  nvme0n1: ios=3583/112577, merge=106272/3468500, ticks=6260/75552, in_queue=13344, util=12.63%
  nvme3n1: ios=3625/112582, merge=107246/3468845, ticks=5812/75836, in_queue=13468, util=12.73%
  nvme2n1: ios=3610/112597, merge=106945/3468757, ticks=5972/75664, in_queue=13340, util=12.70%
  nvme1n1: ios=3537/112593, merge=105151/3468752, ticks=6028/75708, in_queue=13396, util=12.78%
  nvme4n1: ios=925/115749, merge=25575/3546984, ticks=456/81440, in_queue=13676, util=12.88%

4.6 4M 顺序读

fio \
  --name=4m_seq_read \
  --rw=read \
  --bs=4M \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=1 \
  --iodepth=8 \
  --size=50G \
  --time_based \
  --runtime=60 \
  --group_reporting \
  --directory=/mnt/mdadm

Starting 1 process
4m_seq_read: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=0): [f(1)][100.0%][r=1224MiB/s][r=306 IOPS][eta 00m:00s]
4m_seq_read: (groupid=0, jobs=1): err= 0: pid=220920
  read: IOPS=317, BW=1271MiB/s (1333MB/s)(74.5GiB/60019msec)
    slat (usec): min=600, max=39534, avg=1207.89, stdev=363.00
    clat (usec): min=9972, max=57792, avg=23950.28, stdev=2393.21
     lat (usec): min=12659, max=61704, avg=25158.63, stdev=2424.67
    clat percentiles (usec):
     |  1.00th=[19268],  5.00th=[21103], 10.00th=[21890], 20.00th=[22414],
     | 30.00th=[22938], 40.00th=[23200], 50.00th=[23725], 60.00th=[23987],
     | 70.00th=[24511], 80.00th=[25035], 90.00th=[26346], 95.00th=[27657],
     | 99.00th=[32637], 99.50th=[34866], 99.90th=[43254], 99.95th=[48497],
     | 99.99th=[55837]
   bw (  MiB/s): min= 1176, max= 1524, per=100.00%, avg=1271.75, stdev=53.06, samples=119
   iops        : min=  294, max=  381, avg=317.92, stdev=13.25, samples=119
  lat (msec)   : 10=0.01%, 20=1.93%, 50=98.02%, 100=0.05%
  cpu          : usr=0.17%, sys=38.56%, ctx=19122, majf=0, minf=11636
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=19068,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=1271MiB/s (1333MB/s), 1271MiB/s-1271MiB/s (1333MB/s-1333MB/s), io=74.5GiB (80.0GB), run=60019-60019msec

Disk stats (read/write):
    md0: ios=597074/5, merge=0/0, ticks=13437000/92332, in_queue=13529332, util=100.00%, aggrios=124421/647, aggrmerge=3856308/18095, aggrticks=79087/835, aggrin_queue=79923, aggrutil=92.89%
  nvme0n1: ios=125015/30, merge=3874659/2, ticks=77880/5, in_queue=77885, util=92.55%
  nvme3n1: ios=125019/29, merge=3874638/0, ticks=79825/1, in_queue=79826, util=92.65%
  nvme2n1: ios=125014/30, merge=3874631/7, ticks=80032/2, in_queue=80034, util=92.37%
  nvme1n1: ios=125013/31, merge=3874507/2, ticks=78425/5, in_queue=78430, util=92.47%
  nvme4n1: ios=122045/3116, merge=3783108/90467, ticks=79277/4164, in_queue=83442, util=92.89%

5. RAID 6

5.1 创建 RAID 6

查看磁盘

`1`	`lsblk -d -o NAME,SIZE,TYPE \| grep nvme`

nvme0n1 745.2G disk
nvme1n1 745.2G disk
nvme2n1 745.2G disk
nvme3n1 745.2G disk
nvme4n1 745.2G disk
nvme5n1 745.2G disk

停掉 md0

1
2

umount /mnt/mdadm
mdadm --stop /dev/md0

创建 RAID

`1`	`mdadm --create --verbose /dev/md0 --level=6 --raid-devices=5 /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1`

查看 RAID 状态

`1`	`mdadm --detail /dev/md0`

等盘 State : clean 再操作。

格式化 RAID

挂载 RAID

1
2

mkdir -p /mnt/mdadm
mount /dev/md0 /mnt/mdadm

查看磁盘大小

1
2

Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        2.2T  2.3G  2.2T   1% /mnt/mdadm

5.2 4k 随机写

fio \
  --name=4k_randwrite \
  --rw=randwrite \
  --bs=4k \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=8 \
  --iodepth=32 \
  --size=20G \
  --time_based \
  --runtime=120 \
  --group_reporting \
  --allow_mounted_write=1 \
  --filename=/mnt/mdadm/fio-4k-randwrite.dat

Starting 8 processes
4k_randwrite: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [w(8)][100.0%][r=0KiB/s,w=84.7MiB/s][r=0,w=21.7k IOPS][eta 00m:00s]
4k_randwrite: (groupid=0, jobs=8): err= 0: pid=32858
  write: IOPS=23.9k, BW=93.4MiB/s (97.9MB/s)(10.9GiB/120007msec)
    slat (usec): min=4, max=86282, avg=24.05, stdev=502.49
    clat (usec): min=182, max=601133, avg=10678.54, stdev=8101.29
     lat (usec): min=193, max=601159, avg=10702.73, stdev=8117.41
    clat percentiles (msec):
     |  1.00th=[    8],  5.00th=[    9], 10.00th=[    9], 20.00th=[    9],
     | 30.00th=[   10], 40.00th=[   10], 50.00th=[   10], 60.00th=[   11],
     | 70.00th=[   12], 80.00th=[   12], 90.00th=[   14], 95.00th=[   15],
     | 99.00th=[   17], 99.50th=[   25], 99.90th=[   89], 99.95th=[  101],
     | 99.99th=[  338]
   bw (  KiB/s): min= 1539, max=14949, per=12.52%, avg=11970.40, stdev=1676.18, samples=1920
   iops        : min=  384, max= 3737, avg=2992.33, stdev=419.06, samples=1920
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.02%, 10=53.02%, 20=46.35%, 50=0.29%
  lat (msec)   : 100=0.26%, 250=0.02%, 500=0.02%, 750=0.01%
  cpu          : usr=1.02%, sys=7.14%, ctx=2528554, majf=0, minf=366
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,2868897,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=93.4MiB/s (97.9MB/s), 93.4MiB/s-93.4MiB/s (97.9MB/s-97.9MB/s), io=10.9GiB (11.8GB), run=120007-120007msec

Disk stats (read/write):
    md0: ios=1/2872252, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=1165819/1739264, aggrmerge=382175/177222, aggrticks=172486/78324, aggrin_queue=79014, aggrutil=28.17%
  nvme0n1: ios=1166238/1738908, merge=382185/176783, ticks=172300/78104, in_queue=78904, util=27.67%
  nvme3n1: ios=1165951/1738966, merge=382025/177845, ticks=170916/77944, in_queue=76912, util=27.36%
  nvme2n1: ios=1165506/1739719, merge=382121/177330, ticks=171780/78812, in_queue=78568, util=27.66%
  nvme1n1: ios=1165078/1740080, merge=382450/176873, ticks=172332/78628, in_queue=79588, util=27.62%
  nvme4n1: ios=1166326/1738647, merge=382095/177282, ticks=175104/78132, in_queue=81100, util=28.17%

5.3 4k 随机读

fio \
  --name=4k_randread \
  --rw=randread \
  --bs=4k \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=8 \
  --iodepth=32 \
  --size=20G \
  --time_based \
  --runtime=120 \
  --group_reporting \
  --filename=/mnt/mdadm/fio-4k-randread.dat

Starting 8 processes
4k_randread: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [r(8)][100.0%][r=3998MiB/s,w=0KiB/s][r=1023k,w=0 IOPS][eta 00m:00s]
4k_randread: (groupid=0, jobs=8): err= 0: pid=32831
   read: IOPS=999k, BW=3902MiB/s (4091MB/s)(457GiB/120001msec)
    slat (usec): min=2, max=22645, avg= 6.15, stdev=10.62
    clat (usec): min=22, max=29350, avg=249.28, stdev=233.95
     lat (usec): min=30, max=29360, avg=255.50, stdev=234.46
    clat percentiles (usec):
     |  1.00th=[   92],  5.00th=[  110], 10.00th=[  125], 20.00th=[  151],
     | 30.00th=[  172], 40.00th=[  192], 50.00th=[  210], 60.00th=[  233],
     | 70.00th=[  258], 80.00th=[  293], 90.00th=[  355], 95.00th=[  424],
     | 99.00th=[ 1401], 99.50th=[ 1942], 99.90th=[ 2704], 99.95th=[ 2933],
     | 99.99th=[ 4080]
   bw (  KiB/s): min=312313, max=605791, per=12.52%, avg=500364.02, stdev=36037.23, samples=1920
   iops        : min=78078, max=151447, avg=125090.71, stdev=9009.29, samples=1920
  lat (usec)   : 50=0.01%, 100=2.41%, 250=65.06%, 500=29.42%, 750=1.29%
  lat (usec)   : 1000=0.36%
  lat (msec)   : 2=1.00%, 4=0.44%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=21.24%, sys=76.36%, ctx=2153127, majf=0, minf=4745
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwt: total=119863453,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=3902MiB/s (4091MB/s), 3902MiB/s-3902MiB/s (4091MB/s-4091MB/s), io=457GiB (491GB), run=120001-120001msec

Disk stats (read/write):
    md0: ios=119748678/9, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=24049579/25441, aggrmerge=2245611/747233, aggrticks=3894218/4728, aggrin_queue=4242146, aggrutil=100.00%
  nvme0n1: ios=24051815/25428, merge=2245621/747293, ticks=3887344/4740, in_queue=4203696, util=100.00%
  nvme3n1: ios=24050100/25478, merge=2245619/747218, ticks=3891800/4728, in_queue=4184680, util=100.00%
  nvme2n1: ios=24047787/25409, merge=2245611/747216, ticks=3893712/4568, in_queue=4334888, util=100.00%
  nvme1n1: ios=24046232/25437, merge=2245619/747172, ticks=3884764/4656, in_queue=4179812, util=100.00%
  nvme4n1: ios=24051963/25454, merge=2245589/747267, ticks=3913472/4948, in_queue=4307656, util=100.00%

5.4 4k 随机读写

fio \
  --name=4k_randrw \
  --rw=randrw \
  --rwmixread=70 \
  --bs=4k \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=8 \
  --iodepth=1 \
  --size=20G \
  --time_based \
  --runtime=120 \
  --group_reporting \
  --allow_mounted_write=1 \
  --filename=/mnt/mdadm/fio-4k-randrw.dat

Starting 8 processes
4k_randrw: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=56.4MiB/s,w=24.1MiB/s][r=14.4k,w=6157 IOPS][eta 00m:00s]
4k_randrw: (groupid=0, jobs=8): err= 0: pid=14587
   read: IOPS=15.4k, BW=60.4MiB/s (63.3MB/s)(7242MiB/120002msec)
    slat (usec): min=2, max=6413, avg= 9.28, stdev=13.42
    clat (nsec): min=1423, max=11909k, avg=101092.62, stdev=45837.96
     lat (usec): min=17, max=11985, avg=110.52, stdev=48.00
    clat percentiles (usec):
     |  1.00th=[   86],  5.00th=[   87], 10.00th=[   88], 20.00th=[   90],
     | 30.00th=[   91], 40.00th=[   92], 50.00th=[   94], 60.00th=[  101],
     | 70.00th=[  105], 80.00th=[  108], 90.00th=[  114], 95.00th=[  130],
     | 99.00th=[  182], 99.50th=[  212], 99.90th=[  314], 99.95th=[  404],
     | 99.99th=[ 1565]
   bw (  KiB/s): min= 2528, max=10092, per=12.51%, avg=7733.86, stdev=938.41, samples=1918
   iops        : min=  632, max= 2523, avg=1933.25, stdev=234.62, samples=1918
  write: IOPS=6628, BW=25.9MiB/s (27.2MB/s)(3107MiB/120002msec)
    slat (usec): min=4, max=94806, avg=17.18, stdev=414.83
    clat (nsec): min=1210, max=317527k, avg=926268.22, stdev=2000717.09
     lat (usec): min=34, max=317579, avg=943.61, stdev=2046.66
    clat percentiles (usec):
     |  1.00th=[  131],  5.00th=[  157], 10.00th=[  198], 20.00th=[  375],
     | 30.00th=[  611], 40.00th=[  709], 50.00th=[  766], 60.00th=[  824],
     | 70.00th=[  938], 80.00th=[ 1139], 90.00th=[ 1549], 95.00th=[ 2147],
     | 99.00th=[ 4359], 99.50th=[ 5735], 99.90th=[11863], 99.95th=[17957],
     | 99.99th=[52167]
   bw (  KiB/s): min= 1138, max= 4080, per=12.51%, avg=3318.17, stdev=374.55, samples=1918
   iops        : min=  284, max= 1020, avg=829.25, stdev=93.58, samples=1918
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (usec)   : 100=41.58%, 250=32.59%, 500=3.14%, 750=6.69%, 1000=8.09%
  lat (msec)   : 2=6.14%, 4=1.38%, 10=0.33%, 20=0.03%, 50=0.01%
  lat (msec)   : 100=0.01%, 250=0.01%, 500=0.01%
  cpu          : usr=1.08%, sys=4.22%, ctx=2730677, majf=0, minf=2468
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=1854030,795491,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=60.4MiB/s (63.3MB/s), 60.4MiB/s-60.4MiB/s (63.3MB/s-63.3MB/s), io=7242MiB (7594MB), run=120002-120002msec
  WRITE: bw=25.9MiB/s (27.2MB/s), 25.9MiB/s-25.9MiB/s (27.2MB/s-27.2MB/s), io=3107MiB (3258MB), run=120002-120002msec

Disk stats (read/write):
    md0: ios=1851592/794454, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=847523/477463, aggrmerge=3545063/33, aggrticks=123240/28927, aggrin_queue=35743, aggrutil=28.22%
  nvme0n1: ios=847037/477760, merge=3544990/50, ticks=117308/29140, in_queue=30700, util=17.38%
  nvme3n1: ios=847090/477951, merge=3545181/0, ticks=116560/29000, in_queue=29552, util=17.15%
  nvme2n1: ios=848126/477235, merge=3545137/54, ticks=116288/29228, in_queue=29844, util=17.01%
  nvme1n1: ios=847872/477076, merge=3544783/56, ticks=116780/28848, in_queue=30100, util=17.19%
  nvme4n1: ios=847490/477293, merge=3545226/9, ticks=149268/28420, in_queue=58520, util=28.22%

5.5 4M 顺序写

fio \
  --name=4m_seq_write \
  --rw=write \
  --bs=4M \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=1 \
  --iodepth=8 \
  --size=50G \
  --time_based \
  --runtime=60 \
  --group_reporting \
  --directory=/mnt/mdadm

Starting 1 process
4m_seq_write: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=788MiB/s][r=0,w=197 IOPS][eta 00m:00s]
4m_seq_write: (groupid=0, jobs=1): err= 0: pid=22155
  write: IOPS=195, BW=780MiB/s (818MB/s)(45.7GiB/60015msec)
    slat (usec): min=953, max=110735, avg=2128.11, stdev=1116.33
    clat (msec): min=13, max=725, avg=38.82, stdev=20.13
     lat (msec): min=16, max=727, avg=40.95, stdev=20.17
    clat percentiles (msec):
     |  1.00th=[   26],  5.00th=[   29], 10.00th=[   31], 20.00th=[   33],
     | 30.00th=[   34], 40.00th=[   35], 50.00th=[   37], 60.00th=[   39],
     | 70.00th=[   41], 80.00th=[   43], 90.00th=[   49], 95.00th=[   54],
     | 99.00th=[   70], 99.50th=[   79], 99.90th=[  257], 99.95th=[  701],
     | 99.99th=[  718]
   bw (  KiB/s): min=410420, max=952176, per=100.00%, avg=805716.88, stdev=69019.53, samples=119
   iops        : min=  100, max=  232, avg=196.55, stdev=16.85, samples=119
  lat (msec)   : 20=0.01%, 50=92.04%, 100=7.81%, 250=0.01%, 500=0.06%
  lat (msec)   : 750=0.07%
  cpu          : usr=8.52%, sys=32.93%, ctx=7255, majf=0, minf=25476
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.9%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,11703,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=780MiB/s (818MB/s), 780MiB/s-780MiB/s (818MB/s-818MB/s), io=45.7GiB (49.1GB), run=60015-60015msec

Disk stats (read/write):
    md0: ios=0/373778, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=3294/125707, aggrmerge=98650/3869303, aggrticks=5868/77138, aggrin_queue=14148, aggrutil=13.74%
  nvme0n1: ios=3345/125706, merge=100391/3869234, ticks=5944/77612, in_queue=14396, util=13.48%
  nvme3n1: ios=3297/125705, merge=98759/3869335, ticks=5616/77656, in_queue=14568, util=13.54%
  nvme2n1: ios=3250/125713, merge=96926/3869428, ticks=5856/77088, in_queue=14212, util=13.28%
  nvme1n1: ios=3263/125708, merge=97543/3869307, ticks=5860/77048, in_queue=13956, util=13.12%
  nvme4n1: ios=3315/125703, merge=99634/3869214, ticks=6064/76288, in_queue=13608, util=13.74%

5.6 4M 顺序读

fio \
  --name=4m_seq_read \
  --rw=read \
  --bs=4M \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=1 \
  --iodepth=8 \
  --size=50G \
  --time_based \
  --runtime=60 \
  --group_reporting \
  --directory=/mnt/mdadm

Starting 1 process
4m_seq_read: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [R(1)][100.0%][r=11.4GiB/s,w=0KiB/s][r=2926,w=0 IOPS][eta 00m:00s]
4m_seq_read: (groupid=0, jobs=1): err= 0: pid=16590
   read: IOPS=2725, BW=10.6GiB/s (11.4GB/s)(639GiB/60001msec)
    slat (usec): min=187, max=4728, avg=363.72, stdev=111.08
    clat (usec): min=452, max=14674, avg=2569.21, stdev=572.70
     lat (usec): min=891, max=17172, avg=2933.25, stdev=643.23
    clat percentiles (usec):
     |  1.00th=[ 1860],  5.00th=[ 2024], 10.00th=[ 2114], 20.00th=[ 2180],
     | 30.00th=[ 2278], 40.00th=[ 2343], 50.00th=[ 2409], 60.00th=[ 2507],
     | 70.00th=[ 2606], 80.00th=[ 2802], 90.00th=[ 3294], 95.00th=[ 3785],
     | 99.00th=[ 4686], 99.50th=[ 5080], 99.90th=[ 6390], 99.95th=[ 6915],
     | 99.99th=[ 8848]
   bw (  MiB/s): min= 6608, max=13373, per=100.00%, avg=10956.27, stdev=1081.21, samples=120
   iops        : min= 1652, max= 3343, avg=2738.76, stdev=270.18, samples=120
  lat (usec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=3.70%, 4=92.78%, 10=3.50%, 20=0.01%
  cpu          : usr=0.89%, sys=98.26%, ctx=5411, majf=0, minf=16432
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=163549,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=10.6GiB/s (11.4GB/s), 10.6GiB/s-10.6GiB/s (11.4GB/s-11.4GB/s), io=639GiB (686GB), run=60001-60001msec

Disk stats (read/write):
    md0: ios=6380450/5, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=1850211/41, aggrmerge=1758800/7, aggrticks=762670/0, aggrin_queue=676558, aggrutil=99.62%
  nvme0n1: ios=1850105/42, merge=1758771/5, ticks=756424/0, in_queue=674672, util=99.25%
  nvme3n1: ios=1850214/41, merge=1758842/7, ticks=753412/0, in_queue=664152, util=99.18%
  nvme2n1: ios=1850318/42, merge=1758812/10, ticks=755052/0, in_queue=671332, util=99.26%
  nvme1n1: ios=1850167/42, merge=1758755/9, ticks=755708/0, in_queue=670380, util=99.36%
  nvme4n1: ios=1850255/42, merge=1758821/5, ticks=792756/0, in_queue=702256, util=99.62%

6. RAID 10

6.1 创建 RAID 10

查看磁盘

`1`	`lsblk -d -o NAME,SIZE,TYPE \| grep nvme`

nvme0n1 745.2G disk
nvme1n1 745.2G disk
nvme2n1 745.2G disk
nvme3n1 745.2G disk
nvme4n1 745.2G disk
nvme5n1 745.2G disk

停掉 md0

1
2

umount /mnt/mdadm
mdadm --stop /dev/md0

创建 RAID

`1`	`mdadm --create --verbose /dev/md0 --level=10 --raid-devices=5 /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1`

查看 RAID 状态

`1`	`mdadm --detail /dev/md0`

格式化 RAID

挂载 RAID

1
2

mkdir -p /mnt/mdadm
mount /dev/md0 /mnt/mdadm

查看磁盘大小

1
2

Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        1.9T   14G  1.9T   1% /mnt/mdadm

6.2 4k 随机写

fio \
  --name=4k_randwrite \
  --rw=randwrite \
  --bs=4k \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=8 \
  --iodepth=32 \
  --size=20G \
  --time_based \
  --runtime=120 \
  --group_reporting \
  --allow_mounted_write=1 \
  --filename=/mnt/mdadm/fio-4k-randwrite.dat

Starting 8 processes
4k_randwrite: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [w(8)][100.0%][w=366MiB/s][w=93.7k IOPS][eta 00m:00s]
4k_randwrite: (groupid=0, jobs=8): err= 0: pid=2022537
  write: IOPS=61.3k, BW=239MiB/s (251MB/s)(28.1GiB/120001msec); 0 zone resets
    slat (usec): min=5, max=137150, avg=119.06, stdev=261.57
    clat (usec): min=13, max=144186, avg=4050.42, stdev=2085.72
     lat (usec): min=30, max=144397, avg=4169.79, stdev=2134.10
    clat percentiles (usec):
     |  1.00th=[ 1500],  5.00th=[ 1876], 10.00th=[ 2180], 20.00th=[ 2606],
     | 30.00th=[ 2933], 40.00th=[ 3261], 50.00th=[ 3621], 60.00th=[ 4047],
     | 70.00th=[ 4621], 80.00th=[ 5342], 90.00th=[ 6456], 95.00th=[ 7308],
     | 99.00th=[10159], 99.50th=[12256], 99.90th=[17957], 99.95th=[21103],
     | 99.99th=[36963]
   bw (  KiB/s): min=133296, max=483224, per=99.81%, avg=244684.63, stdev=10054.77, samples=1912
   iops        : min=33324, max=120806, avg=61171.05, stdev=2513.69, samples=1912
  lat (usec)   : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%
  lat (usec)   : 750=0.03%, 1000=0.07%
  lat (msec)   : 2=6.83%, 4=52.22%, 10=39.77%, 20=0.99%, 50=0.06%
  lat (msec)   : 100=0.01%, 250=0.01%
  cpu          : usr=6.59%, sys=19.12%, ctx=6709292, majf=0, minf=17204
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,7354420,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=239MiB/s (251MB/s), 239MiB/s-239MiB/s (251MB/s-251MB/s), io=28.1GiB (30.1GB), run=120001-120001msec

Disk stats (read/write):
    md0: ios=0/7533721, merge=0/0, ticks=0/236656, in_queue=236656, util=99.21%, aggrios=0/3007482, aggrmerge=0/12339, aggrticks=0/56780, aggrin_queue=56780, aggrutil=99.12%
  nvme0n1: ios=0/3007171, merge=0/12024, ticks=0/55734, in_queue=55734, util=99.10%
  nvme3n1: ios=0/3008403, merge=0/12566, ticks=0/56594, in_queue=56594, util=99.10%
  nvme2n1: ios=0/3009445, merge=0/12502, ticks=0/57499, in_queue=57499, util=99.12%
  nvme1n1: ios=0/3007620, merge=0/12321, ticks=0/55409, in_queue=55409, util=99.08%
  nvme4n1: ios=0/3004772, merge=0/12283, ticks=0/58667, in_queue=58667, util=99.09%

6.3 4k 随机读

fio \
  --name=4k_randread \
  --rw=randread \
  --bs=4k \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=8 \
  --iodepth=32 \
  --size=20G \
  --time_based \
  --runtime=120 \
  --group_reporting \
  --filename=/mnt/mdadm/fio-4k-randread.dat

Starting 8 processes
4k_randread: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [r(8)][100.0%][r=3564MiB/s][r=912k IOPS][eta 00m:00s]
4k_randread: (groupid=0, jobs=8): err= 0: pid=2056465
  read: IOPS=920k, BW=3594MiB/s (3769MB/s)(421GiB/120001msec)
    slat (usec): min=2, max=4722, avg= 7.04, stdev= 3.35
    clat (usec): min=67, max=6190, avg=270.45, stdev=51.39
     lat (usec): min=85, max=6196, avg=277.57, stdev=51.90
    clat percentiles (usec):
     |  1.00th=[  196],  5.00th=[  212], 10.00th=[  223], 20.00th=[  233],
     | 30.00th=[  241], 40.00th=[  251], 50.00th=[  260], 60.00th=[  273],
     | 70.00th=[  285], 80.00th=[  302], 90.00th=[  334], 95.00th=[  367],
     | 99.00th=[  445], 99.50th=[  482], 99.90th=[  586], 99.95th=[  660],
     | 99.99th=[  898]
   bw (  MiB/s): min= 3095, max= 4105, per=100.00%, avg=3596.72, stdev=21.24, samples=1912
   iops        : min=792330, max=1050942, avg=920759.45, stdev=5438.34, samples=1912
  lat (usec)   : 100=0.01%, 250=39.60%, 500=60.04%, 750=0.33%, 1000=0.02%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%
  cpu          : usr=21.35%, sys=78.63%, ctx=4412, majf=0, minf=4539
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=110420784,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=3594MiB/s (3769MB/s), 3594MiB/s-3594MiB/s (3769MB/s-3769MB/s), io=421GiB (452GB), run=120001-120001msec

Disk stats (read/write):
    md0: ios=110375486/13, merge=0/0, ticks=12357644/0, in_queue=12357644, util=100.00%, aggrios=22084156/34, aggrmerge=0/0, aggrticks=2480023/1, aggrin_queue=2480024, aggrutil=100.00%
  nvme0n1: ios=22202499/40, merge=0/1, ticks=2478531/1, in_queue=2478532, util=100.00%
  nvme3n1: ios=22099756/32, merge=0/0, ticks=2480101/1, in_queue=2480101, util=100.00%
  nvme2n1: ios=22014556/32, merge=0/0, ticks=2481530/1, in_queue=2481531, util=100.00%
  nvme1n1: ios=22094397/37, merge=0/1, ticks=2480140/1, in_queue=2480141, util=100.00%
  nvme4n1: ios=22009576/33, merge=0/0, ticks=2479816/1, in_queue=2479816, util=100.00%

6.4 4k 随机读写

fio \
  --name=4k_randrw \
  --rw=randrw \
  --rwmixread=70 \
  --bs=4k \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=8 \
  --iodepth=1 \
  --size=20G \
  --time_based \
  --runtime=120 \
  --group_reporting \
  --allow_mounted_write=1 \
  --filename=/mnt/mdadm/fio-4k-randrw.dat

Starting 8 processes
4k_randrw: Laying out IO file (1 file / 20480MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=241MiB/s,w=102MiB/s][r=61.8k,w=26.1k IOPS][eta 00m:00s]
4k_randrw: (groupid=0, jobs=8): err= 0: pid=2075206
  read: IOPS=62.8k, BW=245MiB/s (257MB/s)(28.7GiB/120001msec)
    slat (usec): min=3, max=4059, avg= 9.88, stdev= 5.02
    clat (nsec): min=921, max=9798.4k, avg=98826.21, stdev=34501.61
     lat (usec): min=16, max=9826, avg=108.83, stdev=35.18
    clat percentiles (usec):
     |  1.00th=[   83],  5.00th=[   85], 10.00th=[   86], 20.00th=[   88],
     | 30.00th=[   90], 40.00th=[   91], 50.00th=[   93], 60.00th=[   97],
     | 70.00th=[  104], 80.00th=[  106], 90.00th=[  111], 95.00th=[  133],
     | 99.00th=[  167], 99.50th=[  180], 99.90th=[  247], 99.95th=[  277],
     | 99.99th=[  807]
   bw (  KiB/s): min=228448, max=280636, per=100.00%, avg=251275.94, stdev=1185.49, samples=1912
   iops        : min=57112, max=70158, avg=62818.94, stdev=296.37, samples=1912
  write: IOPS=26.9k, BW=105MiB/s (110MB/s)(12.3GiB/120001msec); 0 zone resets
    slat (usec): min=5, max=3753, avg=14.48, stdev= 6.78
    clat (nsec): min=757, max=23490k, avg=23058.93, stdev=76226.61
     lat (usec): min=16, max=23506, avg=37.67, stdev=76.97
    clat percentiles (usec):
     |  1.00th=[   13],  5.00th=[   14], 10.00th=[   15], 20.00th=[   16],
     | 30.00th=[   19], 40.00th=[   22], 50.00th=[   23], 60.00th=[   24],
     | 70.00th=[   24], 80.00th=[   26], 90.00th=[   28], 95.00th=[   31],
     | 99.00th=[   43], 99.50th=[   52], 99.90th=[  124], 99.95th=[  289],
     | 99.99th=[ 3720]
   bw (  KiB/s): min=95504, max=123296, per=100.00%, avg=107703.38, stdev=609.52, samples=1912
   iops        : min=23876, max=30823, avg=26925.78, stdev=152.38, samples=1912
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=10.43%, 50=19.39%
  lat (usec)   : 100=43.70%, 250=26.37%, 500=0.06%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=5.59%, sys=16.59%, ctx=10788462, majf=0, minf=4503
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=7532241,3228565,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=245MiB/s (257MB/s), 245MiB/s-245MiB/s (257MB/s-257MB/s), io=28.7GiB (30.9GB), run=120001-120001msec
  WRITE: bw=105MiB/s (110MB/s), 105MiB/s-105MiB/s (110MB/s-110MB/s), io=12.3GiB (13.2GB), run=120001-120001msec

Disk stats (read/write):
    md0: ios=7527276/3226395, merge=0/0, ticks=708904/60504, in_queue=769408, util=100.00%, aggrios=1506448/1291658, aggrmerge=0/0, aggrticks=146758/20850, aggrin_queue=167610, aggrutil=100.00%
  nvme0n1: ios=1519277/1290683, merge=0/1, ticks=147388/18530, in_queue=165919, util=100.00%
  nvme3n1: ios=1514805/1292302, merge=0/1, ticks=147915/19343, in_queue=167258, util=100.00%
  nvme2n1: ios=1496419/1292112, merge=0/0, ticks=146194/22563, in_queue=168757, util=100.00%
  nvme1n1: ios=1507568/1291756, merge=0/1, ticks=146524/20371, in_queue=166896, util=100.00%
  nvme4n1: ios=1494172/1291439, merge=0/1, ticks=145773/23447, in_queue=169220, util=100.00%

6.5 4M 顺序写

fio \
  --name=4m_seq_write \
  --rw=write \
  --bs=4M \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=1 \
  --iodepth=8 \
  --size=50G \
  --time_based \
  --runtime=60 \
  --group_reporting \
  --directory=/mnt/mdadm

Starting 1 process
4m_seq_write: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=2442MiB/s][w=610 IOPS][eta 00m:00s]
4m_seq_write: (groupid=0, jobs=1): err= 0: pid=2086304
  write: IOPS=613, BW=2452MiB/s (2571MB/s)(144GiB/60018msec); 0 zone resets
    slat (usec): min=542, max=22108, avg=1577.38, stdev=2473.07
    clat (usec): min=3230, max=32255, avg=11468.04, stdev=5453.40
     lat (usec): min=4116, max=33108, avg=13045.85, stdev=5470.07
    clat percentiles (usec):
     |  1.00th=[ 5342],  5.00th=[ 5866], 10.00th=[ 6194], 20.00th=[ 6587],
     | 30.00th=[ 6849], 40.00th=[ 7111], 50.00th=[ 7701], 60.00th=[14746],
     | 70.00th=[16450], 80.00th=[17433], 90.00th=[18220], 95.00th=[19530],
     | 99.00th=[23725], 99.50th=[26084], 99.90th=[28443], 99.95th=[28967],
     | 99.99th=[30278]
   bw (  MiB/s): min= 2352, max= 2560, per=100.00%, avg=2453.67, stdev=40.01, samples=119
   iops        : min=  588, max=  640, avg=613.41, stdev=10.01, samples=119
  lat (msec)   : 4=0.02%, 10=54.90%, 20=41.40%, 50=3.68%
  cpu          : usr=17.28%, sys=41.36%, ctx=15137, majf=0, minf=25
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,36794,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=2452MiB/s (2571MB/s), 2452MiB/s-2452MiB/s (2571MB/s-2571MB/s), io=144GiB (154GB), run=60018-60018msec

Disk stats (read/write):
    md0: ios=0/1197325, merge=0/0, ticks=0/5861456, in_queue=5861456, util=100.00%, aggrios=0/482470, aggrmerge=0/7, aggrticks=0/1511007, aggrin_queue=1511007, aggrutil=97.70%
  nvme0n1: ios=0/482466, merge=0/7, ticks=0/1820645, in_queue=1820645, util=97.70%
  nvme3n1: ios=0/482471, merge=0/9, ticks=0/1352720, in_queue=1352719, util=96.34%
  nvme2n1: ios=0/482472, merge=0/8, ticks=0/1424437, in_queue=1424438, util=96.11%
  nvme1n1: ios=0/482473, merge=0/7, ticks=0/1480971, in_queue=1480971, util=96.62%
  nvme4n1: ios=0/482470, merge=0/7, ticks=0/1476262, in_queue=1476263, util=96.65%

6.6 4M 顺序读

fio \
  --name=4m_seq_read \
  --rw=read \
  --bs=4M \
  --ioengine=libaio \
  --direct=1 \
  --numjobs=1 \
  --iodepth=8 \
  --size=50G \
  --time_based \
  --runtime=60 \
  --group_reporting \
  --directory=/mnt/mdadm

Starting 1 process
4m_seq_read: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=0): [f(1)][100.0%][r=9768MiB/s][r=2442 IOPS][eta 00m:00s]
4m_seq_read: (groupid=0, jobs=1): err= 0: pid=2096984
  read: IOPS=2607, BW=10.2GiB/s (10.9GB/s)(611GiB/60001msec)
    slat (usec): min=268, max=2362, avg=381.17, stdev=50.26
    clat (usec): min=416, max=14426, avg=2685.17, stdev=266.94
     lat (usec): min=810, max=16790, avg=3066.55, stdev=300.73
    clat percentiles (usec):
     |  1.00th=[ 2245],  5.00th=[ 2376], 10.00th=[ 2507], 20.00th=[ 2540],
     | 30.00th=[ 2606], 40.00th=[ 2638], 50.00th=[ 2671], 60.00th=[ 2704],
     | 70.00th=[ 2737], 80.00th=[ 2769], 90.00th=[ 2835], 95.00th=[ 2966],
     | 99.00th=[ 3982], 99.50th=[ 4228], 99.90th=[ 4752], 99.95th=[ 4948],
     | 99.99th=[ 5407]
   bw (  MiB/s): min= 9520, max=12216, per=100.00%, avg=10443.82, stdev=504.01, samples=119
   iops        : min= 2380, max= 3054, avg=2610.84, stdev=125.95, samples=119
  lat (usec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.09%, 4=98.94%, 10=0.96%, 20=0.01%
  cpu          : usr=0.72%, sys=99.27%, ctx=175, majf=0, minf=8214
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=156465,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=10.2GiB/s (10.9GB/s), 10.2GiB/s-10.2GiB/s (10.9GB/s-10.9GB/s), io=611GiB (656GB), run=60001-60001msec

Disk stats (read/write):
    md0: ios=5436104/2, merge=0/0, ticks=2081068/0, in_queue=2081068, util=100.00%, aggrios=1085227/15, aggrmerge=6123/0, aggrticks=411348/0, aggrin_queue=411349, aggrutil=99.90%
  nvme0n1: ios=1086197/15, merge=6120/0, ticks=410605/1, in_queue=410605, util=99.90%
  nvme3n1: ios=1085860/16, merge=6068/0, ticks=411802/0, in_queue=411803, util=99.90%
  nvme2n1: ios=1080697/17, merge=6171/0, ticks=412160/0, in_queue=412162, util=99.90%
  nvme1n1: ios=1089511/16, merge=6136/0, ticks=410432/1, in_queue=410432, util=99.90%
  nvme4n1: ios=1083870/15, merge=6120/0, ticks=411745/1, in_queue=411745, util=99.90%

7. 对比

测试结果（Latency 为 fio 报告之 lat avg，CPU 为 usr+sys）

7.1 4k - 随机写

系统	进程数	IOPS	BW	Latency	CPU
单盘	8	65.1k	254MiB/s	3932.32us	36.86%
RAID 0	8	87.9k	343MiB/s	2911.98us	50.33%
RAID 1	8	82.4k	322MiB/s	3102.50us	30.18%
RAID 5	8	29.4k	115MiB/s	8714.07us	9.93%
RAID 6	8	23.9k	93.4MiB/s	10702.73us	8.16%
RAID 10	8	61.3k	239MiB/s	4169.79us	25.71%

RAID 0 最高（87.9k IOPS / 343MiB/s），条带写多盘并行。
RAID 5/6 最低（约 23～29k），校验写带来读-改-写与校验计算开销。
RAID 1、RAID 10 介于单盘与 RAID 0 之间，镜像/镜像条带写有一定并行度。

7.2 4k - 随机读

系统	进程数	IOPS	BW	Latency	CPU
单盘	8	294k	1147MiB/s	870.84us	39.54%
RAID 0	8	1009k	3940MiB/s	253.20us	99.44%
RAID 1	8	734k	2869MiB/s	347.77us	90.69%
RAID 5	8	112k	437MiB/s	2288.70us	22.12%
RAID 6	8	999k	3902MiB/s	255.50us	97.60%
RAID 10	8	920k	3594MiB/s	277.57us	99.98%

RAID 0、RAID 6、RAID 10 接近且最高（约 920k～1009k IOPS），条带/双校验/镜像条带读均可多盘并发。
RAID 1 次之（734k），多副本可分流读。
RAID 5 明显较低（112k），单校验条带下随机读易受校验块与布局影响。

7.3 4k - 随机读写

系统	进程数	IOPS	BW	Latency	CPU
单盘	8	64.7k	256MiB/s	162.52us	10.70%
RAID 0	8	89.5k	350MiB/s	109.26us	19.52%
RAID 1	8	82.3k	321MiB/s	114.84us	18.94%
RAID 5	8	24.5k	95.6MiB/s	264.55us	6.43%
RAID 6	8	22.0k	86.3MiB/s	110.52us	5.30%
RAID 10	8	89.7k	350MiB/s	108.83us	22.18%

RAID 0、RAID 1、RAID 10 接近且较高（约 82～90k IOPS / 321～350MiB/s），读多写少时条带/镜像表现好。
RAID 5、RAID 6 较低（约 22～25k / 86～96MiB/s），混合负载下校验与写放大仍明显拉低性能。

7.4 4M - 顺序写

系统	进程数	IOPS	BW	Latency	CPU
单盘	1	237	950MiB/s	33680us	18.16%
RAID 0	1	1203	4815MiB/s	6643.65us	80.79%
RAID 1	1	242	969MiB/s	33005.52us	49.18%
RAID 5	1	233	932MiB/s	34290us	45.43%
RAID 6	1	195	780MiB/s	40950us	41.45%
RAID 10	1	613	2452MiB/s	13045.85us	58.64%

RAID 0 最高（1203 IOPS / 4815MiB/s），多盘顺序写条带化效果明显。
RAID 5、RAID 6 较低（195～233 IOPS / 780～932MiB/s），校验与重建参与顺序写。
RAID 1 接近单盘（242 / 969MiB/s），多副本写带宽受单副本限制。
RAID 10 约为 RAID 0 的一半（613 / 2452MiB/s），镜像条带写需写多份。

7.5 4M - 顺序读

系统	进程数	IOPS	BW	Latency	CPU
单盘	1	722	2890MiB/s	11065.58us	29.35%
RAID 0	1	3385	13.2GiB/s	2361.44us	97.85%
RAID 1	1	762	3049MiB/s	10491.80us	41.95%
RAID 5	1	317	1271MiB/s	25159us	38.73%
RAID 6	1	2725	10.6GiB/s	2933.25us	99.15%
RAID 10	1	2607	10.2GiB/s	3066.55us	99.99%

RAID 0、RAID 6、RAID 10 接近且最高（约 2607～3385 IOPS / 10.2～13.2GiB/s），大块顺序读多盘并发优势明显。
RAID 1 次之（762 IOPS / 3049MiB/s），多副本可分流但受单条带带宽影响。
RAID 5 明显较低（317 IOPS / 1271MiB/s），单校验条带下顺序读带宽仍受限。

8. 结论

5 盘 RAID 针对单盘的性能放大对比如下：（实测 / 理论）

RAID Level	4k 小文件随机读	4k 小文件随机写	4M 大文件顺序读	4M 大文件顺序写
RAID 0	3.4× / 4×	1.35× / 4×	4.7× / 4×	5.1× / 4×
RAID 1	2.5× / 4×	1.27× / 1×	1.1× / 4×	1.0× / 1×
RAID 5	0.38× / 3×	0.45× / 0.75×	0.44× / 3×	1.0× / 3×
RAID 6	3.4× / 2×	0.37× / 0.5×	3.8× / 2×	0.82× / 2×
RAID 10	3.1× / 4×	0.94× / 2×	3.6× / 4×	2.6× / 2×

其中，RAID 5 的数据比较差，但反复测试依然是这个结果。

此外还有 RAID 50、RAID 60，采用 RAIDx + RAID 0 的组合方式，具体选型还要结合盘的数量，如果 disk num 在 4 - 16 之间，可以采用 RAID 6、RAID 10；如果 disk num 不少于 16，可以采用 RAID 60。RAID 6、RAID 10 采用硬阵列会有明显的性能提升。

推荐订阅源

陈少文的网站

1. 磁盘基准测试

1.1 4k 随机写

1.2 4k 随机读

1.3 4k 随机读写

1.4 4M 顺序写

1.5 4M 顺序读

2. RAID 0

2.1 创建 RAID 0

2.2 4k 随机写

2.3 4k 随机读

2.4 4k 随机读写

2.5 4M 顺序写

2.6 4M 顺序读

3. RAID 1

3.1 创建 RAID 1

3.2 4k 随机写

3.3 4k 随机读

3.4 4k 随机读写

3.5 4M 顺序写

3.6 4M 顺序读

3.7 4M 多进程读

4. RAID 5

4.1 创建 RAID 5

4.2 4k 随机写

4.3 4k 随机读

4.4 4k 随机读写

4.5 4M 顺序写

4.6 4M 顺序读

5. RAID 6

5.1 创建 RAID 6

5.2 4k 随机写

5.3 4k 随机读

5.4 4k 随机读写

5.5 4M 顺序写

5.6 4M 顺序读

6. RAID 10

6.1 创建 RAID 10

6.2 4k 随机写

6.3 4k 随机读

6.4 4k 随机读写

6.5 4M 顺序写

6.6 4M 顺序读

7. 对比

7.1 4k - 随机写

7.2 4k - 随机读

7.3 4k - 随机读写

7.4 4M - 顺序写

7.5 4M - 顺序读

8. 结论