xfs 文件系统不支持缩容，本篇主要是对 RAID 10 下换盘、扩容进行操作。
1. RAID 状态与盘状态
在 mdadm 中，RAID 状态：
- clean, 正常
- degraded，降级，依然可以读写，但是容错能力降低
- reshaping，重组中，扩容、缩容时会出现
- resyncing，同步数据中
- recovering，恢复数据中，伴随降级，盘重建
- failed，故障
在 mdadm 中，盘状态有：
- active, 参与数据读写
- spare, 热备盘，当有盘故障时，自动接替故障盘
- failed，故障盘
- sync，同步中，正常状态
- rebuilding，重建中
- set-x，相同标记下的全部盘是一组镜像
2. RAID 0
2.1 创建 RAID 0
1
2
| umount /mnt/mdadmdm
mdadm --stop /dev/md0
|
1
| mdadm --create --verbose /dev/md0 --level=0 --raid-devices=4 /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1
|
1
| mdadm --detail /dev/md0
|
1
2
3
4
5
| State : clean
0 259 0 0 active sync /dev/nvme0n1
1 259 1 1 active sync /dev/nvme1n1
2 259 4 2 active sync /dev/nvme2n1
3 259 3 3 active sync /dev/nvme3n1
|
1
2
| mkdir -p /mnt/mdadm
mount /dev/md0 /mnt/mdadm
|
1
2
| Filesystem Size Used Avail Use% Mounted on
/dev/md0 3.0T 21G 2.9T 1% /mnt/mdadm
|
3. RAID 1
3.1 创建 RAID 1
1
2
| umount /mnt/mdadm
mdadm --stop /dev/md0
|
1
| mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 /dev/nvme0n1 /dev/nvme1n1
|
1
2
| mkdir -p /mnt/mdadm
mount /dev/md0 /mnt/mdadm
|
1
2
| Filesystem Size Used Avail Use% Mounted on
/dev/md0 745G 5.3G 740G 1% /mnt/mdadm
|
3.2 单盘替换
1
| mdadm /dev/md0 --add /dev/nvme2n1
|
1
| mdadm /dev/md0 --fail /dev/nvme1n1
|
1
| mdadm /dev/md0 --remove /dev/nvme1n1
|
1
| mdadm --detail /dev/md0
|
1
2
3
4
| State : clean, degraded, recovering
Number Major Minor RaidDevice State
0 259 0 0 active sync /dev/nvme0n1
2 259 4 1 spare rebuilding /dev/nvme2n1
|
3.3 加盘扩容
1
| mdadm /dev/md0 --add /dev/nvme3n1
|
新加的盘默认为热备盘，需要将热备盘标记为数据盘。
1
| mdadm /dev/md0 --grow --raid-devices=3
|
1
| mdadm --detail /dev/md0
|
1
2
3
4
5
| State : clean, degraded, recovering
Number Major Minor RaidDevice State
0 259 0 0 active sync /dev/nvme0n1
2 259 4 1 spare rebuilding /dev/nvme2n1
3 259 3 2 spare rebuilding /dev/nvme3n1
|
因为 RAID 1 每个盘数据都一样，也就没有平衡或者扩容的操作了。
4. RAID 10
4.1 创建 RAID 10
1
2
| umount /mnt/mdadm
mdadm --stop /dev/md0
|
1
| mdadm --create --verbose /dev/md0 --level=10 --raid-devices=4 /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1
|
1
| mdadm --detail /dev/md0
|
1
2
3
4
5
| Number Major Minor RaidDevice State
0 259 0 0 active sync set-A /dev/nvme0n1
1 259 1 1 active sync set-B /dev/nvme1n1
2 259 4 2 active sync set-A /dev/nvme2n1
3 259 3 3 active sync set-B /dev/nvme3n1
|
1
2
| mkdir -p /mnt/mdadm
mount /dev/md0 /mnt/mdadm
|
1
2
| Filesystem Size Used Avail Use% Mounted on
/dev/md0 1.5T 11G 1.5T 1% /mnt/mdadm
|
1
2
| Filesystem Size Used Avail Use% Mounted on
/dev/md0 1.5T 61G 1.4T 5% /mnt/mdadm
|
写了一个 50G 的文件 4m_seq_write.0.0 。
4.2 单盘替换
热备盘可以用于故障自动替换或者替换正常的数据盘。
1
| mdadm /dev/md0 --add /dev/nvme4n1 /dev/nvme5n1
|
1
| mdadm /dev/md0 --remove /dev/nvme5n1
|
1
| mdadm --detail /dev/md0
|
1
2
3
4
5
6
7
| Number Major Minor RaidDevice State
0 259 0 0 active sync set-A /dev/nvme0n1
1 259 1 1 active sync set-B /dev/nvme1n1
2 259 4 2 active sync set-A /dev/nvme2n1
3 259 3 3 active sync set-B /dev/nvme3n1
4 259 2 - spare /dev/nvme4n1
|
1
| mdadm /dev/md0 --fail /dev/nvme3n1
|
1
| mdadm --detail /dev/md0
|
1
2
3
4
5
6
7
8
| State : clean, degraded, recovering
Number Major Minor RaidDevice State
0 259 0 0 active sync set-A /dev/nvme0n1
1 259 1 1 active sync set-B /dev/nvme1n1
2 259 4 2 active sync set-A /dev/nvme2n1
4 259 2 3 spare rebuilding /dev/nvme4n1
3 259 3 - faulty /dev/nvme3n1
|
此时阵列处于降级状态，正在重建数据。
1
| mdadm /dev/md0 --remove /dev/nvme3n1
|
4.3 加盘扩容
1
| mdadm /dev/md0 --add /dev/nvme3n1 /dev/nvme5n1
|
1
| mdadm --detail /dev/md0
|
1
2
3
4
5
6
7
8
9
| State : clean
Number Major Minor RaidDevice State
0 259 0 0 active sync set-A /dev/nvme0n1
1 259 1 1 active sync set-B /dev/nvme1n1
2 259 4 2 active sync set-A /dev/nvme2n1
4 259 2 3 active sync set-B /dev/nvme4n1
3 259 3 - spare /dev/nvme3n1
5 259 5 - spare /dev/nvme5n1
|
新加入的盘默认为热备盘。
1
| mdadm /dev/md0 --grow --raid-devices=6
|
1
| mdadm --detail /dev/md0
|
1
2
3
4
5
6
7
8
| State : clean, reshaping
Number Major Minor RaidDevice State
0 259 0 0 active sync set-A /dev/nvme0n1
1 259 1 1 active sync set-B /dev/nvme1n1
2 259 4 2 active sync set-A /dev/nvme2n1
4 259 2 3 active sync set-B /dev/nvme4n1
5 259 5 4 active sync set-A /dev/nvme5n1
3 259 3 5 active sync set-B /dev/nvme3n1
|
等等 reshaping 状态结束，cat /proc/mdstat 可以看到需要多久时间完成。
1
2
| Filesystem Size Used Avail Use% Mounted on
/dev/md0 2.2T 66G 2.2T 3% /mnt/mdadm
|
文件系统扩容完成。
5. RAID 6
5.1 创建 RAID 6
1
2
| umount /mnt/mdadm
mdadm --stop /dev/md0
|
1
| mdadm --create --verbose /dev/md0 --level=6 --raid-devices=4 /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1
|
1
2
| mkdir -p /mnt/mdadm
mount /dev/md0 /mnt/mdadm
|
1
2
| Filesystem Size Used Avail Use% Mounted on
/dev/md0 1.5T 11G 1.5T 1% /mnt/mdadm
|
5.2 单盘替换
1
| mdadm /dev/md0 --add /dev/nvme4n1
|
1
| mdadm /dev/md0 --fail /dev/nvme3n1
|
1
| mdadm /dev/md0 --remove /dev/nvme3n1
|
1
| mdadm --detail /dev/md0
|
1
2
3
4
5
6
| State : clean, degraded, recovering
Number Major Minor RaidDevice State
0 259 0 0 active sync /dev/nvme0n1
1 259 1 1 active sync /dev/nvme1n1
2 259 4 2 active sync /dev/nvme2n1
4 259 2 3 spare rebuilding /dev/nvme4n1
|
5.3 加盘扩容
1
| mdadm /dev/md0 --add /dev/nvme5n1
|
1
| mdadm /dev/md0 --grow --raid-devices=5
|
1
| mdadm --detail /dev/md0
|
1
2
3
4
5
6
7
| State : clean, degraded, reshaping
Number Major Minor RaidDevice State
0 259 0 0 active sync /dev/nvme0n1
1 259 1 1 active sync /dev/nvme1n1
2 259 4 2 active sync /dev/nvme2n1
4 259 2 3 spare rebuilding /dev/nvme4n1
5 259 5 4 active sync /dev/nvme5n1
|
不会自动平衡数据，而是新的数据会逐步写入新盘。
1
2
| Filesystem Size Used Avail Use% Mounted on
/dev/md0 2.2T 16G 2.2T 1% /mnt/mdadm
|
6. 生成测试数据
1
2
3
4
5
6
7
8
9
10
11
12
13
| fio \
--name=4m_seq_write \
--rw=write \
--bs=4M \
--ioengine=libaio \
--direct=1 \
--numjobs=1 \
--iodepth=8 \
--size=50G \
--time_based \
--runtime=60 \
--group_reporting \
--directory=/mnt/mdadm
|
用于生成测试数据。
1
| md5sum /mnt/mdadm/4m_seq_write.0.0
|
用于校验调整之后文件的完整性。
