1. 环境准备
1.1 数据盘准备
1
| lsblk -d -o NAME,SIZE,TYPE | grep nvme
|
1
2
3
4
| nvme0n1 745.2G disk
nvme1n1 745.2G disk
nvme2n1 745.2G disk
nvme3n1 745.2G disk
|
1
2
3
| for i in {0..3}; do
mkdir -p /mnt/data${i}
done
|
1
2
3
| for i in {0..3}; do
mkfs.xfs -f /dev/nvme${i}n1
done
|
1
2
3
| for i in {0..3}; do
mount /dev/nvme${i}n1 /mnt/data${i}
done
|
1
2
3
4
| for i in {0..3}; do
rm -rf /mnt/data${i}/*
rm -rf /mnt/data${i}/.minio.sys
done
|
1
2
3
4
| /dev/nvme0n1 745G 5.3G 740G 1% /mnt/data0
/dev/nvme1n1 745G 5.3G 740G 1% /mnt/data1
/dev/nvme2n1 745G 5.3G 740G 1% /mnt/data2
/dev/nvme3n1 745G 5.3G 740G 1% /mnt/data3
|
1.2 时间同步
1
2
| apt install -y chrony
systemctl enable --now chrony
|
2. 部署 MinIO
每个节点有 4 块盘，部署两台机器组成一个 minio 集群。
2.1 配置 hosts
1
2
3
4
| cat >> /etc/hosts <<EOF
10.10.10.11 minio1
10.10.10.12 minio2
EOF
|
通过配置 hosts 可以固定节点名，便于运维，能够避免扩容时识别问题。
2.1 环境变量
1
2
3
4
5
6
7
8
9
10
| export CONTAINER_CLI=nerdctl
export IMAGE=minio/minio:RELEASE.2025-04-22T22-12-26Z
export ROOT_USER=minioadmin
export ROOT_PASSWORD=minioadmin
export MINIO_ERASURE_SET_DRIVE_COUNT=4
export MINIO_STORAGE_CLASS_STANDARD=EC:2
export POOL_0="http://minio{1...2}/mnt/data{0...3}"
|
RELEASE.2025-04-22T22-12-26Z 之后的 minio 版本 Web 端极其简陋，只能创建桶，不能配置策略、秘钥等，复杂配置需要使用命令行。
MINIO_ERASURE_SET_DRIVE_COUNT 是纠删组的大小，不能超过 16。
MINIO_STORAGE_CLASS_STANDARD 是纠删码的冗余度，不能超过纠删组大小/2。
一次部署就一个 pool，放在一行提供给 minio 启动。
2.2 启动 minio
每个节点的启动命令是相同的，这里先启动两个节点。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
| $CONTAINER_CLI run -d \
--net host \
--ulimit memlock=-1 \
--ulimit stack=67108864 \
--ulimit nofile=1048576:1048576 \
--memory-swappiness=0 \
--name minio \
-v /mnt/data0:/mnt/data0 \
-v /mnt/data1:/mnt/data1 \
-v /mnt/data2:/mnt/data2 \
-v /mnt/data3:/mnt/data3 \
-e "MINIO_ROOT_USER=$ROOT_USER" \
-e "MINIO_ROOT_PASSWORD=$ROOT_PASSWORD" \
-e "MINIO_ERASURE_SET_DRIVE_COUNT=$MINIO_ERASURE_SET_DRIVE_COUNT" \
-e "MINIO_STORAGE_CLASS_STANDARD=$MINIO_STORAGE_CLASS_STANDARD" \
$IMAGE server \
$POOL_0 \
--console-address ":9090" \
--address ":9000"
|
9090 是 MinIO 的控制台端口，9000 是 MinIO 的 API 端口，访问数据时使用 9000 端口。
一个节点启动之后，会等待其他节点加入集群。
3. 集群管理
3.1 配置凭证
1
2
3
| wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc
mv mc /usr/bin/
|
1
| mc alias set local http://127.0.0.1:9000 minioadmin minioadmin
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
| ● minio1:9000
Uptime: 46 seconds
Version: 2025-04-22T22:12:26Z
Network: 2/2 OK
Drives: 4/4 OK
Pool: 1
● minio2:9000
Uptime: 42 seconds
Version: 2025-04-22T22:12:26Z
Network: 2/2 OK
Drives: 4/4 OK
Pool: 1
┌──────┬───────────────────────┬─────────────────────┬──────────────┐
│ Pool │ Drives Usage │ Erasure stripe size │ Erasure sets │
│ 1st │ 0.7% (total: 2.9 TiB) │ 4 │ 2 │
└──────┴───────────────────────┴─────────────────────┴──────────────┘
8 drives online, 0 drives offline, EC:2
|
Erasure stripe size 表示每个纠删组的盘数，Erasure sets 表示纠删组的数量。这里设置的是 4 盘纠删组，2 个纠删组，EC:2 两个校验盘，得盘率为 50%。
3.2 桶的管理
3.3 用户与策略
1
| mc admin user add local myuser mypassword
|
1
| mc admin user info local myuser
|
1
| mc admin policy attach local readwrite --user myuser
|
内置策略对全部桶生效，可选值：readwrite、writeonly、consoleAdmin、diagnostics、readonly。
1
| mc admin policy detach local readwrite --user myuser
|
1
| mc admin policy rm local mypolicy
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
| cat > test-readwrite.json <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetBucketLocation",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::test"
]
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::test/*"
]
}
]
}
EOF
|
1
| mc admin policy create local test-readwrite test-readwrite.json
|
1
| mc admin policy attach local test-readwrite --user myuser
|
3.4 文件的管理
1
| mc cp ./test.txt local/test/test.txt
|
1
| mc cp -r ./ local/test/
|
1
| mc stat local/test/test.txt
|
3.5 rebalance
minio rebalance 仅对多节点集群有效。
1
| mc admin rebalance start local
|
1
| mc admin rebalance status local
|
1
| mc admin rebalance stop local
|
3.6 桶的版本控制
1
| mc version enable local/test2
|
1
| mc version info local/test2
|
1
| mc cp -r ./ local/test2/
|
1
| mc ls --versions local/test2/
|
1
2
| 5B STANDARD aa673997-a590-4b04-b3be-80627fa3352d v2 PUT bbbb
5B STANDARD f5bcc176-3544-432f-924c-eb654a317ca0 v1 PUT bbbb
|
1
| mc cp --version-id f5bcc176-3544-432f-924c-eb654a317ca0 local/test2/bbbb ./bbbb-v1
|
3.6 统计详情
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
| mc stat local/test2
Name : test2
Size : N/A
Type : folder
Properties:
Versioning: Enabled
Location: us-east-1
Anonymous: Disabled
ILM: Disabled
Usage:
Total size: 1002 MiB
Objects count: 1,652
Versions count: 2,967
Object sizes histogram:
509 object(s) BETWEEN_1024B_AND_1_MB
423 object(s) BETWEEN_1024_B_AND_64_KB
9 object(s) BETWEEN_10_MB_AND_64_MB
|
1
| mc stat local/test2/bbbb
|
1
2
3
4
5
6
7
| Name : bbbb
Size : 5 B
ETag : 9448a1bf333fadd2a57965ec38487b89
VersionID : aa673997-a590-4b04-b3be-80627fa3352d
Type : file
Metadata :
Content-Type: application/octet-stream
|
format.json 是 minio 的格式化文件，记录了集群的配置信息。建议每次调整集群拓扑之后，进行一次备份。
1
2
3
4
5
| mkdir -p /backup/miniosys/
for i in {0..3}; do
mkdir -p /backup/miniosys/data${i}
cp /mnt/data${i}/.minio.sys/format.json /backup/miniosys/data${i}/format.json
done
|
查看备份
1
| ls -l /backup/miniosys/
|
1
2
3
4
| drwxr-xr-x 2 root root 4096 Feb 6 19:40 data0
drwxr-xr-x 2 root root 4096 Feb 6 19:40 data1
drwxr-xr-x 2 root root 4096 Feb 6 19:40 data2
drwxr-xr-x 2 root root 4096 Feb 6 19:40 data3
|
4 备份数据
4.1 mc 全量备份
1
| mc cp -r ./ local/backup/
|
mc 每次都会全量上传一次，而不是增量上传。
4.2 restic 增量备份
1
2
3
4
| export AWS_ACCESS_KEY_ID=minioadmin
export AWS_SECRET_ACCESS_KEY=minioadmin
export RESTIC_REPOSITORY=s3:http://127.0.0.1:9000/backup
export RESTIC_PASSWORD="backuppassword"
|
restic 要求设置备份密码。
restic 是增量备份。
1
| restic restore <snapshot_ID> --target ./restore
|
1
| restic forget --keep-last 365 --prune
|
4.3 rclone 增量备份
1
2
3
4
5
6
| rclone config create minio s3 \
access_key_id minioadmin \
secret_access_key minioadmin \
endpoint http://127.0.0.1:9000 \
region us-east-1 \
force_path_style true
|
1
| rclone sync ./ minio:backup --progress
|
rclone 是增量备份。
1
2
3
| rclone sync ./ minio:backup \
--backup-dir minio:backupold/$(date +%Y%m%d_%H%M%S) \
--progress
|
此时，覆盖或者删除的文件会被备份到 backupold 桶下。
1
| rclone copy minio:backup ./backup_restore --progress
|
5. 节点重启
1
| $CONTAINER_CLI stop minio
|
需要注意这里 local 配置的 endpoint 需要切换到正常运行的节点。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
| ● minio1:9000
Uptime: 2 hours
Version: 2025-04-22T22:12:26Z
Network: 1/2 OK
Drives: 4/4 OK
Pool: 1
● minio2:9000
Uptime: offline
Drives: 0/4 OK
┌──────┬───────────────────────┬─────────────────────┬──────────────┐
│ Pool │ Drives Usage │ Erasure stripe size │ Erasure sets │
│ 1st │ 0.8% (total: 1.5 TiB) │ 4 │ 2 │
└──────┴───────────────────────┴─────────────────────┴──────────────┘
1.5 GiB Used, 2 Buckets, 186 Objects
1 node offline, 4 drives online, 4 drives offline, EC:2
|
如果在线的磁盘数量满足要求才能正常访问数据，这里缺少一个磁盘，无法正常访问数据。
不能调整数据位、校验位数量，其他参数可以调整。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
| ● minio1:9000
Uptime: 2 hours
Version: 2025-04-22T22:12:26Z
Network: 2/2 OK
Drives: 4/4 OK
Pool: 1
● minio2:9000
Uptime: 59 seconds
Version: 2025-04-22T22:12:26Z
Network: 2/2 OK
Drives: 4/4 OK
Pool: 1
┌──────┬───────────────────────┬─────────────────────┬──────────────┐
│ Pool │ Drives Usage │ Erasure stripe size │ Erasure sets │
│ 1st │ 0.8% (total: 2.9 TiB) │ 4 │ 2 │
└──────┴───────────────────────┴─────────────────────┴──────────────┘
1.5 GiB Used, 2 Buckets, 186 Objects
8 drives online, 0 drives offline, EC:2
|
1
| mc cp -r local/test/ ./recreate/
|
6. 添加节点
新增加的节点构成新的 pool 资源池，与原有的 pool 互不干扰。
不要只加一个节点，至少每次两个以上，才能保障新 pool 的数据可靠。
新节点纠删码参数必须与原有节点一致，磁盘数量是纠删集的整数倍。
扩容过程中，需要停止 minio，修改全部实例的启动参数一致。
1
2
3
4
5
6
| cat >> /etc/hosts <<EOF
10.10.10.11 minio1
10.10.10.12 minio2
10.10.10.13 minio3
10.10.10.14 minio4
EOF
|
1
2
3
4
5
6
7
8
9
10
11
| export CONTAINER_CLI=nerdctl
export IMAGE=minio/minio:RELEASE.2025-04-22T22-12-26Z
export ROOT_USER=minioadmin
export ROOT_PASSWORD=minioadmin
export MINIO_ERASURE_SET_DRIVE_COUNT=4
export MINIO_STORAGE_CLASS_STANDARD=EC:2
export POOL_0="http://minio{1...2}/mnt/data{0...3}"
export POOL_1="http://minio{3...4}/mnt/data{0...3}"
|
1
| $CONTAINER_CLI rm -f minio
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
| $CONTAINER_CLI run -d \
--net host \
--ulimit memlock=-1 \
--ulimit stack=67108864 \
--ulimit nofile=1048576:1048576 \
--memory-swappiness=0 \
--name minio \
-v /mnt/data0:/mnt/data0 \
-v /mnt/data1:/mnt/data1 \
-v /mnt/data2:/mnt/data2 \
-v /mnt/data3:/mnt/data3 \
-e "MINIO_ROOT_USER=$ROOT_USER" \
-e "MINIO_ROOT_PASSWORD=$ROOT_PASSWORD" \
-e "MINIO_ERASURE_SET_DRIVE_COUNT=$MINIO_ERASURE_SET_DRIVE_COUNT" \
-e "MINIO_STORAGE_CLASS_STANDARD=$MINIO_STORAGE_CLASS_STANDARD" \
$IMAGE server \
$POOL_0 \
$POOL_1 \
--console-address ":9090" \
--address ":9000"
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
| ● minio1:9000
Uptime: 1 minute
Version: 2025-04-22T22:12:26Z
Network: 4/4 OK
Drives: 4/4 OK
Pool: 1
● minio2:9000
Uptime: 2 minutes
Version: 2025-04-22T22:12:26Z
Network: 4/4 OK
Drives: 4/4 OK
Pool: 1
● minio3:9000
Uptime: 3 minutes
Version: 2025-04-22T22:12:26Z
Network: 4/4 OK
Drives: 4/4 OK
Pool: 2
● minio4:9000
Uptime: 3 minutes
Version: 2025-04-22T22:12:26Z
Network: 4/4 OK
Drives: 4/4 OK
Pool: 2
┌──────┬────────────────────────┬─────────────────────┬──────────────┐
│ Pool │ Drives Usage │ Erasure stripe size │ Erasure sets │
│ 1st │ 0.9% (total: 2.9 TiB) │ 4 │ 2 │
│ 2nd │ 26.2% (total: 3.5 TiB) │ 4 │ 2 │
└──────┴────────────────────────┴─────────────────────┴──────────────┘
3.8 GiB Used, 3 Buckets, 16,022 Objects
16 drives online, 0 drives offline, EC:2
|
此时你会看到两个 pool，1st 是之前的 pool，2nd 是新加入的 pool。
1
2
3
4
5
6
7
| ┌──────┬────────────────────────┬─────────────────────┬──────────────┐
│ Pool │ Drives Usage │ Erasure stripe size │ Erasure sets │
│ 1st │ 1.1% (total: 2.9 TiB) │ 4 │ 2 │
│ 2nd │ 26.6% (total: 3.5 TiB) │ 4 │ 2 │
└──────┴────────────────────────┴─────────────────────┴──────────────┘
13 GiB Used, 3 Buckets, 18,692 Objects
|
新的数据会更多地写入新 pool，旧 pool 也会有数据写入。
1
| mc admin rebalance start local
|
1
| mc admin rebalance status local
|
1
2
3
4
5
6
7
8
| Per-pool usage:
┌────────┬────────┐
│ Pool-0 │ Pool-1 │
│ 1.24% │ 26.23% │
└────────┴────────┘
Summary:
Data: 10 GiB (1684 objects, 1684 versions)
Time: 1m3.22950215s (0s to completion)
|
minio 会在 pool 内和 pool 之间进行均衡操作。
7. 单盘更换
单盘更换之后，保持挂载目录不变。
1
| $CONTAINER_CLI stop minio
|
重新格式化，挂载新磁盘到原有目录。
1
2
| rm -rf /mnt/data0/*
rm -rf /mnt/data0/.minio.sys
|
1
| $CONTAINER_CLI start minio
|
1
| No active healing is detected for new disks, though 1 offline disk(s) found.
|
1
2
3
4
| mc admin heal local
Objects Healed: 16, 56 KiB (53.4%)
Objects Failed: 0
Heal rate: 363 obj/s, 1.2 MiB/s
|
minio 会自动重建数据。
8. 节点重建
四个节点组成两个 pool，如果删除其中一个节点的数据，将会导致其中一个 pool 副本数不足，只能读不能写。
1
| $CONTAINER_CLI stop minio
|
1
2
3
4
| for i in {0..3}; do
rm -rf /mnt/data${i}/*
rm -rf /mnt/data${i}/.minio.sys
done
|
1
2
3
| ● minio4:9000
Uptime: offline
Drives: 0/4 OK
|
节点状态为 offline。
1
| $CONTAINER_CLI start minio
|
1
| mc admin heal local --verbose
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
| mc admin heal local --verbose
Servers status:
==============
Pool 1st:
minio1:9000:
+ /mnt/data0 : HEALING
|__ Progress: 90%
|__ Started: now
|__ Capacity: 5.2 GiB/745 GiB
+ /mnt/data1 : HEALING
|__ Progress: 90%
|__ Started: now
|__ Capacity: 5.2 GiB/745 GiB
+ /mnt/data2 : HEALING
|__ Progress: 95%
|__ Started: now
|__ Capacity: 5.2 GiB/745 GiB
+ /mnt/data3 : HEALING
|__ Progress: 95%
|__ Started: now
|__ Capacity: 5.2 GiB/745 GiB
Pool 2nd:
Summary:
=======
Objects Healed: 4, 856 B (90.4%)
Objects Failed: 0
Heal rate: 142 obj/s, 30 KiB/s
2 of 4 sets exceeds reduced parity count EC:1 lost/offline disks
|
9. 常见报错
1
2
3
| FATAL Unable to initialize backend:
/mnt/data0 drive is already being used in another erasure deployment.
(Number of drives specified: 16 but the number of drives found in the 5th drive's format.json: 8)
|
如果遇到这个问题，暂时无解，需要备份之后重建集群。因为 minio 试图将全部盘组成一个新的 pool，而不是新增一个 pool。建议使用 hosts 固定节点名进行部署。
