1. 配置文件准备
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
| ● minio1:9000
Uptime: 55 minutes
Version: 2025-04-22T22:12:26Z
Network: 2/2 OK
Drives: 4/4 OK
Pool: 1
● minio2:9000
Uptime: 55 minutes
Version: 2025-04-22T22:12:26Z
Network: 2/2 OK
Drives: 4/4 OK
Pool: 1
┌──────┬───────────────────────┬─────────────────────┬──────────────┐
│ Pool │ Drives Usage │ Erasure stripe size │ Erasure sets │
│ 1st │ 0.7% (total: 2.9 TiB) │ 4 │ 2 │
└──────┴───────────────────────┴─────────────────────┴──────────────┘
0 B Used, 1 Bucket, 0 Objects
8 drives online, 0 drives offline, EC:2
|
1
2
3
4
| cat >> /etc/hosts <<EOF
10.10.10.11 minio1
10.10.10.12 minio2
EOF
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
| mkdir -p /etc/ops/haproxy
cat > /etc/ops/haproxy/haproxy.cfg <<'EOF'
global
log stdout format raw local0
maxconn 50000
user root
defaults
mode tcp
timeout connect 10s
timeout client 24h
timeout server 24h
frontend minio_api_front
bind *:80
mode tcp
default_backend minio_api_back
backend minio_api_back
mode tcp
balance roundrobin
option tcp-check
retries 3
option redispatch
default-server inter 10s fall 3 rise 2
server minio1 minio1:9000 check
server minio2 minio2:9000 check
EOF
|
2. 容器部署
1
2
| export CONTAINER_CLI=nerdctl
export IMAGE=haproxy
|
1
2
3
4
5
6
7
8
9
10
11
12
| $CONTAINER_CLI run -d \
--net host \
--ulimit memlock=-1 \
--ulimit stack=67108864 \
--ulimit nofile=1048576:1048576 \
--memory-swappiness=0 \
--privileged \
--user root \
--name haproxy \
-v /etc/ops/haproxy/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro \
$IMAGE \
haproxy -f /usr/local/etc/haproxy/haproxy.cfg
|
3. 客户端使用
配置客户端 hosts 文件指向 haproxy 代理端口。
1
2
3
| cat >> /etc/hosts <<EOF
10.10.10.10 minio
EOF
|
1
2
| mc alias set minio http://minio minioadmin minioadmin
mc ls minio
|
4. 性能测试
安装测试工具
1
2
| wget https://dl.min.io/aistor/warp/release/linux-amd64/archive/warp -O /usr/bin/warp
chmod +x /usr/bin/warp
|
4.1 多客户端直连单 minio 节点
1
2
3
4
5
6
7
8
| warp get \
--host minio2:9000 \
--access-key minioadmin \
--secret-key minioadmin \
--tls=false \
--obj.size 4MiB \
--concurrent 32 \
--duration 5m
|
1
2
3
4
5
6
7
8
9
10
11
12
13
| Reqs: 84305, Errs:0, Objs:84305, Bytes: 329.32GiB
- GET Average: 282 Obj/s, 1127.2MiB/s; Current 282 Obj/s, 1127.7MiB/s, 113.5 ms/req, TTFB: 30.9m
Report: GET. Concurrency: 32. Ran: 4m57s
* Average: 1127.23 MiB/s, 281.81 obj/s
* Reqs: Avg: 113.6ms, 50%: 109.4ms, 90%: 142.0ms, 99%: 201.8ms, Fastest: 24.8ms, Slowest: 628.0ms, StdDev: 28.2ms
* TTFB: Avg: 31ms, Best: 6ms, 25th: 27ms, Median: 30ms, 75th: 32ms, 90th: 34ms, 99th: 54ms, Worst: 479ms StdDev: 17ms
Throughput, split into 297 x 1s:
* Fastest: 1172.9MiB/s, 293.22 obj/s
* 50% Median: 1126.6MiB/s, 281.65 obj/s
* Slowest: 1066.6MiB/s, 266.64 obj/s
|
1
2
3
4
5
6
7
8
| warp put \
--host minio2:9000 \
--access-key minioadmin \
--secret-key minioadmin \
--tls=false \
--obj.size 4MiB \
--concurrent 32 \
--duration 5m
|
1
2
3
4
5
6
7
8
9
10
11
12
| Reqs: 80844, Errs:0, Objs:80844, Bytes: 315.80GiB
- PUT Average: 270 Obj/s, 1080.1MiB/s; Current 271 Obj/s, 1084.6MiB/s, 118.4 ms/req
Report: PUT. Concurrency: 32. Ran: 4m57s
* Average: 1080.02 MiB/s, 270.01 obj/s
* Reqs: Avg: 118.2ms, 50%: 113.8ms, 90%: 151.3ms, 99%: 195.0ms, Fastest: 47.9ms, Slowest: 459.5ms, StdDev: 26.3ms
Throughput, split into 297 x 1s:
* Fastest: 1124.8MiB/s, 281.21 obj/s
* 50% Median: 1081.9MiB/s, 270.48 obj/s
* Slowest: 1012.8MiB/s, 253.21 obj/s
|
4.2 多客户端直连多 minio 节点
1
2
3
4
5
6
7
8
| warp get \
--host minio1:9000,minio2:9000 \
--access-key minioadmin \
--secret-key minioadmin \
--tls=false \
--obj.size 4MiB \
--concurrent 32 \
--duration 5m
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
| Reqs: 122575, Errs:0, Objs:122575, Bytes: 478.81GiB
- GET Average: 410 Obj/s, 1640.2MiB/s; Current 409 Obj/s, 1635.8MiB/s, 77.8 ms/req, TTFB: 31.1ms
Report: GET. Concurrency: 32. Ran: 4m57s
* Average: 1640.26 MiB/s, 410.07 obj/s
* Reqs: Avg: 78.1ms, 50%: 70.5ms, 90%: 115.6ms, 99%: 160.7ms, Fastest: 11.9ms, Slowest: 393.6ms, StdDev: 29.6ms
* TTFB: Avg: 31ms, Best: 5ms, 25th: 27ms, Median: 30ms, 75th: 32ms, 90th: 35ms, 99th: 52ms, Worst: 274ms StdDev: 13ms
Throughput by host:
* http://minio1:9000: Avg: 1028.60 MiB/s, 257.15 obj/s
* http://minio2:9000: Avg: 611.74 MiB/s, 152.94 obj/s
Throughput, split into 297 x 1s:
* Fastest: 1697.3MiB/s, 424.32 obj/s
* 50% Median: 1640.7MiB/s, 410.18 obj/s
* Slowest: 1570.7MiB/s, 392.67 obj/s
|
1
2
3
4
5
6
7
8
| warp put \
--host minio1:9000,minio2:9000 \
--access-key minioadmin \
--secret-key minioadmin \
--tls=false \
--obj.size 4MiB \
--concurrent 32 \
--duration 5m
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
| Reqs: 82451, Errs:0, Objs:82451, Bytes: 322.07GiB
- PUT Average: 275 Obj/s, 1101.0MiB/s; Current 278 Obj/s, 1112.3MiB/s, 116.4 ms/req
Report: PUT. Concurrency: 32. Ran: 4m57s
* Average: 1101.03 MiB/s, 275.26 obj/s
* Reqs: Avg: 116.1ms, 50%: 102.3ms, 90%: 172.6ms, 99%: 344.1ms, Fastest: 42.3ms, Slowest: 930.4ms, StdDev: 54.0ms
Throughput by host:
* http://minio1:9000: Avg: 618.21 MiB/s, 154.55 obj/s
* http://minio2:9000: Avg: 482.79 MiB/s, 120.70 obj/s
Throughput, split into 297 x 1s:
* Fastest: 1131.4MiB/s, 282.86 obj/s
* 50% Median: 1103.8MiB/s, 275.94 obj/s
* Slowest: 1029.5MiB/s, 257.37 obj/s
|
4.3 多客户端经过 HAProxy 压测
1
2
3
4
5
6
7
8
| warp get \
--host minio \
--access-key minioadmin \
--secret-key minioadmin \
--tls=false \
--obj.size 4MiB \
--concurrent 32 \
--duration 5m
|
1
2
3
4
5
6
7
8
9
10
11
12
13
| Reqs: 122680, Errs:0, Objs:122680, Bytes: 479.22GiB
- GET Average: 409 Obj/s, 1636.0MiB/s; Current 418 Obj/s, 1671.1MiB/s, 78.5 ms/req, TTFB: 31.0ms
Report: GET. Concurrency: 32. Ran: 4m57s
* Average: 1635.98 MiB/s, 409.00 obj/s
* Reqs: Avg: 78.2ms, 50%: 70.6ms, 90%: 115.2ms, 99%: 161.8ms, Fastest: 16.4ms, Slowest: 549.6ms, StdDev: 29.8ms
* TTFB: Avg: 31ms, Best: 6ms, 25th: 27ms, Median: 30ms, 75th: 32ms, 90th: 35ms, 99th: 58ms, Worst: 494ms StdDev: 14ms
Throughput, split into 297 x 1s:
* Fastest: 1690.4MiB/s, 422.60 obj/s
* 50% Median: 1636.2MiB/s, 409.06 obj/s
* Slowest: 1498.4MiB/s, 374.60 obj/s
|
1
2
3
4
5
6
7
8
| warp put \
--host minio \
--access-key minioadmin \
--secret-key minioadmin \
--tls=false \
--obj.size 4MiB \
--concurrent 32 \
--duration 5m
|
1
2
3
4
5
6
7
8
9
10
11
12
| Reqs: 82644, Errs:0, Objs:82644, Bytes: 322.83GiB
- PUT Average: 276 Obj/s, 1103.9MiB/s; Current 278 Obj/s, 1111.9MiB/s, 116.3 ms/req
Report: PUT. Concurrency: 32. Ran: 4m57s
* Average: 1103.93 MiB/s, 275.98 obj/s
* Reqs: Avg: 115.8ms, 50%: 104.1ms, 90%: 172.4ms, 99%: 303.1ms, Fastest: 40.8ms, Slowest: 745.1ms, StdDev: 48.2ms
Throughput, split into 297 x 1s:
* Fastest: 1136.4MiB/s, 284.11 obj/s
* 50% Median: 1104.9MiB/s, 276.24 obj/s
* Slowest: 1042.9MiB/s, 260.73 obj/s
|
4.4 总结
测试采用的是一个两节点，每个节点四盘位全闪磁盘的 MinIO 集群。在测试过程中，网卡使用率接近 100%，峰值 10Gbps; 单盘磁盘 IO utils 峰值在 85%，吞吐 250 MB/s。
| 压测方式 | GET 吞吐 | PUT 吞吐 |
|---|
| 直连单节点 | 1127.23 MiB/s | 1080.02 MiB/s |
| 直连多节点 | 1640.26 MiB/s | 1101.03 MiB/s |
| 经过 HAProxy | 1635.98 MiB/s | 1103.93 MiB/s |
HAProxy 代理 MinIO 之后，GET 吞吐高于单节点直连，与多节点接近；PUT 吞吐在直连单节点、多节点直连、经过 HAProxy 下差不多。
