使用 HAProxy 代理 MinIO 及性能测试

1. 配置文件准备

minio 集群

●  minio1:9000
   Uptime: 55 minutes
   Version: 2025-04-22T22:12:26Z
   Network: 2/2 OK
   Drives: 4/4 OK
   Pool: 1

●  minio2:9000
   Uptime: 55 minutes
   Version: 2025-04-22T22:12:26Z
   Network: 2/2 OK
   Drives: 4/4 OK
   Pool: 1

┌──────┬───────────────────────┬─────────────────────┬──────────────┐
│ Pool │ Drives Usage          │ Erasure stripe size │ Erasure sets │
│ 1st  │ 0.7% (total: 2.9 TiB) │ 4                   │ 2            │
└──────┴───────────────────────┴─────────────────────┴──────────────┘

0 B Used, 1 Bucket, 0 Objects
8 drives online, 0 drives offline, EC:2

准备 hosts 文件

cat >> /etc/hosts <<EOF
10.10.10.11 minio1
10.10.10.12 minio2
EOF

准备配置文件

mkdir -p /etc/ops/haproxy
cat > /etc/ops/haproxy/haproxy.cfg <<'EOF'
global
    log stdout format raw local0
    maxconn 50000
    user root

defaults
    mode tcp
    timeout connect 10s
    timeout client  24h
    timeout server  24h

frontend minio_api_front
    bind *:80
    mode tcp
    default_backend minio_api_back

backend minio_api_back
    mode tcp
    balance roundrobin
    option tcp-check
    retries 3
    option redispatch
    default-server inter 10s fall 3 rise 2
    server minio1 minio1:9000 check
    server minio2 minio2:9000 check
EOF

2. 容器部署

环境变量

1
2

export CONTAINER_CLI=nerdctl
export IMAGE=haproxy

启动容器

$CONTAINER_CLI run -d \
  --net host \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  --ulimit nofile=1048576:1048576 \
  --memory-swappiness=0 \
  --privileged \
  --user root \
  --name haproxy \
  -v /etc/ops/haproxy/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro \
  $IMAGE \
  haproxy -f /usr/local/etc/haproxy/haproxy.cfg

3. 客户端使用

配置 hosts 文件

配置客户端 hosts 文件指向 haproxy 代理端口。

1
2
3

cat >> /etc/hosts <<EOF
10.10.10.10 minio
EOF

命令行使用

1
2

mc alias set minio http://minio minioadmin minioadmin
mc ls minio

4. 性能测试

安装测试工具

1
2

wget https://dl.min.io/aistor/warp/release/linux-amd64/archive/warp -O /usr/bin/warp
chmod +x /usr/bin/warp

4.1 多客户端直连单 minio 节点

GET 压测

warp get \
  --host minio2:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 4MiB \
  --concurrent 32 \
  --duration 5m

Reqs: 84305, Errs:0, Objs:84305, Bytes: 329.32GiB
 -       GET Average: 282 Obj/s, 1127.2MiB/s; Current 282 Obj/s, 1127.7MiB/s, 113.5 ms/req, TTFB: 30.9m


Report: GET. Concurrency: 32. Ran: 4m57s
 * Average: 1127.23 MiB/s, 281.81 obj/s
 * Reqs: Avg: 113.6ms, 50%: 109.4ms, 90%: 142.0ms, 99%: 201.8ms, Fastest: 24.8ms, Slowest: 628.0ms, StdDev: 28.2ms
 * TTFB: Avg: 31ms, Best: 6ms, 25th: 27ms, Median: 30ms, 75th: 32ms, 90th: 34ms, 99th: 54ms, Worst: 479ms StdDev: 17ms

Throughput, split into 297 x 1s:
 * Fastest: 1172.9MiB/s, 293.22 obj/s
 * 50% Median: 1126.6MiB/s, 281.65 obj/s
 * Slowest: 1066.6MiB/s, 266.64 obj/s

PUT 压测

warp put \
  --host minio2:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 4MiB \
  --concurrent 32 \
  --duration 5m

Reqs: 80844, Errs:0, Objs:80844, Bytes: 315.80GiB
 -       PUT Average: 270 Obj/s, 1080.1MiB/s; Current 271 Obj/s, 1084.6MiB/s, 118.4 ms/req


Report: PUT. Concurrency: 32. Ran: 4m57s
 * Average: 1080.02 MiB/s, 270.01 obj/s
 * Reqs: Avg: 118.2ms, 50%: 113.8ms, 90%: 151.3ms, 99%: 195.0ms, Fastest: 47.9ms, Slowest: 459.5ms, StdDev: 26.3ms

Throughput, split into 297 x 1s:
 * Fastest: 1124.8MiB/s, 281.21 obj/s
 * 50% Median: 1081.9MiB/s, 270.48 obj/s
 * Slowest: 1012.8MiB/s, 253.21 obj/s

4.2 多客户端直连多 minio 节点

GET 压测

warp get \
  --host minio1:9000,minio2:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 4MiB \
  --concurrent 32 \
  --duration 5m

Reqs: 122575, Errs:0, Objs:122575, Bytes: 478.81GiB
 -       GET Average: 410 Obj/s, 1640.2MiB/s; Current 409 Obj/s, 1635.8MiB/s, 77.8 ms/req, TTFB: 31.1ms


Report: GET. Concurrency: 32. Ran: 4m57s
 * Average: 1640.26 MiB/s, 410.07 obj/s
 * Reqs: Avg: 78.1ms, 50%: 70.5ms, 90%: 115.6ms, 99%: 160.7ms, Fastest: 11.9ms, Slowest: 393.6ms, StdDev: 29.6ms
 * TTFB: Avg: 31ms, Best: 5ms, 25th: 27ms, Median: 30ms, 75th: 32ms, 90th: 35ms, 99th: 52ms, Worst: 274ms StdDev: 13ms

Throughput by host:
 * http://minio1:9000: Avg: 1028.60 MiB/s, 257.15 obj/s
 * http://minio2:9000: Avg: 611.74 MiB/s, 152.94 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 1697.3MiB/s, 424.32 obj/s
 * 50% Median: 1640.7MiB/s, 410.18 obj/s
 * Slowest: 1570.7MiB/s, 392.67 obj/s

PUT 压测

warp put \
  --host minio1:9000,minio2:9000 \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 4MiB \
  --concurrent 32 \
  --duration 5m

Reqs: 82451, Errs:0, Objs:82451, Bytes: 322.07GiB
 -       PUT Average: 275 Obj/s, 1101.0MiB/s; Current 278 Obj/s, 1112.3MiB/s, 116.4 ms/req


Report: PUT. Concurrency: 32. Ran: 4m57s
 * Average: 1101.03 MiB/s, 275.26 obj/s
 * Reqs: Avg: 116.1ms, 50%: 102.3ms, 90%: 172.6ms, 99%: 344.1ms, Fastest: 42.3ms, Slowest: 930.4ms, StdDev: 54.0ms

Throughput by host:
 * http://minio1:9000: Avg: 618.21 MiB/s, 154.55 obj/s
 * http://minio2:9000: Avg: 482.79 MiB/s, 120.70 obj/s

Throughput, split into 297 x 1s:
 * Fastest: 1131.4MiB/s, 282.86 obj/s
 * 50% Median: 1103.8MiB/s, 275.94 obj/s
 * Slowest: 1029.5MiB/s, 257.37 obj/s

4.3 多客户端经过 HAProxy 压测

GET 压测

warp get \
  --host minio \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 4MiB \
  --concurrent 32 \
  --duration 5m

Reqs: 122680, Errs:0, Objs:122680, Bytes: 479.22GiB
 -       GET Average: 409 Obj/s, 1636.0MiB/s; Current 418 Obj/s, 1671.1MiB/s, 78.5 ms/req, TTFB: 31.0ms


Report: GET. Concurrency: 32. Ran: 4m57s
 * Average: 1635.98 MiB/s, 409.00 obj/s
 * Reqs: Avg: 78.2ms, 50%: 70.6ms, 90%: 115.2ms, 99%: 161.8ms, Fastest: 16.4ms, Slowest: 549.6ms, StdDev: 29.8ms
 * TTFB: Avg: 31ms, Best: 6ms, 25th: 27ms, Median: 30ms, 75th: 32ms, 90th: 35ms, 99th: 58ms, Worst: 494ms StdDev: 14ms

Throughput, split into 297 x 1s:
 * Fastest: 1690.4MiB/s, 422.60 obj/s
 * 50% Median: 1636.2MiB/s, 409.06 obj/s
 * Slowest: 1498.4MiB/s, 374.60 obj/s

PUT 压测

warp put \
  --host minio \
  --access-key minioadmin \
  --secret-key minioadmin \
  --tls=false \
  --obj.size 4MiB \
  --concurrent 32 \
  --duration 5m

Reqs: 82644, Errs:0, Objs:82644, Bytes: 322.83GiB
 -       PUT Average: 276 Obj/s, 1103.9MiB/s; Current 278 Obj/s, 1111.9MiB/s, 116.3 ms/req


Report: PUT. Concurrency: 32. Ran: 4m57s
 * Average: 1103.93 MiB/s, 275.98 obj/s
 * Reqs: Avg: 115.8ms, 50%: 104.1ms, 90%: 172.4ms, 99%: 303.1ms, Fastest: 40.8ms, Slowest: 745.1ms, StdDev: 48.2ms

Throughput, split into 297 x 1s:
 * Fastest: 1136.4MiB/s, 284.11 obj/s
 * 50% Median: 1104.9MiB/s, 276.24 obj/s
 * Slowest: 1042.9MiB/s, 260.73 obj/s

4.4 总结

测试采用的是一个两节点，每个节点四盘位全闪磁盘的 MinIO 集群。在测试过程中，网卡使用率接近 100%，峰值 10Gbps; 单盘磁盘 IO utils 峰值在 85%，吞吐 250 MB/s。

压测方式	GET 吞吐	PUT 吞吐
直连单节点	1127.23 MiB/s	1080.02 MiB/s
直连多节点	1640.26 MiB/s	1101.03 MiB/s
经过 HAProxy	1635.98 MiB/s	1103.93 MiB/s

HAProxy 代理 MinIO 之后，GET 吞吐高于单节点直连，与多节点接近；PUT 吞吐在直连单节点、多节点直连、经过 HAProxy 下差不多。

推荐订阅源

陈少文的网站