Calico启用纯BGP模式+RR实战案例

博客园 - 尹正杰

TKE集群LoadBalancer映射案例 k8s集群为每个节点分配不通的ippool实战 ansible剧本实现一键升级K8S集群的NVIDIA GPU驱动 vmagent采集etcd数据并写入victoriametrics集群 VictoriaMetrics集群部署实战 Squid正向代理实战 kubespray生产故障案例之kubelet启动参数案例 filebeat指定名称空间采集pod数据 Calico自定义Ipool实战 ansible剧本实现kibana的部署和卸载 ansible剧本应用案例集合2 ansible剧本实现磁盘格式化 Ansible急速入门实战篇 ansible快速入门篇 k8s集群管理 kubesray实战 kubespray实战案例 kubespray管理k8s的worker集群扩缩容 kubespray快速部署k8s集群实战 Kubeasz使用吐槽博客专题 Kubeasz基于ezctl实现etcd集群的管理实战 Kubeasz基于ezctl实现k8s集群一键升级 Calico 底层原理及IPIP（依赖BGP协议）)和vxlan（不依赖BGP）工作模式切换 kubeasz基于ezctl实现k8s集群的扩容和缩容 kubeasz快速部署K8S集群实战 Ubuntu 24.04.04 LTS版本系统优化 Ubuntu Server 24.04.04 LTS部署指南 windows极速部署Openclaw实战篇 K8S的StatefulSet控制器应用案例之MySQL主从同步实战 k8s底层基于不同运行时集成harbor企业级私有仓库实战二进制K8S集群附加组件部署及CNI网络插件切换实战二进制部署K8S 1.35.0+最新版实战案例 etcd高可用集群部署及K8S周期性备份数据实战基于Docker实现《若依》服务业务容器化实战篇 k8s集群基于Flannel网络插件部署凡人修仙传 k8s集群基于Calico网络插件部署凡人修仙传 ElasticSEearch 9.X环境部署 K8S Vertical Pod Autoscaler(VPA)实战案例 Prometheus监控自定义程序指标

尹正杰 · 2026-03-01 · via 博客园 - 尹正杰

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正杰

一.Route Reflector概述
二.Calico配置Route Reflector

一.Route Reflector概述

1.Route Reflector作用

默认 Calico 是 BGP Full Mesh（全互联）：N 个节点，两两建立 BGP 邻居，N² 连接，规模大了撑不住。


RR（Route Reflector）：选 2–3 个节点当 RR，其他节点只和 RR 建邻居，连接数变成 2N，大幅降低 BGP 压力。

RR适用场景为私有云、物理机、底层网络可控、用BGP模式（不是VXLAN/IPIP overlay）。

2.Route Reflector前提条件

- 1.集群 AS 号统一（默认 64512）

- 2.选 2–3 个稳定节点（推荐 master / 控制面）做 RR

- 3.关闭默认的 nodeToNodeMesh

3.特别注意

3.1 Full Mesh（BGP 全互联）

Full Mesh指的是集群里每一台节点，和集群里所有其余节点全部单独建立一条 BGP 邻居（TCP 179）。

举例：你一共 5 台节点：231、232、233、66、77
	231 ↔ 232、233、66、77（4 条 BGP）
	232 ↔ 231、233、66、77（4 条 BGP）
	233 ↔ 其余 4 台
	66 ↔ 其余 4 台
	77 ↔ 其余 4 台

总连接数：N*(N-1)/2，5 节点 = 10 条 BGP 会话。

因此，节点越多，连接指数暴涨：100 节点≈4950 条 BGP 连接，BIRD 进程负载拉满。

BGP Full Mesh 连接数公式：
	总数 = N × (N - 1) / 2
	
	
假设900个节点，900 × 899 / 2 = 404550条BGP链接。

如果用 2 台 RR，每个节点只建 2 条连接，全集群总连接数：1800(2*900) 条，差距 200 倍！

RR 路由反射器就是用来替代 Full Mesh，启用RR后:
	- 所有普通节点只和 2 台 RR 建 BGP 邻居，不再和其他业务节点互联。
	- 5 节点场景：每个节点仅 2 条 BGP，总连接数大幅下降。

RR 负责收集全网路由、反射转发给所有客户端。

3.2 VXLAN不支持RR模式

Calico 配置 Route Reflector 必须使用纯 BGP/IPIP 模式，VXLAN 模式不支持 RR！

只要开启了"vxlanMode: Always / CrossSubnet"，Calico会自动强制启用 node-to-node BGP 全互联（mesh），你手动设置 nodeToNodeMeshEnabled: false 是不生效的！

VXLAN依赖BGP来同步隧道端点（VTEP），为了保证隧道稳定，强制使用 full mesh，不允许用 Route Reflector。

二.Calico配置Route Reflector

1.更改calicoctl客户端版本

	1.查看Calico的版本【很明显，客户端和服务端版本不一致】
[root@ansible99 ~]# /etc/kubeasz/bin/calicoctl version
Client Version:    v3.28.4  # 默认的客户端版本安装的是3.28.4
Git commit:        122890af8
Cluster Version:   v3.31.5  # 我的k8s v1.33.11安装的是v3.31.5的服务端
Cluster Type:      k8s,kdd,bgp
[root@ansible99 ~]# 



	2.下载和我们Calicoctl服务端版本一致的软件包
[root@ansible99 ~]# wget https://github.com/projectcalico/calico/releases/download/v3.31.5/calicoctl-linux-amd64 -O /etc/kubeasz/bin/calicoctl


	3.覆盖原有的软件包并测试验证
[root@ansible99 ~]# chmod +x /etc/kubeasz/bin/calicoctl 
[root@ansible99 ~]# 
[root@ansible99 ~]# /etc/kubeasz/bin/calicoctl version
Client Version:    v3.31.5
Git commit:        2e4da40144aac869e1ed2cc220b6c4b62f32efdd
Cluster Version:   v3.31.5
Cluster Type:      k8s,kdd,bgp
[root@ansible99 ~]#

2.指定Calico后端存储是Kubernetes

	1.备份calicoctl.cfg.j2文件
[root@ansible99 ~]# cp /etc/kubeasz/roles/calico/templates/calicoctl.cfg.j2{,.bak} 
[root@ansible99 ~]# 
[root@ansible99 ~]# ll /etc/kubeasz/roles/calico/templates/calicoctl.cfg.j2*
-rw-rw-r-- 1 root root 263 Jun  8  2025 /etc/kubeasz/roles/calico/templates/calicoctl.cfg.j2
-rw-r--r-- 1 root root 263 Jun  3 06:25 /etc/kubeasz/roles/calico/templates/calicoctl.cfg.j2.bak
[root@ansible99 ~]# 
[root@ansible99 ~]# 


	2.修改calicoctl.cfg.j2文件的使用Kubernetes进行存储
[root@ansible99 ~]# cat  /etc/kubeasz/roles/calico/templates/calicoctl.cfg.j2
apiVersion: projectcalico.org/v3
kind: CalicoAPIConfig
metadata:
spec:
  datastoreType: "kubernetes"
  kubeconfig: "/root/.kube/config"
[root@ansible99 ~]#

3.修改在远端节点执行打标签操作

	1.在ansible控制节点执行打标签操作
[root@ansible99 ~]# grep connection /etc/kubeasz/roles/calico/tasks/calico-rr.yml 
      connection: local
[root@ansible99 ~]# 
[root@ansible99 ~]# 


	2.打标签不在控制节点执行，因为控制节点可能不属于k8s集群内部
[root@ansible99 ~]# sed -i '/connection/s@connection@#connection@' /etc/kubeasz/roles/calico/tasks/calico-rr.yml 
[root@ansible99 ~]# 
[root@ansible99 ~]# grep connection /etc/kubeasz/roles/calico/tasks/calico-rr.yml 
      #connection: local
[root@ansible99 ~]#

4.查看现有的BGP工作模式【全网互联，Full Mesh】

[root@ansible99 ~]# dk ansible -i /etc/kubeasz/clusters/yinzhengjie-k8s/hosts all -m shell -a 'calicoctl node status'
10.0.0.231 | CHANGED | rc=0 >>
Calico process is running.

IPv4 BGP status
+--------------+-------------------+-------+----------+-------------+
| PEER ADDRESS |     PEER TYPE     | STATE |  SINCE   |    INFO     |
+--------------+-------------------+-------+----------+-------------+
| 10.0.0.232   | node-to-node mesh | up    | 01:59:57 | Established |
| 10.0.0.233   | node-to-node mesh | up    | 01:59:57 | Established |
| 10.0.0.66    | node-to-node mesh | up    | 01:59:58 | Established |
| 10.0.0.77    | node-to-node mesh | up    | 01:59:57 | Established |
+--------------+-------------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.
10.0.0.66 | CHANGED | rc=0 >>
Calico process is running.

IPv4 BGP status
+--------------+-------------------+-------+----------+-------------+
| PEER ADDRESS |     PEER TYPE     | STATE |  SINCE   |    INFO     |
+--------------+-------------------+-------+----------+-------------+
| 10.0.0.231   | node-to-node mesh | up    | 01:59:58 | Established |
| 10.0.0.232   | node-to-node mesh | up    | 01:59:56 | Established |
| 10.0.0.233   | node-to-node mesh | up    | 01:59:56 | Established |
| 10.0.0.77    | node-to-node mesh | up    | 01:59:56 | Established |
+--------------+-------------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.
10.0.0.232 | CHANGED | rc=0 >>
Calico process is running.

IPv4 BGP status
+--------------+-------------------+-------+----------+-------------+
| PEER ADDRESS |     PEER TYPE     | STATE |  SINCE   |    INFO     |
+--------------+-------------------+-------+----------+-------------+
| 10.0.0.231   | node-to-node mesh | up    | 01:59:57 | Established |
| 10.0.0.233   | node-to-node mesh | up    | 01:59:57 | Established |
| 10.0.0.66    | node-to-node mesh | up    | 01:59:55 | Established |
| 10.0.0.77    | node-to-node mesh | up    | 01:59:55 | Established |
+--------------+-------------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.
10.0.0.77 | CHANGED | rc=0 >>
Calico process is running.

IPv4 BGP status
+--------------+-------------------+-------+----------+-------------+
| PEER ADDRESS |     PEER TYPE     | STATE |  SINCE   |    INFO     |
+--------------+-------------------+-------+----------+-------------+
| 10.0.0.231   | node-to-node mesh | up    | 01:59:58 | Established |
| 10.0.0.232   | node-to-node mesh | up    | 01:59:56 | Established |
| 10.0.0.233   | node-to-node mesh | up    | 01:59:56 | Established |
| 10.0.0.66    | node-to-node mesh | up    | 01:59:56 | Established |
+--------------+-------------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.
10.0.0.233 | CHANGED | rc=0 >>
Calico process is running.

IPv4 BGP status
+--------------+-------------------+-------+----------+-------------+
| PEER ADDRESS |     PEER TYPE     | STATE |  SINCE   |    INFO     |
+--------------+-------------------+-------+----------+-------------+
| 10.0.0.231   | node-to-node mesh | up    | 01:59:58 | Established |
| 10.0.0.232   | node-to-node mesh | up    | 01:59:58 | Established |
| 10.0.0.66    | node-to-node mesh | up    | 01:59:56 | Established |
| 10.0.0.77    | node-to-node mesh | up    | 01:59:56 | Established |
+--------------+-------------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.
[root@ansible99 ~]#

5.开启RR模式

	1.开启BGP的RR模式
[root@ansible99 ~]# vim /etc/kubeasz/clusters/yinzhengjie-k8s/config.yml 
...
CALICO_RR_ENABLED: true


	2.重新安装网络插件
[root@ansible99 ~]# dk ezctl setup yinzhengjie-k8s 06 -v

6.再次查看BGP的工作模式

[root@ansible99 ~]# dk ansible -i /etc/kubeasz/clusters/yinzhengjie-k8s/hosts all -m shell -a 'calicoctl node status'
10.0.0.232 | CHANGED | rc=0 >>
Calico process is running.

IPv4 BGP status
+--------------+---------------+-------+----------+-------------+
| PEER ADDRESS |   PEER TYPE   | STATE |  SINCE   |    INFO     |
+--------------+---------------+-------+----------+-------------+
| 10.0.0.231   | node specific | up    | 09:52:22 | Established |
| 10.0.0.233   | node specific | up    | 09:52:22 | Established |
| 10.0.0.77    | node specific | up    | 09:52:22 | Established |
| 10.0.0.66    | node specific | up    | 11:24:09 | Established |
+--------------+---------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.
10.0.0.231 | CHANGED | rc=0 >>
Calico process is running.

IPv4 BGP status
+--------------+---------------+-------+----------+-------------+
| PEER ADDRESS |   PEER TYPE   | STATE |  SINCE   |    INFO     |
+--------------+---------------+-------+----------+-------------+
| 10.0.0.232   | node specific | up    | 09:52:21 | Established |
| 10.0.0.233   | node specific | up    | 09:52:21 | Established |
+--------------+---------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.
10.0.0.77 | CHANGED | rc=0 >>
Calico process is running.

IPv4 BGP status
+--------------+---------------+-------+----------+-------------+
| PEER ADDRESS |   PEER TYPE   | STATE |  SINCE   |    INFO     |
+--------------+---------------+-------+----------+-------------+
| 10.0.0.232   | node specific | up    | 09:52:22 | Established |
| 10.0.0.233   | node specific | up    | 09:52:22 | Established |
+--------------+---------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.
10.0.0.233 | CHANGED | rc=0 >>
Calico process is running.

IPv4 BGP status
+--------------+---------------+-------+----------+-------------+
| PEER ADDRESS |   PEER TYPE   | STATE |  SINCE   |    INFO     |
+--------------+---------------+-------+----------+-------------+
| 10.0.0.231   | node specific | up    | 09:52:22 | Established |
| 10.0.0.232   | node specific | up    | 09:52:22 | Established |
| 10.0.0.77    | node specific | up    | 09:52:22 | Established |
| 10.0.0.66    | node specific | up    | 11:24:09 | Established |
+--------------+---------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.
10.0.0.66 | CHANGED | rc=0 >>
Calico process is running.

IPv4 BGP status
+--------------+---------------+-------+----------+-------------+
| PEER ADDRESS |   PEER TYPE   | STATE |  SINCE   |    INFO     |
+--------------+---------------+-------+----------+-------------+
| 10.0.0.232   | node specific | up    | 11:24:09 | Established |
| 10.0.0.233   | node specific | up    | 11:24:09 | Established |
+--------------+---------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.
[root@ansible99 ~]#

此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。原文来自 — 版权归原作者所有。

推荐订阅源

博客园 - 尹正杰

一.Route Reflector概述

1.Route Reflector作用

2.Route Reflector前提条件

3.特别注意

3.1 Full Mesh（BGP 全互联）

3.2 VXLAN不支持RR模式

二.Calico配置Route Reflector

1.更改calicoctl客户端版本

2.指定Calico后端存储是Kubernetes

3.修改在远端节点执行打标签操作

4.查看现有的BGP工作模式【全网互联，Full Mesh】

5.开启RR模式

6.再次查看BGP的工作模式