项目文档：https://docs.tigera.io/calico/latest/networking/configuring/bgp#route-reflectors
BGP 路由反射器（RR）减少了网络中对全互联（full-mesh）内部 BGP（iBGP）对等连接的需求，使用 full-mesh 时：
而使用路由反射器（RR）时，同一自治系统（AS）内的路由器只需与 RR 建立邻居关系即可。RR 将路由信息从一台 iBGP 路由器 "反射" 给其他的 iBGP 邻居。通过这种方式，就不需要构建 iBGP 全互联（full-mesh）拓扑了。这是一种 服务器-客户端 解决方案：其中 RR 充当服务器，其余所有 iBGP 路由器作为客户端。这种方案简化了配置，减少了 iBGP 建立邻居关系的数量，同时也节省了 CPU 和网络资源。
在大型网络中使用 full-mesh 全互联时，iBGP 邻居关系的数量会成为一个问题。通过下图可以更清晰的说明这点：
如上图所示，一个包含六台 iBGP 路由器的网络。利用全互联（Full Mesh）公式 N(N-1)/2，可以计算出 iBGP 对等体的数量：
当我们使用路由反射器时，网络拓扑大概长这样：
这六台路由器其中五台，仅与顶层的路由反射器建立了 iBGP 对等连接。当其中一台 iBGP 路由器向路由反射器通告某条路由时，该路由将被反射给所有其他的 iBGP 路由器。
这极大地简化了 iBGP 配置，但也存在一个弊端：如果路由反射器发生故障怎么办？在上图场景中，它构成了一个单点故障。对此的解决方案是，可以在网络中部署多个路由反射器，对路由反射器之间配置 full-mesh 全互联模式。
但在真实场景中，有物理交换机 ToR（Top of Rack），他们天然充当 RR 节点，这时做法就变成了：
🔴 注：本文中出现的交换机均指硬件上是交换机，同时支持路由功能 3 层交换机。
- 本实验中 VyOS 叫"软路由"，干的活和真实数据中心的"三层交换机"是一样的：既做二层交换又跑 BGP 路由；
- 所以，在 Spine-Leaf 架构里，不要把"交换机"和"路由器"分开理解，它们都是能跑 BGP 的网络设备，只是所处层级不同！
本文实验场景构造，通过 VyOS 模拟真实场景案例，不从 k8s node 选择 RR，直接连接模拟出来的 leaf 路由器：
VyOS 底层是用 FRR 做路由引擎，他只是提供了一个封装层（CLI + 配置管理），核心路由能力来自 FRR。
flowchart TD subgraph SPINE["Spine 核心层"] spine0["spine0"] spine1["spine1"] end subgraph RACK0["Rack 0 · 10.1.5.0/24 · AS 65005"] leaf0["leaf0 路由器\n10.1.5.1"] cp["server1 ↔ control-plane\n10.1.5.10\nPodCIDR: 10.244.0.0/24"] w1["server2 ↔ worker\n10.1.5.11\nPodCIDR: 10.244.1.0/24"] end subgraph RACK1["Rack 1 · 10.1.8.0/24 · AS 65008"] leaf1["leaf1 路由器\n10.1.8.1"] w2["server3 ↔ worker2\n10.1.8.10\nPodCIDR: 10.244.2.0/24"] w3["server4 ↔ worker3\n10.1.8.11\nPodCIDR: 10.244.3.0/24"] end spine0 --- leaf0 spine1 --- leaf0 spine0 --- leaf1 spine1 --- leaf1 leaf0 --- cp leaf0 --- w1 leaf1 --- w2 leaf1 --- w3
flowchart TD subgraph SPINE_BGP["Spine eBGP"] spine0_bgp["spine0 · AS 500"] spine1_bgp["spine1 · AS 800"] end subgraph RACK0_BGP["Rack 0 · AS 65005 iBGP"] leaf0_rr[["leaf0 路由反射器\n10.1.5.1"]] cp_bgp["control-plane\n10.1.5.10"] w1_bgp["worker\n10.1.5.11"] end subgraph RACK1_BGP["Rack 1 · AS 65008 iBGP"] leaf1_rr[["leaf1 路由反射器\n10.1.8.1"]] w2_bgp["worker2\n10.1.8.10"] w3_bgp["worker3\n10.1.8.11"] end spine0_bgp ===|"eBGP"| leaf0_rr spine1_bgp ===|"eBGP"| leaf0_rr spine0_bgp ===|"eBGP"| leaf1_rr spine1_bgp ===|"eBGP"| leaf1_rr leaf0_rr -.->|"iBGP · RR Client\n同 AS 65005"| cp_bgp leaf0_rr -.->|"iBGP · RR Client\n同 AS 65005"| w1_bgp leaf1_rr -.->|"iBGP · RR Client\n同 AS 65008"| w2_bgp leaf1_rr -.->|"iBGP · RR Client\n同 AS 65008"| w3_bgp
Kind 创建的集群所有节点在同一子网中，没有中间路由器，无法模拟真实的跨机架/跨交换机场景，所以需要下面 containerlab 脚本创建额外的容器，模拟路由器。
#!/bin/bash
set -v
# 1. Prepare NoCNI environment
cat << EOF | HTTP_PROXY= HTTPS_PROXY= http_proxy= https_proxy= kind create cluster --name=calico-bgp-rr --image=kindest/node:v1.27.3 --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
disableDefaultCNI: true
podSubnet: "10.244.0.0/16"
nodes:
- role: control-plane
kubeadmConfigPatches:
- |
kind: InitConfiguration
nodeRegistration:
kubeletExtraArgs:
node-ip: 10.1.5.10
node-labels: "rack=rack0"
- role: worker
kubeadmConfigPatches:
- |
kind: JoinConfiguration
nodeRegistration:
kubeletExtraArgs:
node-ip: 10.1.5.11
node-labels: "rack=rack0"
- role: worker
kubeadmConfigPatches:
- |
kind: JoinConfiguration
nodeRegistration:
kubeletExtraArgs:
node-ip: 10.1.8.10
node-labels: "rack=rack1"
- role: worker
kubeadmConfigPatches:
- |
kind: JoinConfiguration
nodeRegistration:
kubeletExtraArgs:
node-ip: 10.1.8.11
node-labels: "rack=rack1"
EOF
# 2. Remove taints
controller_node_ip=`kubectl get node -o wide --no-headers | grep -E "control-plane|bpf1" | awk -F " " '{print $6}'`
kubectl taint nodes $(kubectl get nodes -o name | grep control-plane) node-role.kubernetes.io/control-plane:NoSchedule-
./2-setup-clab.sh
# 3. Collect startup message
controller_node_name=$(kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | grep control-plane)
if [ -n "$controller_node_name" ]; then
timeout 1 docker exec -t $controller_node_name bash -c 'cat << EOF > /root/monitor_startup.sh
#!/bin/bash
ip -ts monitor all > /root/startup_monitor.txt 2>&1
EOF
chmod +x /root/monitor_startup.sh && /root/monitor_startup.sh'
else
echo "No such controller_node!"
fi
# 4. Install calico and enabel bgp configuration
./3-prep-calico-bgp.sh
创建 VyOS（一种虚拟路由器系统）容器模拟物理交换机，并用 Linux 网桥把 Kind 节点和虚拟交换机连接起来，形成一个完整的 Spine-Leaf 网络拓扑。
./2-setup-clab.sh
#!/bin/bash
set -v
for br in br-leaf0 br-leaf1; do
ip link set $br down > /dev/null 2>&1
ip link delete $br
ip link add $br type bridge
ip link set $br up
done
cat << EOF >clab.yaml | containerlab deploy -t clab.yaml -
name: calico-bgp-rr
topology:
nodes:
## 模拟核心层交换机
spine0:
kind: linux
image: burlyluo/vyos:1.4.9
cmd: /sbin/init
binds:
- /lib/modules:/lib/modules
- ./startup-conf/spine0-boot.cfg:/opt/vyatta/etc/config/config.boot
spine1:
kind: linux
image: burlyluo/vyos:1.4.9
cmd: /sbin/init
binds:
- /lib/modules:/lib/modules
- ./startup-conf/spine1-boot.cfg:/opt/vyatta/etc/config/config.boot
## 这两个 leaf 网桥模拟接入层交换机（BGP Route Reflector）
leaf0:
kind: linux
image: burlyluo/vyos:1.4.9
cmd: /sbin/init
binds:
- /lib/modules:/lib/modules
- ./startup-conf/leaf0-boot.cfg:/opt/vyatta/etc/config/config.boot
leaf1:
kind: linux
image: burlyluo/vyos:1.4.9
cmd: /sbin/init
binds:
- /lib/modules:/lib/modules
- ./startup-conf/leaf1-boot.cfg:/opt/vyatta/etc/config/config.boot
## 二层网桥
br-leaf0:
kind: bridge
br-leaf1:
kind: bridge
server1:
kind: linux
image: hub.deepflow.yunshan.net/network-demo/nettool
network-mode: container:calico-bgp-rr-control-plane
exec:
- ip addr add 10.1.5.10/24 dev net0
- ip route replace default via 10.1.5.1
server2:
kind: linux
image: hub.deepflow.yunshan.net/network-demo/nettool
network-mode: container:calico-bgp-rr-worker
exec:
- ip addr add 10.1.5.11/24 dev net0
- ip route replace default via 10.1.5.1
server3:
kind: linux
image: burlyluo/nettool:latest
network-mode: container:calico-bgp-rr-worker2
exec:
- ip addr add 10.1.8.10/24 dev net0
- ip route replace default via 10.1.8.1
server4:
kind: linux
image: burlyluo/nettool:latest
network-mode: container:calico-bgp-rr-worker3
exec:
- ip addr add 10.1.8.11/24 dev net0
- ip route replace default via 10.1.8.1
## 用 links 把所有设备连起来后,Kind 节点流量就必须经过 leaf 交换机才能跨机架通信:
## - server1/server2 → br-leaf0(rack0 的节点连到 rack0 的网桥)
## - server3/server4 → br-leaf1(rack1 的节点连到 rack1 的网桥)
## - leaf0 → br-leaf0(leaf 交换机连到对应网桥)
## - leaf1 → br-leaf1
## - leaf0/leaf1 → spine0/spine1(上联到核心交换机)
links:
- endpoints: ["br-leaf0:br-leaf0-net0", "server1:net0"]
mtu: 1500
- endpoints: ["br-leaf0:br-leaf0-net1", "server2:net0"]
mtu: 1500
- endpoints: ["br-leaf1:br-leaf1-net0", "server3:net0"]
mtu: 1500
- endpoints: ["br-leaf1:br-leaf1-net1", "server4:net0"]
mtu: 1500
- endpoints: ["leaf0:eth1", "spine0:eth1"]
mtu: 1500
- endpoints: ["leaf0:eth2", "spine1:eth1"]
mtu: 1500
- endpoints: ["leaf0:eth3", "br-leaf0:br-leaf0-net2"]
mtu: 1500
- endpoints: ["leaf1:eth1", "spine0:eth2"]
mtu: 1500
- endpoints: ["leaf1:eth2", "spine1:eth2"]
mtu: 1500
- endpoints: ["leaf1:eth3", "br-leaf1:br-leaf1-net2"]
mtu: 1500
EOF
route-reflector-client 是整个实验的核心：它让 leaf 交换机成为 BGP 路由反射器。K8s 节点不需要彼此建立 BGP 会话，只需要和 leaf 交换机建立会话，leaf 会把从一个节点学到的路由 "反射" 给其他节点。
interfaces {
## 通过上方脚本 2 将 leaf0:eth1 连向 spine0:eth1
ethernet eth1 {
address "10.1.10.1/24"
duplex "auto"
mtu "9000"
speed "auto"
}
## 通过上方脚本 2 将 leaf0:eth2 连向 spine1:eth1
ethernet eth2 {
address "10.1.12.1/24"
duplex "auto"
mtu "9000"
speed "auto"
}
## 通过上方脚本 2 将 leaf0:eth3 连向 br-leaf0:br-leaf0-net2
ethernet eth3 {
address "10.1.5.1/24"
duplex "auto"
mtu "9000"
speed "auto"
}
loopback lo {
}
}
## 让容器内的所有内网 IP 能通过 eth0 访问外网
nat {
source {
rule 100 {
outbound-interface {
## eth0 是 containerlab 自动创建的管理口
name "eth0"
}
source {
## 匹配的内网范围,这覆盖了所有内网子网:
## - 10.1.5.0/24(rack0 节点)
## - 10.1.8.0/24(rack1 节点)
## - 10.1.10.0/24、10.1.12.0/24(互联子网)
## - 10.244.0.0/16(Pod 子网,虽然不在 /16 范围内,但 Pod 路由会被 BGP 处理)
address "10.1.0.0/16"
}
translation {
## 动作: SNAT
## 就是把源地址改成 eth0 自己的 IP(宿主机 docker 网络分配的地址),对端看到的来源就是 eth0 的地址,而不是原始的 10.1.x.x 内网地址
## 当源地址是 10.1.0.0/16 且从 eth0 出去的流量,就把源地址伪装成 eth0 的 IP
## 没有这条规则的话,K8s 节点发出的包到达外网时,外网不知道怎么回包到 10.1.x.x 这些私有地址
address "masquerade"
}
}
}
}
protocols {
bgp {
## 自治系统编号
system-as "65005"
address-family {
## 主动宣告: 要去这些网段,把包交给我(leaf0)
ipv4-unicast {
network 10.1.5.0/24 {
}
network 10.1.10.0/24 {
}
network 10.1.12.0/24 {
}
}
}
## 与谁建立 BGP 关系:
## 与同子网 k8s node 建立 iBGP 互联: system-as/remote-as 一致
neighbor 10.1.5.10 {
address-family {
ipv4-unicast {
## 下一跳是我自己
nexthop-self
## 正常情况下 BGP 从邻居学来的路由不会再转发给其他邻居;
## 添加此配置后,leaf0 就变成了路由反射器
route-reflector-client
}
}
remote-as "65005"
}
neighbor 10.1.5.11 {
address-family {
ipv4-unicast {
nexthop-self
route-reflector-client
}
}
remote-as "65005"
}
neighbor 10.1.10.2 {
address-family {
ipv4-unicast
}
remote-as "500"
}
neighbor 10.1.12.2 {
address-family {
ipv4-unicast
}
remote-as "800"
}
parameters {
bestpath {
as-path {
multipath-relax
}
}
router-id "10.1.5.1"
}
}
}
system {
config-management {
commit-revisions "100"
}
console {
device ttyS0 {
speed "9600"
}
}
host-name "leaf0"
login {
user vyos {
authentication {
encrypted-password "$6$QxPS.uk6mfo$9QBSo8u1FkH16gMyAVhus6fU3LOzvLR9Z9.82m3tiHFAxTtIkhaZSWssSgzt4v4dGAL8rhVQxTg0oAG9/q11h/"
plaintext-password ""
}
}
}
time-zone "UTC"
}
interfaces {
ethernet eth1 {
address "10.1.34.1/24"
duplex "auto"
mtu "9000"
speed "auto"
}
ethernet eth2 {
address "10.1.11.1/24"
duplex "auto"
mtu "9000"
speed "auto"
}
ethernet eth3 {
address "10.1.8.1/24"
duplex "auto"
mtu "9000"
speed "auto"
}
loopback lo {
}
}
nat {
source {
rule 100 {
outbound-interface {
name "eth0"
}
source {
address "10.1.0.0/16"
}
translation {
address "masquerade"
}
}
}
}
protocols {
bgp {
address-family {
ipv4-unicast {
network 10.1.8.0/24 {
}
network 10.1.11.0/24 {
}
network 10.1.34.0/24 {
}
}
}
neighbor 10.1.8.10 {
address-family {
ipv4-unicast {
nexthop-self
route-reflector-client
}
}
remote-as "65008"
}
neighbor 10.1.8.11 {
address-family {
ipv4-unicast {
nexthop-self
route-reflector-client
}
}
remote-as "65008"
}
neighbor 10.1.11.2 {
address-family {
ipv4-unicast
}
remote-as "800"
}
neighbor 10.1.34.2 {
address-family {
ipv4-unicast
}
remote-as "500"
}
parameters {
bestpath {
as-path {
multipath-relax
}
}
router-id "10.1.8.1"
}
system-as "65008"
}
}
system {
config-management {
commit-revisions "100"
}
console {
device ttyS0 {
speed "9600"
}
}
host-name "leaf1"
login {
user vyos {
authentication {
encrypted-password "$6$QxPS.uk6mfo$9QBSo8u1FkH16gMyAVhus6fU3LOzvLR9Z9.82m3tiHFAxTtIkhaZSWssSgzt4v4dGAL8rhVQxTg0oAG9/q11h/"
plaintext-password ""
}
}
}
time-zone "UTC"
}
interfaces {
ethernet eth1 {
address "10.1.10.2/24"
duplex "auto"
speed "auto"
}
ethernet eth2 {
address "10.1.34.2/24"
duplex "auto"
speed "auto"
}
loopback lo {
}
}
protocols {
bgp {
address-family {
ipv4-unicast {
network 10.1.10.0/24 {
}
network 10.1.34.0/24 {
}
}
}
neighbor 10.1.10.1 {
address-family {
ipv4-unicast
}
remote-as "65005"
}
neighbor 10.1.34.1 {
address-family {
ipv4-unicast
}
remote-as "65008"
}
parameters {
bestpath {
as-path {
multipath-relax
}
}
}
system-as "500"
}
}
system {
config-management {
commit-revisions "100"
}
console {
device ttyS0 {
speed "9600"
}
}
host-name "spine0"
login {
user vyos {
authentication {
encrypted-password "$6$QxPS.uk6mfo$9QBSo8u1FkH16gMyAVhus6fU3LOzvLR9Z9.82m3tiHFAxTtIkhaZSWssSgzt4v4dGAL8rhVQxTg0oAG9/q11h/"
plaintext-password ""
}
}
}
time-zone "UTC"
}
interfaces {
ethernet eth1 {
address "10.1.12.2/24"
duplex "auto"
mtu "9000"
offload {
gso
sg
}
speed "auto"
}
ethernet eth2 {
address "10.1.11.2/24"
duplex "auto"
mtu "9000"
offload {
gso
sg
}
speed "auto"
}
loopback lo {
}
}
protocols {
bgp {
address-family {
ipv4-unicast {
network 10.1.11.0/24 {
}
network 10.1.12.0/24 {
}
}
}
neighbor 10.1.11.1 {
address-family {
ipv4-unicast
}
remote-as "65008"
}
neighbor 10.1.12.1 {
address-family {
ipv4-unicast
}
remote-as "65005"
}
parameters {
bestpath {
as-path {
multipath-relax
}
}
router-id "10.1.8.1"
}
system-as "800"
}
}
system {
config-management {
commit-revisions "100"
}
conntrack {
modules {
ftp
h323
nfs
pptp
sip
sqlnet
tftp
}
}
console {
device ttyS0 {
speed "9600"
}
}
host-name "spine1"
login {
user vyos {
authentication {
encrypted-password "$6$QxPS.uk6mfo$9QBSo8u1FkH16gMyAVhus6fU3LOzvLR9Z9.82m3tiHFAxTtIkhaZSWssSgzt4v4dGAL8rhVQxTg0oAG9/q11h/"
plaintext-password ""
}
}
}
time-zone "UTC"
}
## https://github.com/projectcalico/calico/blob/v3.31.5/manifests/calico.yaml
# Auto-detect the BGP IP address.
- name: IP
value: "autodetect"
# Enable IPIP
- name: CALICO_IPV4POOL_IPIP
value: "Never"
# Enable or Disable VXLAN on the default IP pool.
- name: CALICO_IPV4POOL_VXLAN
value: "Never"
# Enable or Disable VXLAN on the default IPv6 IP pool.
- name: CALICO_IPV6POOL_VXLAN
value: "Never"
#!/bin/bash
set -v
# 1. Install CNI[Calico v3.23.2]
kubectl apply -f ./calico.yaml
kubectl wait --timeout=100s --for=condition=Ready=true pods --all -A
# 1.2. disable bgp fullmesh
cat << EOF | calicoctl --allow-version-mismatch apply -f -
apiVersion: projectcalico.org/v3
items:
- apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
name: default
spec:
logSeverityScreen: Info
nodeToNodeMeshEnabled: false
kind: BGPConfigurationList
metadata:
EOF
# 1.3. add() bgp configuration for the nodes
cat << EOF | calicoctl --allow-version-mismatch apply -f -
apiVersion: projectcalico.org/v3
kind: Node
metadata:
annotations:
projectcalico.org/kube-labels: '{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"calico-bgp-rr-control-plane","kubernetes.io/os":"linux","node-role.kubernetes.io/control-plane":"","node-role.kubernetes.io/master":"","node.kubernetes.io/exclude-from-external-load-balancers":"","rack":"rack0"}'
labels:
beta.kubernetes.io/arch: amd64
beta.kubernetes.io/os: linux
kubernetes.io/arch: amd64
kubernetes.io/hostname: calico-bgp-rr-control-plane
kubernetes.io/os: linux
node-role.kubernetes.io/control-plane: ""
node-role.kubernetes.io/master: ""
node.kubernetes.io/exclude-from-external-load-balancers: ""
rack: rack0
name: calico-bgp-rr-control-plane
spec:
addresses:
- address: 10.1.5.10
type: InternalIP
bgp:
asNumber: 65005
ipv4Address: 10.1.5.10/24
orchRefs:
- nodeName: calico-bgp-rr-control-plane
orchestrator: k8s
status:
podCIDRs:
- 10.244.0.0/24
EOF
cat << EOF | calicoctl --allow-version-mismatch apply -f -
apiVersion: projectcalico.org/v3
kind: Node
metadata:
annotations:
projectcalico.org/kube-labels: '{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"calico-bgp-rr-worker","kubernetes.io/os":"linux","rack":"rack0"}'
creationTimestamp: "2022-12-05T08:40:29Z"
labels:
beta.kubernetes.io/arch: amd64
beta.kubernetes.io/os: linux
kubernetes.io/arch: amd64
kubernetes.io/hostname: calico-bgp-rr-worker
kubernetes.io/os: linux
rack: rack0
name: calico-bgp-rr-worker
spec:
addresses:
- address: 10.1.5.11
type: InternalIP
bgp:
asNumber: 65005
ipv4Address: 10.1.5.11/24
orchRefs:
- nodeName: calico-bgp-rr-worker
orchestrator: k8s
status:
podCIDRs:
- 10.244.1.0/24
EOF
cat << EOF | calicoctl --allow-version-mismatch apply -f -
apiVersion: projectcalico.org/v3
kind: Node
metadata:
annotations:
projectcalico.org/kube-labels: '{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"calico-bgp-rr-worker2","kubernetes.io/os":"linux","rack":"rack1"}'
creationTimestamp: "2022-12-05T08:40:29Z"
labels:
beta.kubernetes.io/arch: amd64
beta.kubernetes.io/os: linux
kubernetes.io/arch: amd64
kubernetes.io/hostname: calico-bgp-rr-worker2
kubernetes.io/os: linux
rack: rack1
name: calico-bgp-rr-worker2
spec:
addresses:
- address: 10.1.8.10
type: InternalIP
bgp:
asNumber: 65008
ipv4Address: 10.1.8.10/24
orchRefs:
- nodeName: calico-bgp-rr-worker2
orchestrator: k8s
status:
podCIDRs:
- 10.244.2.0/24
EOF
cat << EOF | calicoctl --allow-version-mismatch apply -f -
apiVersion: projectcalico.org/v3
kind: Node
metadata:
annotations:
projectcalico.org/kube-labels: '{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"calico-bgp-rr-worker3","kubernetes.io/os":"linux","rack":"rack1"}'
creationTimestamp: "2022-12-05T08:40:29Z"
labels:
beta.kubernetes.io/arch: amd64
beta.kubernetes.io/os: linux
kubernetes.io/arch: amd64
kubernetes.io/hostname: calico-bgp-rr-worker3
kubernetes.io/os: linux
rack: rack1
name: calico-bgp-rr-worker3
spec:
addresses:
- address: 10.1.8.11
type: InternalIP
bgp:
asNumber: 65008
ipv4Address: 10.1.8.11/24
orchRefs:
- nodeName: calico-bgp-rr-worker3
orchestrator: k8s
status:
podCIDRs:
- 10.244.3.0/24
EOF
# 1.4. peer to leaf0 switch
cat << EOF | calicoctl --allow-version-mismatch apply -f -
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
name: rack0-to-leaf0
spec:
peerIP: 10.1.5.1
asNumber: 65005
nodeSelector: rack == 'rack0'
EOF
# 1.5. peer to leaf1 switch
cat << EOF | calicoctl --allow-version-mismatch apply -f -
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
name: rack1-to-leaf1
spec:
peerIP: 10.1.8.1
asNumber: 65008
nodeSelector: rack == 'rack1'
EOF
本质是 Nginx，用于后续请求抓包使用
apiVersion: apps/v1
kind: StatefulSet
metadata:
labels:
app: nginx
name: pod
spec:
replicas: 4
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- image: burlyluo/nettool:latest
name: nettoolbox
env:
- name: NETTOOL_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
securityContext:
privileged: true
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: nginx
topologyKey: kubernetes.io/hostname
root@network-demo:~# docker ps --format '{{.Names}}'
clab-calico-bgp-rr-server2
clab-calico-bgp-rr-server4
clab-calico-bgp-rr-server1
clab-calico-bgp-rr-server3
clab-calico-bgp-rr-leaf1
clab-calico-bgp-rr-spine1
clab-calico-bgp-rr-leaf0
clab-calico-bgp-rr-spine0
calico-bgp-rr-control-plane
calico-bgp-rr-worker2
calico-bgp-rr-worker3
calico-bgp-rr-worker
## br-leaf0 上的三个接口:
## - br-leaf0-net0: 对应 2.1. 中 link 配置 ["br-leaf0:br-leaf0-net0", "server1:net0"] 即 server1(control-plane，10.1.5.10)
## - br-leaf0-net1: 对应 2.1. 中 link 配置 ["br-leaf0:br-leaf0-net1", "server2:net0"] 即 server2(worker，10.1.5.11)
## - br-leaf0-net2: 对应 2.1. 中 link 配置 ["leaf0:eth3", "br-leaf0:br-leaf0-net2"] 即 leaf0 的 eth3(10.1.5.1)
## br-leaf1 上的三个接口:
## - br-leaf0-net0: 对应 2.1. 中 link 配置 ["br-leaf1:br-leaf0-net0", "server3:net0"] 即 server3(worker2，10.1.8.10)
## - br-leaf0-net1: 对应 2.1. 中 link 配置 ["br-leaf1:br-leaf0-net1", "server4:net0"] 即 server4(worker3，10.1.8.11)
## - br-leaf0-net2: 对应 2.1. 中 link 配置 ["leaf1:eth3", "br-leaf1:br-leaf1-net2"] 即 leaf1 的 eth3(10.1.8.1)
root@network-demo:~# bridge link show | grep -E 'br-leaf0|br-leaf1'
351: br-leaf1-net0@if350: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br-leaf1 state forwarding priority 32 cost 2
353: br-leaf0-net0@if352: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br-leaf0 state forwarding priority 32 cost 2
355: br-leaf1-net1@if354: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br-leaf1 state forwarding priority 32 cost 2
357: br-leaf0-net1@if356: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br-leaf0 state forwarding priority 32 cost 2
359: br-leaf0-net2@if358: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br-leaf0 state forwarding priority 32 cost 2
361: br-leaf1-net2@if360: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br-leaf1 state forwarding priority 32 cost 2
root@network-demo:~# kubectl get pods -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
default pod-0 1/1 Running 0 2m46s 10.244.81.65 calico-bgp-rr-worker
default pod-1 1/1 Running 0 2m40s 10.244.210.64 calico-bgp-rr-worker3
default pod-2 1/1 Running 0 2m34s 10.244.192.0 calico-bgp-rr-worker2
default pod-3 1/1 Running 0 2m27s 10.244.205.68 calico-bgp-rr-control-plane
kube-system calico-kube-controllers 1/1 Running 0 30m 10.244.205.66 calico-bgp-rr-control-plane
kube-system calico-node-55n4m 1/1 Running 0 30m 10.1.5.10 calico-bgp-rr-control-plane
kube-system calico-node-6s6d9 1/1 Running 0 30m 10.1.8.11 calico-bgp-rr-worker3
kube-system calico-node-x6lkz 1/1 Running 0 30m 10.1.5.11 calico-bgp-rr-worker
kube-system calico-node-zc5mp 1/1 Running 0 30m 10.1.8.10 calico-bgp-rr-worker2
kube-system coredns-5d78c9869d-9pxdl 1/1 Running 0 32m 10.244.205.67 calico-bgp-rr-control-plane
kube-system coredns-5d78c9869d-klrjh 1/1 Running 0 32m 10.244.205.64 calico-bgp-rr-control-plane
kube-system etcd-calico-bgp-rr 1/1 Running 0 32m 10.1.5.10 calico-bgp-rr-control-plane
kube-system kube-apiserver-calico-bgp-rr 1/1 Running 0 33m 10.1.5.10 calico-bgp-rr-control-plane
kube-system kube-controller-manager-calico-bgp-rr 1/1 Running 0 32m 10.1.5.10 calico-bgp-rr-control-plane
kube-system kube-proxy-gfs8l 1/1 Running 0 32m 10.1.5.11 calico-bgp-rr-worker
kube-system kube-proxy-kkx2m 1/1 Running 0 32m 10.1.8.11 calico-bgp-rr-worker3
kube-system kube-proxy-kmvpg 1/1 Running 0 32m 10.1.8.10 calico-bgp-rr-worker2
kube-system kube-proxy-zt5wt 1/1 Running 0 32m 10.1.5.10 calico-bgp-rr-control-plane
kube-system kube-scheduler-calico-bgp-rr 1/1 Running 0 32m 10.1.5.10 calico-bgp-rr-control-plane
root@network-demo:~# kubectl describe pods -n kube-system calico-node-55n4m | grep 'CALICO_IPV4POOL'
CALICO_IPV4POOL_IPIP: Never
CALICO_IPV4POOL_VXLAN: Never
root@network-demo:~# kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP
calico-bgp-rr-control-plane Ready control-plane 36m v1.27.3 10.1.5.10
calico-bgp-rr-worker Ready <none> 35m v1.27.3 10.1.5.11
calico-bgp-rr-worker2 Ready <none> 35m v1.27.3 10.1.8.10
calico-bgp-rr-worker3 Ready <none> 35m v1.27.3 10.1.8.11
从以下两条输出看到：
root@calico-bgp-rr-control-plane:/# calicoctl node status
Calico process is running.
IPv4 BGP status
+--------------+---------------+-------+----------+-------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+--------------+---------------+-------+----------+-------------+
| 10.1.5.1 | node specific | up | 11:27:40 | Established |
+--------------+---------------+-------+----------+-------------+
root@calico-bgp-rr-control-plane:/# ss -anp | grep 'bird'
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
tcp LISTEN 0 8 0.0.0.0:179 0.0.0.0:* users:(("bird",pid=2828,fd=7))
tcp ESTAB 0 0 10.1.5.10:44123 10.1.5.1:179 users:(("bird",pid=2828,fd=9))
root@network-demo:~# kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP
calico-bgp-rr-control-plane Ready control-plane 15h v1.27.3 10.1.5.10
calico-bgp-rr-worker Ready <none> 15h v1.27.3 10.1.5.11
calico-bgp-rr-worker2 Ready <none> 15h v1.27.3 10.1.8.10
calico-bgp-rr-worker3 Ready <none> 15h v1.27.3 10.1.8.11
root@network-demo:~# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
pod-0 1/1 Running 0 14h 10.244.81.65 calico-bgp-rr-worker
pod-1 1/1 Running 0 14h 10.244.210.64 calico-bgp-rr-worker3
pod-2 1/1 Running 0 14h 10.244.192.0 calico-bgp-rr-worker2
pod-3 1/1 Running 0 14h 10.244.205.68 calico-bgp-rr-control-plane
root@network-demo:~# kubectl exec -it pod-3 -- curl -s 10.244.81.65
PodName: pod-0 | PodIP: eth0 10.244.81.65/32
root@network-demo:~# kubectl exec -it pod-3 -- ip address show eth0
3: eth0@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether a2:58:b0:10:1f:73 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.244.205.68/32 scope global eth0
valid_lft forever preferred_lft forever
root@network-demo:~# docker exec -it calico-bgp-rr-control-plane ip -d link show cali87081bf6f89
10: cali87081bf6f89@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-4f256a86-723c-ed5f-29f5-9123a3fcc5d0 promiscuity 0 minmtu 68 maxmtu 65535
veth addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
两条命令都是在查路由表，只是展示形式不一样
root@network-demo:~# kubectl exec -it pod-3 -- ip route show
default via 169.254.1.1 dev eth0
169.254.1.1 dev eth0 scope link
root@network-demo:~# kubectl exec -it pod-3 -- route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 169.254.1.1 0.0.0.0 UG 0 0 0 eth0
169.254.1.1 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
由于 Pod 子网掩码是 32，只能发往默认路由。Calico 在主机处开启 proxy_arp 后，由主机对应的 veth pair cali87081bf6f89 响应 Pod 发出的 ARP 广播。
root@network-demo:~# kubectl exec -it pod-3 -- ip neighbor show
169.254.1.1 dev eth0 lladdr ee:ee:ee:ee:ee:ee STALE
172.18.0.2 dev eth0 lladdr ee:ee:ee:ee:ee:ee STALE
root@network-demo:~# kubectl exec -it pod-3 -- arp -n
Address HWtype HWaddress Flags Mask Iface
169.254.1.1 ether ee:ee:ee:ee:ee:ee C eth0
172.18.0.2 ether ee:ee:ee:ee:ee:ee C eth0
root@network-demo:~# kubectl exec -it pod-3 -- tcpdump -pnei eth0
03:44:17.237008 a2:58:b0:10:1f:73 > ee:ee:ee:ee:ee:ee, ethertype IPv4 (0x0800), length 74: 10.244.205.68.36282 > 10.244.81.65.80: Flags [S], seq 911090674, win 64240, options [mss 1460,sackOK,TS val 4079862502 ecr 0,nop,wscale 7], length 0
03:44:17.237245 ee:ee:ee:ee:ee:ee > a2:58:b0:10:1f:73, ethertype IPv4 (0x0800), length 74: 10.244.81.65.80 > 10.244.205.68.36282: Flags [S.], seq 643861606, ack 911090675, win 65160, options [mss 1460,sackOK,TS val 1748110101 ecr 4079862502,nop,wscale 7], length 0
03:44:17.237263 a2:58:b0:10:1f:73 > ee:ee:ee:ee:ee:ee, ethertype IPv4 (0x0800), length 66: 10.244.205.68.36282 > 10.244.81.65.80: Flags [.], ack 1, win 502, options [nop,nop,TS val 4079862503 ecr 1748110101], length 0
03:44:17.237360 a2:58:b0:10:1f:73 > ee:ee:ee:ee:ee:ee, ethertype IPv4 (0x0800), length 142: 10.244.205.68.36282 > 10.244.81.65.80: Flags [P.], seq 1:77, ack 1, win 502, options [nop,nop,TS val 4079862503 ecr 1748110101], length 76: HTTP: GET / HTTP/1.1
03:44:17.237413 ee:ee:ee:ee:ee:ee > a2:58:b0:10:1f:73, ethertype IPv4 (0x0800), length 66: 10.244.81.65.80 > 10.244.205.68.36282: Flags [.], ack 77, win 509, options [nop,nop,TS val 1748110101 ecr 4079862503], length 0
03:44:17.237543 ee:ee:ee:ee:ee:ee > a2:58:b0:10:1f:73, ethertype IPv4 (0x0800), length 302: 10.244.81.65.80 > 10.244.205.68.36282: Flags [P.], seq 1:237, ack 77, win 509, options [nop,nop,TS val 1748110101 ecr 4079862503], length 236: HTTP: HTTP/1.1 200 OK
03:44:17.237563 a2:58:b0:10:1f:73 > ee:ee:ee:ee:ee:ee, ethertype IPv4 (0x0800), length 66: 10.244.205.68.36282 > 10.244.81.65.80: Flags [.], ack 237, win 501, options [nop,nop,TS val 4079862503 ecr 1748110101], length 0
03:44:17.237622 ee:ee:ee:ee:ee:ee > a2:58:b0:10:1f:73, ethertype IPv4 (0x0800), length 111: 10.244.81.65.80 > 10.244.205.68.36282: Flags [P.], seq 237:282, ack 77, win 509, options [nop,nop,TS val 1748110101 ecr 4079862503], length 45: HTTP
03:44:17.237629 a2:58:b0:10:1f:73 > ee:ee:ee:ee:ee:ee, ethertype IPv4 (0x0800), length 66: 10.244.205.68.36282 > 10.244.81.65.80: Flags [.], ack 282, win 501, options [nop,nop,TS val 4079862503 ecr 1748110101], length 0
03:44:17.237738 a2:58:b0:10:1f:73 > ee:ee:ee:ee:ee:ee, ethertype IPv4 (0x0800), length 66: 10.244.205.68.36282 > 10.244.81.65.80: Flags [F.], seq 77, ack 282, win 501, options [nop,nop,TS val 4079862503 ecr 1748110101], length 0
03:44:17.237864 ee:ee:ee:ee:ee:ee > a2:58:b0:10:1f:73, ethertype IPv4 (0x0800), length 66: 10.244.81.65.80 > 10.244.205.68.36282: Flags [F.], seq 282, ack 78, win 509, options [nop,nop,TS val 1748110101 ecr 4079862503], length 0
03:44:17.237871 a2:58:b0:10:1f:73 > ee:ee:ee:ee:ee:ee, ethertype IPv4 (0x0800), length 66: 10.244.205.68.36282 > 10.244.81.65.80: Flags [.], ack 283, win 501, options [nop,nop,TS val 4079862503 ecr 1748110101], length 0
root@network-demo:~# docker exec -it calico-bgp-rr-control-plane ip address show net0
352: net0@if353: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether aa:c1:ab:37:f5:c6 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.1.5.10/24 scope global net0
valid_lft forever preferred_lft forever
root@network-demo:~# docker exec -it calico-bgp-rr-control-plane ip route show
default via 10.1.5.1 dev net0
10.1.5.0/24 dev net0 proto kernel scope link src 10.1.5.10
10.1.8.0/24 via 10.1.5.1 dev net0 proto bird
10.1.10.0/24 via 10.1.5.1 dev net0 proto bird
10.1.11.0/24 via 10.1.5.1 dev net0 proto bird
10.1.12.0/24 via 10.1.5.1 dev net0 proto bird
10.1.34.0/24 via 10.1.5.1 dev net0 proto bird
## 包发往 10.244.81.65 匹配这条路由
10.244.81.64/26 via 10.1.5.11 dev net0 proto bird
10.244.192.0/26 via 10.1.5.1 dev net0 proto bird
10.244.205.64 dev cali331467cb209 scope link
blackhole 10.244.205.64/26 proto bird
10.244.205.65 dev cali44fa7d08008 scope link
10.244.205.66 dev cali445aa579c8d scope link
10.244.205.67 dev cali1b63684dfe1 scope link
10.244.205.68 dev cali87081bf6f89 scope link
10.244.210.64/26 via 10.1.5.1 dev net0 proto bird
172.18.0.0/16 dev eth0 proto kernel scope link src 172.18.0.2
root@network-demo:~# docker exec -it calico-bgp-rr-control-plane route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.1.5.1 0.0.0.0 UG 0 0 0 net0
10.1.5.0 0.0.0.0 255.255.255.0 U 0 0 0 net0
10.1.8.0 10.1.5.1 255.255.255.0 UG 0 0 0 net0
10.1.10.0 10.1.5.1 255.255.255.0 UG 0 0 0 net0
10.1.11.0 10.1.5.1 255.255.255.0 UG 0 0 0 net0
10.1.12.0 10.1.5.1 255.255.255.0 UG 0 0 0 net0
10.1.34.0 10.1.5.1 255.255.255.0 UG 0 0 0 net0
## 包发往 10.244.81.65 匹配这条路由
10.244.81.64 10.1.5.11 255.255.255.192 UG 0 0 0 net0
10.244.192.0 10.1.5.1 255.255.255.192 UG 0 0 0 net0
10.244.205.64 0.0.0.0 255.255.255.255 UH 0 0 0 cali331467cb209
10.244.205.64 0.0.0.0 255.255.255.192 U 0 0 0 *
10.244.205.65 0.0.0.0 255.255.255.255 UH 0 0 0 cali44fa7d08008
10.244.205.66 0.0.0.0 255.255.255.255 UH 0 0 0 cali445aa579c8d
10.244.205.67 0.0.0.0 255.255.255.255 UH 0 0 0 cali1b63684dfe1
10.244.205.68 0.0.0.0 255.255.255.255 UH 0 0 0 cali87081bf6f89
10.244.210.64 10.1.5.1 255.255.255.192 UG 0 0 0 net0
172.18.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
root@network-demo:~# docker exec -it calico-bgp-rr-control-plane ip neighbor show
172.18.0.1 dev eth0 lladdr d2:6a:15:c7:e3:41 STALE
10.244.205.64 dev cali331467cb209 lladdr 6e:15:43:e5:f8:bb REACHABLE
172.18.0.3 dev eth0 lladdr f2:41:9a:5e:9e:33 REACHABLE
10.1.5.1 dev net0 lladdr aa:c1:ab:07:5f:2b STALE
10.244.205.68 dev cali87081bf6f89 lladdr a2:58:b0:10:1f:73 STALE
172.18.0.5 dev eth0 lladdr 66:44:9d:b6:41:99 REACHABLE
10.244.205.66 dev cali445aa579c8d lladdr 8e:98:2e:07:20:64 REACHABLE
10.244.205.65 dev cali44fa7d08008 lladdr 66:2a:ef:95:0d:6b REACHABLE
172.18.0.4 dev eth0 lladdr 2a:9a:ad:ea:a2:56 REACHABLE
10.244.205.67 dev cali1b63684dfe1 lladdr ca:5f:cf:a7:b8:ed REACHABLE
## 获取 10.1.5.11 mac 地址
10.1.5.11 dev net0 lladdr aa:c1:ab:17:02:49 STALE
172:18:0:1::3 dev eth0 lladdr f2:41:9a:5e:9e:33 REACHABLE
172:18:0:1::1 dev eth0 lladdr d2:6a:15:c7:e3:41 router STALE
172:18:0:1::4 dev eth0 lladdr 2a:9a:ad:ea:a2:56 REACHABLE
fe80::d06a:15ff:fec7:e341 dev eth0 lladdr d2:6a:15:c7:e3:41 router STALE
172:18:0:1::5 dev eth0 lladdr 66:44:9d:b6:41:99 REACHABLE
root@network-demo:~# docker exec -it calico-bgp-rr-control-plane arp -n
Address HWtyp HWaddress Flags Mask Iface
172.18.0.1 ether d2:6a:15:c7:e3:41 C eth0
10.244.205.64 ether 6e:15:43:e5:f8:bb C cali331467cb209
172.18.0.3 ether f2:41:9a:5e:9e:33 C eth0
10.1.5.1 ether aa:c1:ab:07:5f:2b C net0
10.244.205.68 ether a2:58:b0:10:1f:73 C cali87081bf6f89
172.18.0.5 ether 66:44:9d:b6:41:99 C eth0
10.244.205.66 ether 8e:98:2e:07:20:64 C cali445aa579c8d
10.244.205.65 ether 66:2a:ef:95:0d:6b C cali44fa7d08008
172.18.0.4 ether 2a:9a:ad:ea:a2:56 C eth0
10.244.205.67 ether ca:5f:cf:a7:b8:ed C cali1b63684dfe1
10.1.5.11 ether aa:c1:ab:17:02:49 C net0
通过 Node 路由表发现，发往同子网的包匹配 10.244.81.64/26 via 10.1.5.11 dev net0 路由，但在 Node veth pair 处抓包时可能会产生疑问：
产生这个疑问，是因为把 calixxx 与 net0 理解为了两个网络命名空间。实际上它们共存于同一网络空间中，通过 ip address show 能同时看到这两个设备就证明了这一点。 既然在同一网络空间内，内核自然可以直接做 IP 转发：收包 --> 查路由表 --> 重写 mac --> 从 net0 发出。通过上面 1.4 Pod 抓包处与本代码块对比，会发现每个包对应的 Sequence number 是一样的，也能说明 IP 层及以上是一样的，只是经过内核转发时 mac 头被重写了。
root@network-demo:~# docker exec -it calico-bgp-rr-control-plane tcpdump -pnei net0
03:44:17.237110 aa:c1:ab:37:f5:c6 > aa:c1:ab:17:02:49, ethertype IPv4 (0x0800), length 74: 10.244.205.68.36282 > 10.244.81.65.80: Flags [S], seq 911090674, win 64240, options [mss 1460,sackOK,TS val 4079862502 ecr 0,nop,wscale 7], length 0
03:44:17.237228 aa:c1:ab:17:02:49 > aa:c1:ab:37:f5:c6, ethertype IPv4 (0x0800), length 74: 10.244.81.65.80 > 10.244.205.68.36282: Flags [S.], seq 643861606, ack 911090675, win 65160, options [mss 1460,sackOK,TS val 1748110101 ecr 4079862502,nop,wscale 7], length 0
03:44:17.237272 aa:c1:ab:37:f5:c6 > aa:c1:ab:17:02:49, ethertype IPv4 (0x0800), length 66: 10.244.205.68.36282 > 10.244.81.65.80: Flags [.], ack 1, win 502, options [nop,nop,TS val 4079862503 ecr 1748110101], length 0
03:44:17.237371 aa:c1:ab:37:f5:c6 > aa:c1:ab:17:02:49, ethertype IPv4 (0x0800), length 142: 10.244.205.68.36282 > 10.244.81.65.80: Flags [P.], seq 1:77, ack 1, win 502, options [nop,nop,TS val 4079862503 ecr 1748110101], length 76: HTTP: GET / HTTP/1.1
03:44:17.237407 aa:c1:ab:17:02:49 > aa:c1:ab:37:f5:c6, ethertype IPv4 (0x0800), length 66: 10.244.81.65.80 > 10.244.205.68.36282: Flags [.], ack 77, win 509, options [nop,nop,TS val 1748110101 ecr 4079862503], length 0
03:44:17.237523 aa:c1:ab:17:02:49 > aa:c1:ab:37:f5:c6, ethertype IPv4 (0x0800), length 302: 10.244.81.65.80 > 10.244.205.68.36282: Flags [P.], seq 1:237, ack 77, win 509, options [nop,nop,TS val 1748110101 ecr 4079862503], length 236: HTTP: HTTP/1.1 200 OK
03:44:17.237573 aa:c1:ab:37:f5:c6 > aa:c1:ab:17:02:49, ethertype IPv4 (0x0800), length 66: 10.244.205.68.36282 > 10.244.81.65.80: Flags [.], ack 237, win 501, options [nop,nop,TS val 4079862503 ecr 1748110101], length 0
03:44:17.237618 aa:c1:ab:17:02:49 > aa:c1:ab:37:f5:c6, ethertype IPv4 (0x0800), length 111: 10.244.81.65.80 > 10.244.205.68.36282: Flags [P.], seq 237:282, ack 77, win 509, options [nop,nop,TS val 1748110101 ecr 4079862503], length 45: HTTP
03:44:17.237634 aa:c1:ab:37:f5:c6 > aa:c1:ab:17:02:49, ethertype IPv4 (0x0800), length 66: 10.244.205.68.36282 > 10.244.81.65.80: Flags [.], ack 282, win 501, options [nop,nop,TS val 4079862503 ecr 1748110101], length 0
03:44:17.237756 aa:c1:ab:37:f5:c6 > aa:c1:ab:17:02:49, ethertype IPv4 (0x0800), length 66: 10.244.205.68.36282 > 10.244.81.65.80: Flags [F.], seq 77, ack 282, win 501, options [nop,nop,TS val 4079862503 ecr 1748110101], length 0
03:44:17.237853 aa:c1:ab:17:02:49 > aa:c1:ab:37:f5:c6, ethertype IPv4 (0x0800), length 66: 10.244.81.65.80 > 10.244.205.68.36282: Flags [F.], seq 282, ack 78, win 509, options [nop,nop,TS val 1748110101 ecr 4079862503], length 0
03:44:17.237884 aa:c1:ab:37:f5:c6 > aa:c1:ab:17:02:49, ethertype IPv4 (0x0800), length 66: 10.244.205.68.36282 > 10.244.81.65.80: Flags [.], ack 283, win 501, options [nop,nop,TS val 4079862503 ecr 1748110101], length 0
发现请求的跨子网 Pod IP 是 10.244.192.0。通常情况下 .0 是网络号，.255 是广播地址，这两个都会保留，但 /32 子网掩码只有一个主机地址，没有网络号和广播地址的概念
root@network-demo:~# kubectl exec -it pod-3 -- curl -s 10.244.192.0
PodName: pod-2 | PodIP: eth0 10.244.192.0/32
其余情况与 3.1 Pod 环节一致，不再重复
见 3.2.1. 不再重复
root@network-demo:~# docker exec -it calico-bgp-rr-control-plane ip route get 10.244.192.0
10.244.192.0 via 10.1.5.1 dev net0 src 10.1.5.10 uid 0
root@network-demo:~# docker exec -it calico-bgp-rr-control-plane route -n
Destination Gateway Genmask Flags Metric Ref Use Iface
10.244.192.0 10.1.5.1 255.255.255.192 UG 0 0 0 net0
root@network-demo:~# docker exec -it calico-bgp-rr-control-plane ip neighbor show 10.1.5.1
10.1.5.1 dev net0 lladdr aa:c1:ab:07:5f:2b DELAY
root@network-demo:~# docker exec -it calico-bgp-rr-control-plane arp -n
Address HWtype HWaddress Flags Mask Iface
10.1.5.1 ether aa:c1:ab:07:5f:2b C net0
root@network-demo:~# docker exec -it calico-bgp-rr-control-plane tcpdump -pnei net0
08:05:45.464437 aa:c1:ab:37:f5:c6 > aa:c1:ab:07:5f:2b, ethertype IPv4 (0x0800), length 74: 10.244.205.68.33414 > 10.244.192.0.80: Flags [S], seq 1634412291, win 64240, options [mss 1460,sackOK,TS val 2864904157 ecr 0,nop,wscale 7], length 0
08:05:45.464612 aa:c1:ab:07:5f:2b > aa:c1:ab:37:f5:c6, ethertype IPv4 (0x0800), length 74: 10.244.192.0.80 > 10.244.205.68.33414: Flags [S.], seq 1423221632, ack 1634412292, win 65160, options [mss 1460,sackOK,TS val 3311054241 ecr 2864904157,nop,wscale 7], length 0
08:05:45.464647 aa:c1:ab:37:f5:c6 > aa:c1:ab:07:5f:2b, ethertype IPv4 (0x0800), length 66: 10.244.205.68.33414 > 10.244.192.0.80: Flags [.], ack 1, win 502, options [nop,nop,TS val 2864904157 ecr 3311054241], length 0
08:05:45.464728 aa:c1:ab:37:f5:c6 > aa:c1:ab:07:5f:2b, ethertype IPv4 (0x0800), length 142: 10.244.205.68.33414 > 10.244.192.0.80: Flags [P.], seq 1:77, ack 1, win 502, options [nop,nop,TS val 2864904157 ecr 3311054241], length 76: HTTP: GET / HTTP/1.1
08:05:45.464770 aa:c1:ab:07:5f:2b > aa:c1:ab:37:f5:c6, ethertype IPv4 (0x0800), length 66: 10.244.192.0.80 > 10.244.205.68.33414: Flags [.], ack 77, win 509, options [nop,nop,TS val 3311054241 ecr 2864904157], length 0
08:05:45.464902 aa:c1:ab:07:5f:2b > aa:c1:ab:37:f5:c6, ethertype IPv4 (0x0800), length 302: 10.244.192.0.80 > 10.244.205.68.33414: Flags [P.], seq 1:237, ack 77, win 509, options [nop,nop,TS val 3311054241 ecr 2864904157], length 236: HTTP: HTTP/1.1 200 OK
08:05:45.464944 aa:c1:ab:37:f5:c6 > aa:c1:ab:07:5f:2b, ethertype IPv4 (0x0800), length 66: 10.244.205.68.33414 > 10.244.192.0.80: Flags [.], ack 237, win 501, options [nop,nop,TS val 2864904157 ecr 3311054241], length 0
08:05:45.465029 aa:c1:ab:07:5f:2b > aa:c1:ab:37:f5:c6, ethertype IPv4 (0x0800), length 111: 10.244.192.0.80 > 10.244.205.68.33414: Flags [P.], seq 237:282, ack 77, win 509, options [nop,nop,TS val 3311054241 ecr 2864904157], length 45: HTTP
08:05:45.465050 aa:c1:ab:37:f5:c6 > aa:c1:ab:07:5f:2b, ethertype IPv4 (0x0800), length 66: 10.244.205.68.33414 > 10.244.192.0.80: Flags [.], ack 282, win 501, options [nop,nop,TS val 2864904157 ecr 3311054241], length 0
08:05:45.465184 aa:c1:ab:37:f5:c6 > aa:c1:ab:07:5f:2b, ethertype IPv4 (0x0800), length 66: 10.244.205.68.33414 > 10.244.192.0.80: Flags [F.], seq 77, ack 282, win 501, options [nop,nop,TS val 2864904158 ecr 3311054241], length 0
08:05:45.465324 aa:c1:ab:07:5f:2b > aa:c1:ab:37:f5:c6, ethertype IPv4 (0x0800), length 66: 10.244.192.0.80 > 10.244.205.68.33414: Flags [F.], seq 282, ack 78, win 509, options [nop,nop,TS val 3311054242 ecr 2864904158], length 0
08:05:45.465357 aa:c1:ab:37:f5:c6 > aa:c1:ab:07:5f:2b, ethertype IPv4 (0x0800), length 66: 10.244.205.68.33414 > 10.244.192.0.80: Flags [.], ack 283, win 501, options [nop,nop,TS val 2864904158 ecr 3311054242], length 0
## 精简掉了一些无用网卡设备
root@network-demo:~# docker exec -it clab-calico-bgp-rr-leaf0 ip address show
3: eth0@if369: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 22:eb:79:e9:64:29 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.20.20.5/24 brd 172.20.20.255 scope global eth0
valid_lft forever preferred_lft forever
358: eth3@if359: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default
link/ether aa:c1:ab:07:5f:2b brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.1.5.1/24 brd 10.1.5.255 scope global eth3
valid_lft forever preferred_lft forever
365: eth2@if366: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default
link/ether aa:c1:ab:ab:7d:ab brd ff:ff:ff:ff:ff:ff link-netnsid 2
inet 10.1.12.1/24 brd 10.1.12.255 scope global eth2
valid_lft forever preferred_lft forever
372: eth1@if373: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default
link/ether aa:c1:ab:fe:1f:43 brd ff:ff:ff:ff:ff:ff link-netnsid 1
inet 10.1.10.1/24 brd 10.1.10.255 scope global eth1
valid_lft forever preferred_lft forever
查询 VyOS 镜像中使用 FRRouting 引擎由 BGP 进程 bgpd 维护的内部路由表（非内核路由表）
root@network-demo:~# docker exec -it clab-calico-bgp-rr-leaf0 vtysh -c "show ip bgp"
BGP table version is 12, local router ID is 10.1.5.1, vrf id 0
Default local pref 100, local AS 65005
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
# Network: 目的网络
# Next Hop: 下一跳地址
# Metric: MED 值(影响入站选路)
# LocPrf: 本地优先级,越高越优先(默认100)
# Weight: Cisco 权重,仅本地有效,越高越优先
# Path: AS 路径,流量经过的自治系统序列
# * --> valid,路由可用
# > --> best,BGP 选出的最优路径,会放入路由表转发
# = --> multipath,等价多路径候选(ECMP)
# i --> internal,通过 iBGP 学到的路由
Network Next Hop Metric LocPrf Weight Path
*> 10.1.5.0/24 0.0.0.0 0 32768 i
*= 10.1.8.0/24 10.1.12.2 0 800 65008 i
*> 10.1.10.2 0 500 65008 i
* 10.1.10.0/24 10.1.10.2 0 0 500 i
*> 0.0.0.0 0 32768 i
*> 10.1.11.0/24 10.1.12.2 0 0 800 i
* 10.1.10.2 0 500 65008 i
* 10.1.12.0/24 10.1.12.2 0 0 800 i
*> 0.0.0.0 0 32768 i
* 10.1.34.0/24 10.1.12.2 0 800 65008 i
*> 10.1.10.2 0 0 500 i
*>i 10.244.81.64/26 10.1.5.11 100 0 i
## 请求匹配这条路由
*> 10.244.192.0/26 10.1.12.2 0 800 65008 i
*= 10.1.10.2 0 500 65008 i
*>i 10.244.205.64/26 10.1.5.10 100 0 i
*> 10.244.210.64/26 10.1.12.2 0 800 65008 i
*= 10.1.10.2 0 500 65008 i
在内核路由表中查询到的 BGP 路由，都是通过上面 show ip bgp 看到的 BGP 进程维护的"候选池"挑选出来的最优路由：
root@network-demo:~# docker exec -it clab-calico-bgp-rr-leaf0 ip route show
default via 172.20.20.1 dev eth0
10.1.5.0/24 dev eth3 proto kernel scope link src 10.1.5.1
10.1.8.0/24 nhid 31 proto bgp metric 20
nexthop via 10.1.10.2 dev eth1 weight 1
nexthop via 10.1.12.2 dev eth2 weight 1
10.1.10.0/24 dev eth1 proto kernel scope link src 10.1.10.1
10.1.11.0/24 nhid 27 via 10.1.12.2 dev eth2 proto bgp metric 20
10.1.12.0/24 dev eth2 proto kernel scope link src 10.1.12.1
10.1.34.0/24 nhid 24 via 10.1.10.2 dev eth1 proto bgp metric 20
10.244.81.64/26 nhid 35 via 10.1.5.11 dev eth3 proto bgp metric 20
## 请求匹配这条路由
10.244.192.0/26 nhid 31 proto bgp metric 20
nexthop via 10.1.10.2 dev eth1 weight 1
nexthop via 10.1.12.2 dev eth2 weight 1
10.244.205.64/26 nhid 33 via 10.1.5.10 dev eth3 proto bgp metric 20
10.244.210.64/26 nhid 31 proto bgp metric 20
nexthop via 10.1.10.2 dev eth1 weight 1
nexthop via 10.1.12.2 dev eth2 weight 1
172.20.20.0/24 dev eth0 proto kernel scope link src 172.20.20.5
root@network-demo:~# docker exec -it clab-calico-bgp-rr-leaf0 ip route show proto bgp
10.1.8.0/24 nhid 31 metric 20
nexthop via 10.1.10.2 dev eth1 weight 1
nexthop via 10.1.12.2 dev eth2 weight 1
10.1.11.0/24 nhid 27 via 10.1.12.2 dev eth2 metric 20
10.1.34.0/24 nhid 24 via 10.1.10.2 dev eth1 metric 20
10.244.81.64/26 nhid 35 via 10.1.5.11 dev eth3 metric 20
## 请求匹配这条路由
10.244.192.0/26 nhid 31 metric 20
nexthop via 10.1.10.2 dev eth1 weight 1
nexthop via 10.1.12.2 dev eth2 weight 1
10.244.205.64/26 nhid 33 via 10.1.5.10 dev eth3 metric 20
10.244.210.64/26 nhid 31 metric 20
nexthop via 10.1.10.2 dev eth1 weight 1
nexthop via 10.1.12.2 dev eth2 weight 1
root@network-demo:~# docker exec -it clab-calico-bgp-rr-leaf0 ip neighbor show
10.1.5.10 dev eth3 lladdr aa:c1:ab:37:f5:c6 STALE
10.1.10.2 dev eth1 lladdr aa:c1:ab:2e:d9:95 STALE
172.20.20.1 dev eth0 lladdr 06:2c:25:d4:d7:49 STALE
10.1.12.2 dev eth2 lladdr aa:c1:ab:46:7b:1f STALE
10.1.5.11 dev eth3 lladdr aa:c1:ab:17:02:49 REACHABLE
请求 10.244.192.0 通过 eth3 到达 leaf0 后，查询本机路由表，匹配这条路由：
10.244.192.0/26 nhid 31 metric 20
nexthop via 10.1.10.2 dev eth1 weight 1
nexthop via 10.1.12.2 dev eth2 weight 1
看到这条路由可能会有个疑问，为什么 leaf0 中看到的是 10.244.192.0/26，而 Pod eth0 中显示的是 10.244.192.0/32 呢？
这是因为 Calico 本身的路由聚合行为。每个 Pod 拿到 /32 位独立地址，由 Calico vRouter 维护。这里的 "vRouter" 并不是某个程序，在官网描述中被解释为每个节点都充当为其上所有端点的路由器，所以它本质上应该是 Linux 内核路由 + Felix + bird 的组合。
Calico 节点对外通告路由时，不会把每个 /32 都通过 BGP 发出去，而是把同一个 Node 上的 Pod IP 聚合成 /26 网段统一通告。这样 leaf0 收到的就是一条 /26 掩码的网段，而不是最多 64 条 /32 掩码的 IP，这样大幅缩减路由表规模。
其中，聚合的 /26 网段来自 Calico IP Pool 的默认配置
root@network-demo:~# docker exec -it calico-bgp-rr-control-plane calicoctl get ippool default-ipv4-ippool -o yaml
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
name: default-ipv4-ippool
spec:
allowedUses:
- Workload
- Tunnel
assignmentMode: Automatic
## 表示每个 Node 分配一个 /26 的 IP 块(64 个地址)
blockSize: 26
## 整个 Pod IP 池共 65536 个地址
cidr: 10.244.0.0/16
ipipMode: Never
natOutgoing: true
nodeSelector: all()
vxlanMode: Never
## 显示每个节点实际分配到的 /26 IP 块,包括每个块的使用情况: 总数/已用/空闲
root@network-demo:~# docker exec -it calico-bgp-rr-control-plane calicoctl ipam show --show-blocks
+----------+------------------+-----------+------------+--------------+
| GROUPING | CIDR | IPS TOTAL | IPS IN USE | IPS FREE |
+----------+------------------+-----------+------------+--------------+
| IP Pool | 10.244.0.0/16 | 65536 | 8 (0%) | 65528 (100%) |
| Block | 10.244.192.0/26 | 64 | 1 (2%) | 63 (98%) |
| Block | 10.244.205.64/26 | 64 | 5 (8%) | 59 (92%) |
| Block | 10.244.210.64/26 | 64 | 1 (2%) | 63 (98%) |
| Block | 10.244.81.64/26 | 64 | 1 (2%) | 63 (98%) |
+----------+------------------+-----------+------------+--------------+
由于配置了 ECMP，有两条下一跳路由且权重均为 1，判断后选择其中一条转发，通过 ARP 确认对应 MAC 地址
10.1.10.2 dev eth1 lladdr aa:c1:ab:2e:d9:95 STALE
10.1.12.2 dev eth2 lladdr aa:c1:ab:46:7b:1f STALE
另外需要注意的是，ECMP 路由是内核基于哈希进行路由，而不是 Round-Robin 这种传统意义上的轮询方式，所以他的行为受内核参数控制。在前文中也提到过 multipath 等价多路径候选(ECMP)，内核网络参数中对应 fib_multipath_hash_policy，通过这个参数控制 hash 范围：
所以在本次测试场景中，Pod-3 访问多少次 Pod-2，网络层源/目的 IP 是一样的，hash 值自然也相同，所以流量会一直走 via 10.1.12.2 通过 eth2 发出：
eth3 是接收从 control-plane 节点 net0 发来请求的设备，在此网卡抓包，看到的 src/dst MAC 自然是这两个设备的。至于 eth3 如何把包发给 eth2，这点逻辑与 3.2.4 中 veth pair 设备发给 net0 一样，都是在统一网络空间中，直接改了下 MAC 信息做 IP 转发了
root@network-demo:~# docker exec -it clab-calico-bgp-rr-leaf0 tcpdump -pnei eth3
08:05:45.464466 aa:c1:ab:37:f5:c6 > aa:c1:ab:07:5f:2b, ethertype IPv4 (0x0800), length 74: 10.244.205.68.33414 > 10.244.192.0.80: Flags [S], seq 1634412291, win 64240, options [mss 1460,sackOK,TS val 2864904157 ecr 0,nop,wscale 7], length 0
08:05:45.464607 aa:c1:ab:07:5f:2b > aa:c1:ab:37:f5:c6, ethertype IPv4 (0x0800), length 74: 10.244.192.0.80 > 10.244.205.68.33414: Flags [S.], seq 1423221632, ack 1634412292, win 65160, options [mss 1460,sackOK,TS val 3311054241 ecr 2864904157,nop,wscale 7], length 0
08:05:45.464651 aa:c1:ab:37:f5:c6 > aa:c1:ab:07:5f:2b, ethertype IPv4 (0x0800), length 66: 10.244.205.68.33414 > 10.244.192.0.80: Flags [.], ack 1, win 502, options [nop,nop,TS val 2864904157 ecr 3311054241], length 0
08:05:45.464732 aa:c1:ab:37:f5:c6 > aa:c1:ab:07:5f:2b, ethertype IPv4 (0x0800), length 142: 10.244.205.68.33414 > 10.244.192.0.80: Flags [P.], seq 1:77, ack 1, win 502, options [nop,nop,TS val 2864904157 ecr 3311054241], length 76: HTTP: GET / HTTP/1.1
08:05:45.464767 aa:c1:ab:07:5f:2b > aa:c1:ab:37:f5:c6, ethertype IPv4 (0x0800), length 66: 10.244.192.0.80 > 10.244.205.68.33414: Flags [.], ack 77, win 509, options [nop,nop,TS val 3311054241 ecr 2864904157], length 0
08:05:45.464894 aa:c1:ab:07:5f:2b > aa:c1:ab:37:f5:c6, ethertype IPv4 (0x0800), length 302: 10.244.192.0.80 > 10.244.205.68.33414: Flags [P.], seq 1:237, ack 77, win 509, options [nop,nop,TS val 3311054241 ecr 2864904157], length 236: HTTP: HTTP/1.1 200 OK
08:05:45.464949 aa:c1:ab:37:f5:c6 > aa:c1:ab:07:5f:2b, ethertype IPv4 (0x0800), length 66: 10.244.205.68.33414 > 10.244.192.0.80: Flags [.], ack 237, win 501, options [nop,nop,TS val 2864904157 ecr 3311054241], length 0
08:05:45.465024 aa:c1:ab:07:5f:2b > aa:c1:ab:37:f5:c6, ethertype IPv4 (0x0800), length 111: 10.244.192.0.80 > 10.244.205.68.33414: Flags [P.], seq 237:282, ack 77, win 509, options [nop,nop,TS val 3311054241 ecr 2864904157], length 45: HTTP
08:05:45.465054 aa:c1:ab:37:f5:c6 > aa:c1:ab:07:5f:2b, ethertype IPv4 (0x0800), length 66: 10.244.205.68.33414 > 10.244.192.0.80: Flags [.], ack 282, win 501, options [nop,nop,TS val 2864904157 ecr 3311054241], length 0
08:05:45.465192 aa:c1:ab:37:f5:c6 > aa:c1:ab:07:5f:2b, ethertype IPv4 (0x0800), length 66: 10.244.205.68.33414 > 10.244.192.0.80: Flags [F.], seq 77, ack 282, win 501, options [nop,nop,TS val 2864904158 ecr 3311054241], length 0
08:05:45.465317 aa:c1:ab:07:5f:2b > aa:c1:ab:37:f5:c6, ethertype IPv4 (0x0800), length 66: 10.244.192.0.80 > 10.244.205.68.33414: Flags [F.], seq 282, ack 78, win 509, options [nop,nop,TS val 3311054242 ecr 2864904158], length 0
08:05:45.465361 aa:c1:ab:37:f5:c6 > aa:c1:ab:07:5f:2b, ethertype IPv4 (0x0800), length 66: 10.244.205.68.33414 > 10.244.192.0.80: Flags [.], ack 283, win 501, options [nop,nop,TS val 2864904158 ecr 3311054242], length 0
本机 eth2 MAC 为 aa:c1🆎ab:7d:ab，发往 via 10.1.12.2（spine1 eth1）MAC 为 aa:c1🆎46:7b:1f。通过抓包验证，结果没问题。
root@network-demo:~# docker exec -it clab-calico-bgp-rr-leaf0 tcpdump -pnei eth2
13:22:36.099229 aa:c1:ab:ab:7d:ab > aa:c1:ab:46:7b:1f, ethertype IPv4 (0x0800), length 74: 10.244.205.68.34734 > 10.244.192.0.80: Flags [S], seq 1406055769, win 64240, options [mss 1460,sackOK,TS val 2883914792 ecr 0,nop,wscale 7], length 0
13:22:36.099335 aa:c1:ab:46:7b:1f > aa:c1:ab:ab:7d:ab, ethertype IPv4 (0x0800), length 74: 10.244.192.0.80 > 10.244.205.68.34734: Flags [S.], seq 3189099348, ack 1406055770, win 65160, options [mss 1460,sackOK,TS val 3330064876 ecr 2883914792,nop,wscale 7], length 0
13:22:36.099367 aa:c1:ab:ab:7d:ab > aa:c1:ab:46:7b:1f, ethertype IPv4 (0x0800), length 66: 10.244.205.68.34734 > 10.244.192.0.80: Flags [.], ack 1, win 502, options [nop,nop,TS val 2883914792 ecr 3330064876], length 0
13:22:36.099432 aa:c1:ab:ab:7d:ab > aa:c1:ab:46:7b:1f, ethertype IPv4 (0x0800), length 142: 10.244.205.68.34734 > 10.244.192.0.80: Flags [P.], seq 1:77, ack 1, win 502, options [nop,nop,TS val 2883914792 ecr 3330064876], length 76: HTTP: GET / HTTP/1.1
13:22:36.099476 aa:c1:ab:46:7b:1f > aa:c1:ab:ab:7d:ab, ethertype IPv4 (0x0800), length 66: 10.244.192.0.80 > 10.244.205.68.34734: Flags [.], ack 77, win 509, options [nop,nop,TS val 3330064876 ecr 2883914792], length 0
13:22:36.099610 aa:c1:ab:46:7b:1f > aa:c1:ab:ab:7d:ab, ethertype IPv4 (0x0800), length 302: 10.244.192.0.80 > 10.244.205.68.34734: Flags [P.], seq 1:237, ack 77, win 509, options [nop,nop,TS val 3330064876 ecr 2883914792], length 236: HTTP: HTTP/1.1 200 OK
13:22:36.099675 aa:c1:ab:ab:7d:ab > aa:c1:ab:46:7b:1f, ethertype IPv4 (0x0800), length 66: 10.244.205.68.34734 > 10.244.192.0.80: Flags [.], ack 237, win 501, options [nop,nop,TS val 2883914792 ecr 3330064876], length 0
13:22:36.099736 aa:c1:ab:46:7b:1f > aa:c1:ab:ab:7d:ab, ethertype IPv4 (0x0800), length 111: 10.244.192.0.80 > 10.244.205.68.34734: Flags [P.], seq 237:282, ack 77, win 509, options [nop,nop,TS val 3330064876 ecr 2883914792], length 45: HTTP
13:22:36.099766 aa:c1:ab:ab:7d:ab > aa:c1:ab:46:7b:1f, ethertype IPv4 (0x0800), length 66: 10.244.205.68.34734 > 10.244.192.0.80: Flags [.], ack 282, win 501, options [nop,nop,TS val 2883914792 ecr 3330064876], length 0
13:22:36.099884 aa:c1:ab:ab:7d:ab > aa:c1:ab:46:7b:1f, ethertype IPv4 (0x0800), length 66: 10.244.205.68.34734 > 10.244.192.0.80: Flags [F.], seq 77, ack 282, win 501, options [nop,nop,TS val 2883914792 ecr 3330064876], length 0
13:22:36.100008 aa:c1:ab:46:7b:1f > aa:c1:ab:ab:7d:ab, ethertype IPv4 (0x0800), length 66: 10.244.192.0.80 > 10.244.205.68.34734: Flags [F.], seq 282, ack 78, win 509, options [nop,nop,TS val 3330064876 ecr 2883914792], length 0
13:22:36.100050 aa:c1:ab:ab:7d:ab > aa:c1:ab:46:7b:1f, ethertype IPv4 (0x0800), length 66: 10.244.205.68.34734 > 10.244.192.0.80: Flags [.], ack 283, win 501, options [nop,nop,TS val 2883914792 ecr 3330064876], length 0
## 精简掉了一些无用网卡设备
root@network-demo:~# docker exec -it clab-calico-bgp-rr-spine1 ip address show
3: eth0@if362: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether b6:9e:a1:4b:4f:88 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.20.20.2/24 brd 172.20.20.255 scope global eth0
valid_lft forever preferred_lft forever
366: eth1@if365: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default
link/ether aa:c1:ab:46:7b:1f brd ff:ff:ff:ff:ff:ff link-netnsid 2
inet 10.1.12.2/24 brd 10.1.12.255 scope global eth1
valid_lft forever preferred_lft forever
368: eth2@if367: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default
link/ether aa:c1:ab:de:f1:bf brd ff:ff:ff:ff:ff:ff link-netnsid 1
inet 10.1.11.2/24 brd 10.1.11.255 scope global eth2
valid_lft forever preferred_lft forever
root@network-demo:~# docker exec -it clab-calico-bgp-rr-spine1 vtysh -c "show ip bgp"
BGP table version is 20, local router ID is 10.1.8.1, vrf id 0
Default local pref 100, local AS 800
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 10.1.5.0/24 10.1.12.1 0 0 65005 i
* 10.1.11.1 0 65008 500 65005 i
* 10.1.8.0/24 10.1.12.1 0 65005 500 65008 i
*> 10.1.11.1 0 0 65008 i
*> 10.1.10.0/24 10.1.12.1 0 0 65005 i
* 10.1.11.1 0 65008 500 i
* 10.1.11.0/24 10.1.11.1 0 0 65008 i
*> 0.0.0.0 0 32768 i
* 10.1.12.0/24 10.1.12.1 0 0 65005 i
*> 0.0.0.0 0 32768 i
* 10.1.34.0/24 10.1.12.1 0 65005 500 i
*> 10.1.11.1 0 0 65008 i
*> 10.244.81.64/26 10.1.12.1 0 65005 i
* 10.1.11.1 0 65008 500 65005 i
* 10.244.192.0/26 10.1.12.1 0 65005 500 65008 i
*> 10.1.11.1 0 65008 i
*> 10.244.205.64/26 10.1.12.1 0 65005 i
* 10.1.11.1 0 65008 500 65005 i
* 10.244.210.64/26 10.1.12.1 0 65005 500 65008 i
*> 10.1.11.1 0 65008 i
root@network-demo:~# docker exec -it clab-calico-bgp-rr-spine1 ip route show
default via 172.20.20.1 dev eth0
10.1.5.0/24 nhid 34 via 10.1.12.1 dev eth1 proto bgp metric 20
10.1.8.0/24 nhid 22 via 10.1.11.1 dev eth2 proto bgp metric 20
10.1.10.0/24 nhid 34 via 10.1.12.1 dev eth1 proto bgp metric 20
10.1.11.0/24 dev eth2 proto kernel scope link src 10.1.11.2
10.1.12.0/24 dev eth1 proto kernel scope link src 10.1.12.2
10.1.34.0/24 nhid 22 via 10.1.11.1 dev eth2 proto bgp metric 20
10.244.81.64/26 nhid 34 via 10.1.12.1 dev eth1 proto bgp metric 20
## 请求匹配这条路由
10.244.192.0/26 nhid 22 via 10.1.11.1 dev eth2 proto bgp metric 20
10.244.205.64/26 nhid 34 via 10.1.12.1 dev eth1 proto bgp metric 20
10.244.210.64/26 nhid 22 via 10.1.11.1 dev eth2 proto bgp metric 20
172.20.20.0/24 dev eth0 proto kernel scope link src 172.20.20.2
root@network-demo:~# docker exec -it clab-calico-bgp-rr-spine1 ip route show proto bgp
10.1.5.0/24 nhid 34 via 10.1.12.1 dev eth1 metric 20
10.1.8.0/24 nhid 22 via 10.1.11.1 dev eth2 metric 20
10.1.10.0/24 nhid 34 via 10.1.12.1 dev eth1 metric 20
10.1.34.0/24 nhid 22 via 10.1.11.1 dev eth2 metric 20
10.244.81.64/26 nhid 34 via 10.1.12.1 dev eth1 metric 20
## 请求匹配这条路由
10.244.192.0/26 nhid 22 via 10.1.11.1 dev eth2 metric 20
10.244.205.64/26 nhid 34 via 10.1.12.1 dev eth1 metric 20
10.244.210.64/26 nhid 22 via 10.1.11.1 dev eth2 metric 20
root@network-demo:~# docker exec -it clab-calico-bgp-rr-spine1 route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 172.20.20.1 0.0.0.0 UG 0 0 0 eth0
10.1.5.0 10.1.12.1 255.255.255.0 UG 20 0 0 eth1
10.1.8.0 10.1.11.1 255.255.255.0 UG 20 0 0 eth2
10.1.10.0 10.1.12.1 255.255.255.0 UG 20 0 0 eth1
10.1.11.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2
10.1.12.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
10.1.34.0 10.1.11.1 255.255.255.0 UG 20 0 0 eth2
10.244.81.64 10.1.12.1 255.255.255.192 UG 20 0 0 eth1
## 请求匹配这条路由
10.244.192.0 10.1.11.1 255.255.255.192 UG 20 0 0 eth2
10.244.205.64 10.1.12.1 255.255.255.192 UG 20 0 0 eth1
10.244.210.64 10.1.11.1 255.255.255.192 UG 20 0 0 eth2
172.20.20.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
root@network-demo:~# docker exec -it clab-calico-bgp-rr-spine1 ip neighbor show
10.1.11.1 dev eth2 lladdr aa:c1:ab:59:95:57 REACHABLE
172.20.20.1 dev eth0 lladdr 06:2c:25:d4:d7:49 STALE
10.1.12.1 dev eth1 lladdr aa:c1:ab:ab:7d:ab STALE
root@network-demo:~# docker exec -it clab-calico-bgp-rr-spine1 arp -n
Address HWtype HWaddress Flags Mask Iface
10.1.11.1 ether aa:c1:ab:59:95:57 C eth2
172.20.20.1 ether 06:2c:25:d4:d7:49 C eth0
10.1.12.1 ether aa:c1:ab:ab:7d:ab C eth1
文章 Client 处 Node calixxx --> net0 以及 leaf 处 eth3 --> eth2 已经验证两次同网络空间中，包进来后匹配到出去的路由时，内核直接把包改了个 MAC 信息，做 IP 转发了。这里就不验证 spine 的 eth1 --> eth2 了，直接在 eth2 处抓包：
经过抓包验证，源 MAC 地址为 spine1 eth2，目的 MAC 为 leaf1 eth2，环境部署时通过 link 配置 "leaf1:eth2", "spine1:eth2" 绑定。
root@network-demo:~# docker exec -it clab-calico-bgp-rr-spine1 tcpdump -pnei eth2
13:41:56.783379 aa:c1:ab:de:f1:bf > aa:c1:ab:59:95:57, ethertype IPv4 (0x0800), length 74: 10.244.205.68.45912 > 10.244.192.0.80: Flags [S], seq 1279531781, win 64240, options [mss 1460,sackOK,TS val 2885075476 ecr 0,nop,wscale 7], length 0
13:41:56.783473 aa:c1:ab:59:95:57 > aa:c1:ab:de:f1:bf, ethertype IPv4 (0x0800), length 74: 10.244.192.0.80 > 10.244.205.68.45912: Flags [S.], seq 787129943, ack 1279531782, win 65160, options [mss 1460,sackOK,TS val 3331225560 ecr 2885075476,nop,wscale 7], length 0
13:41:56.783510 aa:c1:ab:de:f1:bf > aa:c1:ab:59:95:57, ethertype IPv4 (0x0800), length 66: 10.244.205.68.45912 > 10.244.192.0.80: Flags [.], ack 1, win 502, options [nop,nop,TS val 2885075476 ecr 3331225560], length 0
13:41:56.783577 aa:c1:ab:de:f1:bf > aa:c1:ab:59:95:57, ethertype IPv4 (0x0800), length 142: 10.244.205.68.45912 > 10.244.192.0.80: Flags [P.], seq 1:77, ack 1, win 502, options [nop,nop,TS val 2885075476 ecr 3331225560], length 76: HTTP: GET / HTTP/1.1
13:41:56.783606 aa:c1:ab:59:95:57 > aa:c1:ab:de:f1:bf, ethertype IPv4 (0x0800), length 66: 10.244.192.0.80 > 10.244.205.68.45912: Flags [.], ack 77, win 509, options [nop,nop,TS val 3331225560 ecr 2885075476], length 0
13:41:56.783707 aa:c1:ab:59:95:57 > aa:c1:ab:de:f1:bf, ethertype IPv4 (0x0800), length 302: 10.244.192.0.80 > 10.244.205.68.45912: Flags [P.], seq 1:237, ack 77, win 509, options [nop,nop,TS val 3331225560 ecr 2885075476], length 236: HTTP: HTTP/1.1 200 OK
13:41:56.783770 aa:c1:ab:de:f1:bf > aa:c1:ab:59:95:57, ethertype IPv4 (0x0800), length 66: 10.244.205.68.45912 > 10.244.192.0.80: Flags [.], ack 237, win 501, options [nop,nop,TS val 2885075476 ecr 3331225560], length 0
13:41:56.783809 aa:c1:ab:59:95:57 > aa:c1:ab:de:f1:bf, ethertype IPv4 (0x0800), length 111: 10.244.192.0.80 > 10.244.205.68.45912: Flags [P.], seq 237:282, ack 77, win 509, options [nop,nop,TS val 3331225560 ecr 2885075476], length 45: HTTP
13:41:56.783834 aa:c1:ab:de:f1:bf > aa:c1:ab:59:95:57, ethertype IPv4 (0x0800), length 66: 10.244.205.68.45912 > 10.244.192.0.80: Flags [.], ack 282, win 501, options [nop,nop,TS val 2885075476 ecr 3331225560], length 0
13:41:56.783966 aa:c1:ab:de:f1:bf > aa:c1:ab:59:95:57, ethertype IPv4 (0x0800), length 66: 10.244.205.68.45912 > 10.244.192.0.80: Flags [F.], seq 77, ack 282, win 501, options [nop,nop,TS val 2885075476 ecr 3331225560], length 0
13:41:56.784100 aa:c1:ab:59:95:57 > aa:c1:ab:de:f1:bf, ethertype IPv4 (0x0800), length 66: 10.244.192.0.80 > 10.244.205.68.45912: Flags [F.], seq 282, ack 78, win 509, options [nop,nop,TS val 3331225560 ecr 2885075476], length 0
13:41:56.784169 aa:c1:ab:de:f1:bf > aa:c1:ab:59:95:57, ethertype IPv4 (0x0800), length 66: 10.244.205.68.45912 > 10.244.192.0.80: Flags [.], ack 283, win 501, options [nop,nop,TS val 2885075477 ecr 3331225560], length 0
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。