Multus 项目地址：https://github.com/k8snetworkplumbingwg/multus-cni
Whereabouts 项目地址：https://github.com/k8snetworkplumbingwg/whereabouts
CNI（容器网络接口）是云原生计算基金会（CNCF）的一个项目，它包含一套规范及开发库。通过 CNI 编写网络插件用于配置容器网络，同时 CNI 本身也供了一系列现成的插件。CNI 仅专注于容器的网络连通性，以及在容器被删除时清理已分配的网络资源。正是由于这种专注性，它获得了广泛的支持和易于实现的规范。
IPVLAN 就是 CNI 自带的网络插件之一，它也对宿主机的网络接口进行了虚拟化，所有 IPVLAN 设备共享同一个 MAC 地址。
# 在 ns0 与 ns1 两个网络空间中,他们的网卡设备 ipvl0/ipvl1 ip 不一致,但 MAC 地址均与 host1 主机 eth0 相同
+=============================================================+
| Host: host1 |
| |
| +----------------------+ +----------------------+ |
| | NS:ns0 | | NS:ns1 | |
| | | | | |
| | | | | |
| | ipvl0 | | ipvl1 | |
| +----------#-----------+ +-----------#----------+ |
| # # |
| ################################ |
| # |
| # |
| # eth0 |
+=============================================================+
Multus CNI 允许将多个网络接口连接到 K8s Pod 中。在安装他之前，K8s 集群必须安装一个默认的 CNI 插件，例如 Flannel、Calico 等。
#!/bin/bash
set -v
# 1.kind 创建集群时关闭默认 CNI 部署
cat << EOF | HTTP_PROXY= HTTPS_PROXY= http_proxy= https_proxy= kind create cluster --name=cni-multus --image=kindest/node:v1.27.3 --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
disableDefaultCNI: true
nodes:
- role: control-plane
- role: worker
- role: worker
EOF
# 2.remove taints
kubectl taint nodes $(kubectl get nodes -o name | grep control-plane) node-role.kubernetes.io/control-plane:NoSchedule-
# 3. install CNI
kubectl apply -f ./k8snetworkplumbingwg
部署 Calico v3.31.5 示例（IPIP）作为集群默认 CNI：
# Cluster type to identify the deployment type
- name: CLUSTER_TYPE
value: "k8s,bgp"
# Auto-detect the BGP IP address.
- name: IP
value: "autodetect"
# Enable IPIP
- name: CALICO_IPV4POOL_IPIP
value: "Always"
# Enable or Disable VXLAN on the default IP pool.
- name: CALICO_IPV4POOL_VXLAN
value: "Never"
# Enable or Disable VXLAN on the default IPv6 IP pool.
- name: CALICO_IPV6POOL_VXLAN
value: "Never"
使用"厚插件（Thick）"形式部署 Multus。与薄插件（Thin）两种部署方式对比：
shim 和 daemon。日志集中在 daemon 中统一管理、且通过 daemon 暴露 Metrics 指标。
┌─────────┐ ┌───────┐ ┌────────┐ ┌──────────┐
│ │ cni ADD/DEL │ │ REST POST │ │ cni ADD/DEL │ │
│ runtime ├────────────►│ shim │===========│ daemon ├────────────►│ delegate │
│ │<------------│ │ │ │<------------│ │
└─────────┘ └───────┘ └────────┘ └──────────┘
按照上面引用的 GitHub 仓库地址中提供的部署方式部署即可。
Multus CNI 的 IPAM 地址管理插件。按照 GitHub 仓库中提供的一键部署方式部署即可
root@network-demo:~# kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP
cni-multus-control-plane Ready control-plane 4m56s v1.27.3 172.18.0.4
cni-multus-worker Ready <none> 4m31s v1.27.3 172.18.0.3
cni-multus-worker2 Ready <none> 4m33s v1.27.3 172.18.0.2
root@network-demo:~# kubectl get pods -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
kube-system calico-kube-controllers-547c488cf5 1/1 Running 0 4m31s 10.244.226.132 cni-multus-control-plane
kube-system calico-node-5lgrw 1/1 Running 0 4m31s 172.18.0.4 cni-multus-control-plane
kube-system calico-node-7fk72 1/1 Running 0 4m31s 172.18.0.3 cni-multus-worker
kube-system calico-node-t67s5 1/1 Running 0 4m31s 172.18.0.2 cni-multus-worker2
kube-system coredns-5d78c9869d-p874p 1/1 Running 0 4m40s 10.244.226.131 cni-multus-control-plane
kube-system coredns-5d78c9869d-z6bcj 1/1 Running 0 4m40s 10.244.226.130 cni-multus-control-plane
kube-system etcd-cni-multus 1/1 Running 0 4m55s 172.18.0.4 cni-multus-control-plane
kube-system kube-apiserver-cni-multus 1/1 Running 0 4m55s 172.18.0.4 cni-multus-control-plane
kube-system kube-controller-manager-cni-multus 1/1 Running 0 4m56s 172.18.0.4 cni-multus-control-plane
kube-system kube-multus-ds-6nvfz 1/1 Running 0 4m31s 172.18.0.2 cni-multus-worker2
kube-system kube-multus-ds-hn6gj 1/1 Running 0 4m31s 172.18.0.3 cni-multus-worker
kube-system kube-multus-ds-r8qfk 1/1 Running 0 4m31s 172.18.0.4 cni-multus-control-plane
kube-system kube-proxy-4vfdj 1/1 Running 0 4m35s 172.18.0.2 cni-multus-worker2
kube-system kube-proxy-cz8sl 1/1 Running 0 4m40s 172.18.0.4 cni-multus-control-plane
kube-system kube-proxy-q9hv5 1/1 Running 0 4m33s 172.18.0.3 cni-multus-worker
kube-system kube-scheduler-cni-multus 1/1 Running 0 4m55s 172.18.0.4 cni-multus-control-plane
kube-system whereabouts-98fn6 1/1 Running 0 4m31s 172.18.0.3 cni-multus-worker
kube-system whereabouts-tqnqk 1/1 Running 0 4m31s 172.18.0.4 cni-multus-control-plane
kube-system whereabouts-xfbgj 1/1 Running 0 4m31s 172.18.0.2 cni-multus-worker2
root@network-demo:~# kubectl get crd network-attachment-definitions.k8s.cni.cncf.io
NAME CREATED AT
network-attachment-definitions.k8s.cni.cncf.io 2026-05-23T00:58:25Z
先通过 Multus CNI 使用 NetworkAttachmentDefinition CRD 创建 Pod 附加网络模板后，在生成 Pod 时引用对应模板：
#!/bin/bash
set -v
controller_node=`kubectl get nodes --no-headers -o custom-columns=NAME:.metadata.name| grep control-plane`
worker_node=`kubectl get nodes --no-headers -o custom-columns=NAME:.metadata.name| grep worker2`
cat <<EOF | kubectl apply -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: ipvlanl2-whereabouts-conf
spec:
config: '{
"cniVersion": "0.3.0",
"name": "whereaboutsexample",
"type": "ipvlan",
"mode": "l2",
"master": "eth0",
"ipam": {
"type": "whereabouts",
"range": "172.18.0.200-172.18.0.205/24"
}
}'
EOF
cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: ipvlan-pod1
annotations:
k8s.v1.cni.cncf.io/networks: ipvlanl2-whereabouts-conf@eth1
spec:
containers:
- name: nettool
image: burlyluo/nettool:latest
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN"]
nodeName: ${controller_node}
EOF
cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: ipvlan-pod-x
annotations:
k8s.v1.cni.cncf.io/networks: ipvlanl2-whereabouts-conf@eth1
spec:
containers:
- name: nettool
image: burlyluo/nettool:latest
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN"]
nodeName: ${controller_node}
EOF
cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: ipvlan-pod2
annotations:
k8s.v1.cni.cncf.io/networks: ipvlanl2-whereabouts-conf@eth1
spec:
containers:
- name: nettool
image: burlyluo/nettool:latest
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN"]
nodeName: ${worker_node}
EOF
创建测试 Pod 后，可能会一直卡在 ContainerCreating 状态，通过 describe pod 发现创建流程中的关键报错 failed to find plugin "ipvlan" in path [/opt/cni/bin]，这是因为 kind 创建 k8s 集群时的基础镜像中没有包含 IPVLAN 这个 CNI 插件。在主机上手动拉一下 CNI 包解压到几个 k8s node 容器中即可修复：
root@network-demo:~# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
ipvlan-pod-x 0/1 ContainerCreating 0 103s <none> cni-multus-control-plane
ipvlan-pod1 0/1 ContainerCreating 0 103s <none> cni-multus-control-plane
ipvlan-pod2 0/1 ContainerCreating 0 102s <none> cni-multus-worker2
root@network-demo:~# kubectl describe pods ipvlan-pod-x | grep -A4 'Events'
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal AddedInterface 110s multus Add eth0 [10.244.226.134/32] from k8s-pod-network
Warning FailedCreatePodSandBox 109s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "31265d3618de74e470e1a39c0670a463566a99176cbb3c7774d56589c81ba3a2": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"31265d3618de74e470e1a39c0670a463566a99176cbb3c7774d56589c81ba3a2" Netns:"/var/run/netns/cni-6db962c0-2e51-298e-9cbd-8599f6f18e71" IfName:"eth0" Args:"K8S_POD_NAME=ipvlan-pod-x;K8S_POD_INFRA_CONTAINER_ID=31265d3618de74e470e1a39c0670a463566a99176cbb3c7774d56589c81ba3a2;K8S_POD_UID=e8872b13-c212-4b3b-9158-794adf3a8527;IgnoreUnknown=1;K8S_POD_NAMESPACE=default" Path:"" ERRORED: error configuring pod [default/ipvlan-pod-x] networking: [default/ipvlan-pod-x/e8872b13-c212-4b3b-9158-794adf3a8527:whereaboutsexample]: error adding container to network "whereaboutsexample": failed to find plugin "ipvlan" in path [/opt/cni/bin]
': StdinData: {"capabilities":{"portMappings":true},"clusterNetwork":"/host/etc/cni/net.d/10-calico.conflist","cniVersion":"0.3.1","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","type":"multus-shim"}
root@network-demo:~# wget https://github.com/containernetworking/plugins/releases/download/v1.9.1/cni-plugins-linux-amd64-v1.9.1.tgz
root@network-demo:~# tar tf cni-plugins-linux-amd64-v1.9.1.tgz
./
./sbr
./tap
./dhcp
./dummy
./bridge
./host-device
./README.md
./LICENSE
./static
./portmap
./host-local
./firewall
./tuning
./vlan
./ptp
./macvlan
./loopback
./ipvlan
./vrf
./bandwidth
root@network-demo:~# for node in $(docker ps --format '{{.Names}}' | grep 'cni-multus'); do
docker cp cni-plugins-linux-amd64-v1.9.1.tgz ${node}:/
docker exec ${node} tar -C /opt/cni/bin/ -xzf /cni-plugins-linux-amd64-v1.9.1.tgz --no-same-owner ./ipvlan
docker exec ${node} rm -rf /cni-plugins-linux-amd64-v1.9.1.tgz
echo "${node} done"
done
通过 Pod 事件信息看出，由集群默认 CNI Calico 生成了 eth0 网卡，然后 Multus CNI 通过 NetworkAttachmentDefinition CRD 配置生成了 IPVLAN 模式的 eth1 网卡：
root@network-demo:~# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
ipvlan-pod-x 1/1 Running 0 18m 10.244.228.174 cni-multus-control-plane
ipvlan-pod1 1/1 Running 0 18m 10.244.228.175 cni-multus-control-plane
ipvlan-pod2 1/1 Running 0 18m 10.244.4.89 cni-multus-worker2
root@network-demo:~# kubectl describe pods ipvlan-pod-x | grep -A8 'Events'
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
## 通过创建流程可以看出,eth0 网卡是通过集群默认 CNI Calico 生成的
Normal AddedInterface 10s multus Add eth0 [10.244.228.176/32] from k8s-pod-network
## 而 eth1 网卡则是通过 IPVLAN 读取 NetworkAttachmentDefinition 配置生成的
Normal AddedInterface 9s multus Add eth1 [172.18.0.200/24] from default/ipvlanl2-whereabouts-conf
Normal Pulling 9s kubelet Pulling image "burlyluo/nettool:latest"
Normal Pulled 9s kubelet Successfully pulled image "burlyluo/nettool:latest" in 205.281392ms (205.292364ms including waiting)
Normal Created 9s kubelet Created container nettool
Normal Started 9s kubelet Started container nettool
以 control-plane 节点中的父网卡 eth0 与其上两个 Pod 子网卡信息为例，可以发现他们的 MAC 地址是一致的：
## Pod-x 子网卡信息
root@network-demo:~# kubectl exec -it ipvlan-pod-x -- ip address show eth1
4: eth1@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether a6:7c:81:af:f4:be brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.18.0.200/24 brd 172.18.0.255 scope global eth1
valid_lft forever preferred_lft forever
root@network-demo:~# kubectl exec -it ipvlan-pod-x -- ip -d link show eth1
4: eth1@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/ether a6:7c:81:af:f4:be brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0 minmtu 68 maxmtu 65535
ipvlan mode l2 bridge addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
## Control-plane 节点父网卡信息
root@network-demo:~# docker exec -it cni-multus-control-plane ip address show eth0
3: eth0@if26: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether a6:7c:81:af:f4:be brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.18.0.4/16 brd 172.18.255.255 scope global eth0
valid_lft forever preferred_lft forever
root@network-demo:~# docker exec -it cni-multus-control-plane ip -d link show eth0
3: eth0@if26: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
link/ether a6:7c:81:af:f4:be brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0 minmtu 68 maxmtu 65535
veth addrgenmode eui64 numtxqueues 8 numrxqueues 8 gso_max_size 65536 gso_max_segs 65535
## Pod-1 子网卡信息
root@network-demo:~# kubectl exec -it ipvlan-pod1 -- ip address show eth1
4: eth1@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether a6:7c:81:af:f4:be brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.18.0.202/24 brd 172.18.0.255 scope global eth1
valid_lft forever preferred_lft forever
root@network-demo:~# kubectl exec -it ipvlan-pod1 -- ip -d link show eth1
4: eth1@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/ether a6:7c:81:af:f4:be brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0 minmtu 68 maxmtu 65535
ipvlan mode l2 bridge addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
Pod-x（10.244.228.176）请求 Pod-1（10.244.228.175）
root@network-demo:~# kubectl exec -it ipvlan-pod-x -- ip route show
default via 169.254.1.1 dev eth0
169.254.1.1 dev eth0 scope link
172.18.0.0/24 dev eth1 proto kernel scope link src 172.18.0.200
## ARP 表是请求后看的
root@network-demo:~# kubectl exec -it ipvlan-pod-x -- ip neighbor show
169.254.1.1 dev eth0 lladdr ee:ee:ee:ee:ee:ee STALE
172.18.0.202 dev eth1 lladdr a6:7c:81:af:f4:be STALE
10.244.228.175 dev eth1 lladdr a6:7c:81:af:f4:be STALE
10.244.226.128 dev eth0 lladdr ee:ee:ee:ee:ee:ee STALE
root@network-demo:~#
root@network-demo:~# docker exec cni-multus-control-plane ip route show
default via 172.18.0.1 dev eth0
10.244.2.64/26 via 172.18.0.2 dev tunl0 proto bird onlink
10.244.4.64/26 via 172.18.0.2 dev tunl0 proto bird onlink
10.244.171.0/26 via 172.18.0.3 dev tunl0 proto bird onlink
blackhole 10.244.226.128/26 proto bird
10.244.226.129 dev cali65d6a29e132 scope link
10.244.226.130 dev calibbcace8eb36 scope link
10.244.226.131 dev calife013ec3a25 scope link
10.244.226.132 dev cali36b0b7ed002 scope link
blackhole 10.244.228.128/26 proto bird
## 同节点 Pod 之间请求直接走 Pod-1 的 calic903a597e8b veth pair 设备转发了
10.244.228.175 dev calic903a597e8b scope link
10.244.228.176 dev calic46f11b893d scope link
172.18.0.0/16 dev eth0 proto kernel scope link src 172.18.0.4
root@network-demo:~# docker exec cni-multus-control-plane ip neighbor show
10.244.228.176 dev calic46f11b893d lladdr a2:6a:47:6d:d3:15 STALE
10.244.226.132 dev cali36b0b7ed002 lladdr be:9d:a8:51:65:77 REACHABLE
172.18.0.2 dev eth0 lladdr aa:18:7e:65:a2:af REACHABLE
10.244.226.130 dev calibbcace8eb36 lladdr ae:79:98:32:49:98 REACHABLE
10.244.226.129 dev cali65d6a29e132 lladdr f2:c1:bc:28:d4:7f REACHABLE
172.18.0.1 dev eth0 lladdr 46:e0:1c:d1:1e:e0 STALE
10.244.226.131 dev calife013ec3a25 lladdr 0a:35:d1:7b:bc:ba REACHABLE
172.18.0.3 dev eth0 lladdr ae:66:81:13:7d:ee REACHABLE
10.244.228.175 dev calic903a597e8b lladdr 0e:c0:f7:6e:35:63 STALE
root@network-demo:~# kubectl exec -it ipvlan-pod-x -- ping -I eth1 -c2 10.244.228.175
PING 10.244.228.175 (10.244.228.175): 56 data bytes
64 bytes from 10.244.228.175: seq=0 ttl=64 time=0.137 ms
64 bytes from 10.244.228.175: seq=1 ttl=64 time=0.091 ms
--- 10.244.228.175 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.091/0.114/0.137 ms
这里可能会有疑问：eth1 并没有去 10.244.228.175 的路由，那他是怎么发出去的？
10.244.228.175 dev calic903a597e8b lladdr 0e:c0:f7:6e:35:63 STALE 由 Pod-1 的 veth pair 发给 Pod-1。root@network-demo:~# kubectl exec -it ipvlan-pod-x -- tcpdump -pnei eth1
02:36:14.450486 a6:7c:81:af:f4:be > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.244.228.175 tell 172.18.0.200, length 28
02:36:14.450511 a6:7c:81:af:f4:be > a6:7c:81:af:f4:be, ethertype ARP (0x0806), length 42: Reply 10.244.228.175 is-at a6:7c:81:af:f4:be, length 28
02:36:14.450513 a6:7c:81:af:f4:be > a6:7c:81:af:f4:be, ethertype IPv4 (0x0800), length 98: 172.18.0.200 > 10.244.228.175: ICMP echo request, id 41, seq 0, length 64
02:36:14.450569 a6:7c:81:af:f4:be > a6:7c:81:af:f4:be, ethertype IPv4 (0x0800), length 98: 10.244.228.175 > 172.18.0.200: ICMP echo reply, id 41, seq 0, length 64
02:36:15.450654 a6:7c:81:af:f4:be > a6:7c:81:af:f4:be, ethertype IPv4 (0x0800), length 98: 172.18.0.200 > 10.244.228.175: ICMP echo request, id 41, seq 1, length 64
02:36:15.450708 a6:7c:81:af:f4:be > a6:7c:81:af:f4:be, ethertype IPv4 (0x0800), length 98: 10.244.228.175 > 172.18.0.200: ICMP echo reply, id 41, seq 1, length 64
如果在父网卡 eth0 抓包，会发现只有 request 没有 response。这是因为请求到了 Pod-1 响应时，直接走了 Pod-1 中的 eth1 网卡，没有再通过主机 eth0 这个父网卡转发
root@network-demo:~# docker exec -it cni-multus-control-plane tcpdump -pnei eth0 icmp
05:33:05.028947 a6:7c:81:af:f4:be > a6:7c:81:af:f4:be, ethertype IPv4 (0x0800), length 98: 172.18.0.200 > 10.244.228.175: ICMP echo request, id 150, seq 0, length 64
05:33:06.029084 a6:7c:81:af:f4:be > a6:7c:81:af:f4:be, ethertype IPv4 (0x0800), length 98: 172.18.0.200 > 10.244.228.175: ICMP echo request, id 150, seq 1, length 64
root@network-demo:~# kubectl exec -it pods/ipvlan-pod1 -- ip route show
default via 169.254.1.1 dev eth0
169.254.1.1 dev eth0 scope link
172.18.0.0/24 dev eth1 proto kernel scope link src 172.18.0.202
root@network-demo:~# kubectl exec -it pods/ipvlan-pod1 -- ip neighbor show
169.254.1.1 dev eth0 lladdr ee:ee:ee:ee:ee:ee STALE
172.18.0.200 dev eth1 lladdr a6:7c:81:af:f4:be STALE
10.244.226.128 dev eth0 lladdr ee:ee:ee:ee:ee:ee STALE
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。