作者：尹正杰

版权声明：原创作品，谢绝转载！否则将追究法律责任。

一.kubeasz 部署准备

1.kubeasz概述

kubeasz 致力于提供快速部署高可用k8s集群的工具, 同时也努力成为k8s实践、使用的参考书；基于二进制方式部署和利用ansible-playbook实现自动化；既提供一键安装脚本, 也可以根据安装指南分步执行安装各个组件。

kubeasz 从每一个单独部件组装到完整的集群，提供最灵活的配置能力，几乎可以设置任何组件的任何参数；同时又为集群创建预置一套运行良好的默认配置，甚至自动化创建适合大规模集群的BGP Route Reflector网络模式。

kubeasz 通过cncf一致性测试，其支持的K8S版本如上表所示。
 
推荐阅读:
	https://github.com/easzlab/kubeasz/tree/master

2.kubeasz部署节点环境准备

	1.kubeasz部署节点安装ansible环境
[root@ansible99 ~]# apt -y install ansible

 
	2.配置免秘钥认证
[root@ansible99 ~]# ssh-keygen -t rsa -f ~/.ssh/id_rsa -P '' -q
[root@ansible99 ~]# 

	3.kubeasz部署节点配置master，etcd，worker节点的免秘钥登录
[root@ansible99 ~]# ssh-copy-id 10.0.0.231
[root@ansible99 ~]# ssh-copy-id 10.0.0.232
[root@ansible99 ~]# ssh-copy-id 10.0.0.233
[root@ansible99 ~]# ssh-copy-id 10.0.0.66
[root@ansible99 ~]# ssh-copy-id 10.0.0.77


	4.通过ansible部署的所有节点(master，etcd，worker，rr)均需要配置Python的软连接
ln -sv /usr/bin/python3.8 /usr/bin/python

3.下载项目源码、二进制及离线镜像

参考链接:
	https://github.com/easzlab/kubeasz/blob/master/docs/setup/00-planning_and_overall_intro.md
	

	1.下载工具脚本ezdown，比如我要部署K8S 1.33，则使用3.6.7版本
export RELEASE="3.6.7"
wget https://github.com/easzlab/kubeasz/releases/download/${RELEASE}/ezdown
chmod +x ezdown


	2.查看帮助信息
[root@ansible99 ~]# ./ezdown 
Usage: ezdown [options] [args]
  option:
    -C         stop&clean all local containers
    -D         download default binaries/images into "/etc/kubeasz"
    -P <OS>    download system packages of the OS (ubuntu_22,debian_11,...)
    -R         download Registry(harbor) offline installer
    -S         start kubeasz in a container
    -X <opt>   download extra images
    -d <ver>   set docker-ce version, default "28.0.4"
    -e <ver>   set kubeasz-ext-bin version, default "1.12.5"
    -k <ver>   set kubeasz-k8s-bin version, default "v1.33.1"
    -m <str>   set docker registry mirrors, default "CN"(used in Mainland,China)
    -z <ver>   set kubeasz version, default "3.6.7"
[root@ansible99 ~]# 


	3.修改部署脚本文件
[root@k8s-cluster241 ~]# vim ezdown 
...
DOCKER_VER=20.10.24
K8S_BIN_VER=v1.33.11 # 官方已经更新到12版本，但是kubeasz目前最高支持11版本。
calicoVer=v3.31.5  # k8s v1.33版本依赖于Calico v3.31版本。
	
	4.检查更改的配置版本
[root@ansible99 ~]# ./ezdown 
Usage: ezdown [options] [args]
  option:
    -C         stop&clean all local containers
    -D         download default binaries/images into "/etc/kubeasz"
    -P <OS>    download system packages of the OS (ubuntu_22,debian_11,...)
    -R         download Registry(harbor) offline installer
    -S         start kubeasz in a container
    -X <opt>   download extra images
    -d <ver>   set docker-ce version, default "20.10.24"
    -e <ver>   set kubeasz-ext-bin version, default "1.12.5"
    -k <ver>   set kubeasz-k8s-bin version, default "v1.33.11"
    -m <str>   set docker registry mirrors, default "CN"(used in Mainland,China)
    -z <ver>   set kubeasz version, default "3.6.7"
[root@ansible99 ~]# 



	5.下载kubeasz代码、二进制、默认容器镜像
./ezdown -D  # 国内环境
#./ezdown -D -m standard  # 海外环境



温馨提示:
	1.有可能官方的脚本抽风，下载不下来软件，就根据脚本的链接手动下载到/etc/kubeasz/down/目录即可;
		[root@ansible99 ~]# wget https://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux/static/stable/x86_64/docker-20.10.24.tgz -O /etc/kubeasz/down/docker-20.10.24.tgz

	2.k8s官方1.33.12是最新版本，但是hub仓库只有1.33.11版本的镜像;
		执行如下命令即可:
			[root@k8s-cluster241 ~]# ./ezdown -D -k v1.33.11
			
	3.如果拉取不了镜像，需要配置docker的VPN代理，效果如下:
[root@k8s-cluster241 ~]# systemctl cat docker
...
[Service]
...
# 主要添加如下3行代码，仅需要修改你自己的VPN地址即可.
Environment="HTTP_PROXY=http://10.0.0.1:7890"
Environment="HTTPS_PROXY=http://10.0.0.1:7890"
Environment="NO_PROXY=localhost,127.0.0.1,easzlab.io.local,.docker.internal,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16"
ExecStart=/opt/kube/bin/dockerd
...
[root@k8s-cluster241 ~]# 

二.kubeasz部署节点部署K8S集群

1.创建集群配置实例

1.1 容器化运行kubeasz

[root@ansible99 ~]# docker  ps -a
CONTAINER ID   IMAGE        COMMAND                  CREATED         STATUS         PORTS     NAMES
f39ce4cd24bf   registry:2   "/entrypoint.sh /etc…"   3 minutes ago   Up 3 minutes             local_registry
[root@ansible99 ~]# 
[root@ansible99 ~]# ./ezdown -S
Loaded image: easzlab/kubeasz:3.6.7
2026-05-31 01:56:07 [ezdown:619] INFO try to run kubeasz in a container
2026-05-31 01:56:08 [ezdown:623] DEBUG get host IP: 10.0.0.99
5402a192a6f91867e38462ba339ae94b50e7ef034c1b2d429c2d70690f32d9ca
2026-05-31 01:56:08 [ezdown:719] INFO Action successed: start_kubeasz_docker
[root@ansible99 ~]# 
[root@ansible99 ~]# docker  ps -a
CONTAINER ID   IMAGE                   COMMAND                  CREATED         STATUS         PORTS     NAMES
5402a192a6f9   easzlab/kubeasz:3.6.7   "tail -f /dev/null"      2 seconds ago   Up 2 seconds             kubeasz
f39ce4cd24bf   registry:2              "/entrypoint.sh /etc…"   3 minutes ago   Up 3 minutes             local_registry
[root@ansible99 ~]# 

1.2 创建新集群"yinzhengjie-k8s"

[root@ansible99 ~]# docker exec -it kubeasz ezctl new yinzhengjie-k8s
[root@ansible99 ~]# 
[root@ansible99 ~]# ll /etc/kubeasz/clusters/yinzhengjie-k8s/  # 生成如下2个文件。
total 20
drwxr-xr-x 2 root root 4096 May 31 01:56 ./
drwxr-xr-x 3 root root 4096 May 31 01:56 ../
-rw-r--r-- 1 root root 7801 May 31 01:56 config.yml
-rw-r--r-- 1 root root 2383 May 31 01:56 hosts
[root@ansible99 ~]# 

1.3 配置 Ansible Inventory（hosts 文件）

[root@ansible99 ~]# cat  /etc/kubeasz/clusters/yinzhengjie-k8s/hosts
# 'etcd' cluster should have odd member(s) (1,3,5,...)
[etcd]
10.0.0.231
10.0.0.232
10.0.0.233


# master node(s), set unique 'k8s_nodename' for each node
# CAUTION: 'k8s_nodename' must consist of lower case alphanumeric characters, '-' or '.',
# and must start and end with an alphanumeric character
[kube_master]
10.0.0.231 k8s_nodename='master231'
10.0.0.232 k8s_nodename='master232'


# work node(s), set unique 'k8s_nodename' for each node
# CAUTION: 'k8s_nodename' must consist of lower case alphanumeric characters, '-' or '.',
# and must start and end with an alphanumeric character
[kube_node]
10.0.0.66 k8s_nodename='worker66'
10.0.0.77 k8s_nodename='worker77'

# [optional] harbor server, a private docker registry
# 'NEW_INSTALL': 'true' to install a harbor server; 'false' to integrate with existed one
[harbor]
#192.168.1.8 NEW_INSTALL=false

# [optional] loadbalance for accessing k8s from outside，此配置无效，貌似并不会给指定节点安装高可用组件，配置vip等功能，可跳过。
[ex_lb]
10.0.0.232 LB_ROLE=backup EX_APISERVER_VIP=10.0.0.230 EX_APISERVER_PORT=8443
10.0.0.231 LB_ROLE=master EX_APISERVER_VIP=10.0.0.230 EX_APISERVER_PORT=8443

# [optional] ntp server for the cluster
[chrony]
#192.168.1.1

[all:vars]
# --------- Main Variables ---------------
# Secure port for apiservers
SECURE_PORT="6443"

# Cluster container-runtime supported: docker, containerd
# if k8s version >= 1.24, docker is not supported
CONTAINER_RUNTIME="containerd"

# Network plugins supported: calico, flannel, kube-router, cilium, kube-ovn
CLUSTER_NETWORK="calico"

# Service proxy mode of kube-proxy: 'iptables' or 'ipvs'
PROXY_MODE="ipvs"

# K8S Service CIDR, not overlap with node(host) networking
SERVICE_CIDR="10.200.0.0/16"

# Cluster CIDR (Pod CIDR), not overlap with node(host) networking，我尝试修改了这个参数，但是并不能生效！
CLUSTER_CIDR="172.20.0.0/16"

# NodePort Range
NODE_PORT_RANGE="30000-32767"

# Cluster DNS Domain
#CLUSTER_DNS_DOMAIN="cluster.local"
CLUSTER_DNS_DOMAIN="yinzhengjie.com"

# -------- Additional Variables (don't change the default value right now) ---
# Binaries Directory
# bin_dir="/opt/kube/bin"
bin_dir="/usr/local/bin"

# Deploy Directory (kubeasz workspace)
base_dir="/etc/kubeasz"

# Directory for a specific cluster
cluster_dir="{{ base_dir }}/clusters/yinzhengjie-k8s"

# CA and other components cert/key Directory
ca_dir="/etc/kubernetes/ssl"

# Default 'k8s_nodename' is empty
k8s_nodename=''

# Default python interpreter
ansible_python_interpreter=/usr/bin/python3
[root@ansible99 ~]# 

1.4 核心配置文件（config.yml）

  1.修改关键参数
[root@ansible99 ~]# vim /etc/kubeasz/clusters/yinzhengjie-k8s/config.yml 
...
K8S_VER: "1.33.11" 
...
MASTER_CERT_HOSTS:
  - "10.0.0.231"
  - "10.0.0.232"
  - "10.0.0.233"
  - "master231"
  - "master232"
  - "master233"
  - "master.yinzhengjie.com"
...
# 我们部署的k8s v1.33版本，默认的v3.28.4版本最高支持k8s v1.30
calico_ver: "v3.31.5" 
...

CALICO_IPV4POOL_IPIP: "Never" 
...


  2.完整文件如下
[root@ansible99 ~]# cat  /etc/kubeasz/clusters/yinzhengjie-k8s/config.yml
############################
# prepare
############################
# 可选离线安装系统软件包 (offline|online)
INSTALL_SOURCE: "online"

# 可选进行系统安全加固 github.com/dev-sec/ansible-collection-hardening
# (deprecated) 未更新上游项目，未验证最新k8s集群安装，不建议启用
OS_HARDEN: false


############################
# role:deploy
############################
# default: ca will expire in 100 years
# default: certs issued by the ca will expire in 50 years
CA_EXPIRY: "876000h"
CERT_EXPIRY: "438000h"

# force to recreate CA and other certs, not suggested to set 'true'
CHANGE_CA: false

# kubeconfig 配置参数
CLUSTER_NAME: "cluster1"
CONTEXT_NAME: "context-{{ CLUSTER_NAME }}"

# k8s version
K8S_VER: "1.33.11"

# set unique 'k8s_nodename' for each node, if not set(default:'') ip add will be used
# CAUTION: 'k8s_nodename' must consist of lower case alphanumeric characters, '-' or '.',
# and must start and end with an alphanumeric character (e.g. 'example.com'),
# regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'
K8S_NODENAME: "{%- if k8s_nodename != '' -%} \
                    {{ k8s_nodename|replace('_', '-')|lower }} \
               {%- else -%} \
                    k8s-{{ inventory_hostname|replace('.', '-') }} \
               {%- endif -%}"

# use 'K8S_NODENAME' to set hostname
ENABLE_SETTING_HOSTNAME: true


############################
# role:etcd
############################
# 设置不同的wal目录，可以避免磁盘io竞争，提高性能
ETCD_DATA_DIR: "/var/lib/etcd"
ETCD_WAL_DIR: ""


############################
# role:runtime [containerd,docker]
############################
# [.]启用拉取加速镜像仓库
ENABLE_MIRROR_REGISTRY: true

# [.]添加信任的私有仓库
# 必须按照如下示例格式，协议头'http://'和'https://'不能省略
INSECURE_REG:
  - "http://easzlab.io.local:5000"
  - "https://reg.yourcompany.com"

# [.]基础容器镜像
SANDBOX_IMAGE: "easzlab.io.local:5000/easzlab/pause:3.10"

# [containerd]容器持久化存储目录
CONTAINERD_STORAGE_DIR: "/var/lib/containerd"

# [docker]容器存储目录
DOCKER_STORAGE_DIR: "/var/lib/docker"

# [docker]开启Restful API
DOCKER_ENABLE_REMOTE_API: false


############################
# role:kube-master
############################
# k8s 集群 master 节点证书配置，可以添加多个ip和域名（比如增加公网ip和域名）
MASTER_CERT_HOSTS:
  - "10.0.0.231"
  - "10.0.0.232"
  - "10.0.0.233"
  - "master231"
  - "master232"
  - "master233"
  - "master.yinzhengjie.com"

# node 节点上 pod 网段掩码长度（决定每个节点最多能分配的pod ip地址）
# 如果flannel 使用 --kube-subnet-mgr 参数，那么它将读取该设置为每个节点分配pod网段
# https://github.com/coreos/flannel/issues/847
NODE_CIDR_LEN: 24


############################
# role:kube-node
############################
# Kubelet 根目录
KUBELET_ROOT_DIR: "/var/lib/kubelet"

# node节点最大pod 数
MAX_PODS: 110

# 配置为kube组件（kubelet,kube-proxy,dockerd等）预留的资源量
# 数值设置详见templates/kubelet-config.yaml.j2
KUBE_RESERVED_ENABLED: "no"

# k8s 官方不建议草率开启 system-reserved, 除非你基于长期监控，了解系统的资源占用状况；
# 并且随着系统运行时间，需要适当增加资源预留，数值设置详见templates/kubelet-config.yaml.j2
# 系统预留设置基于 4c/8g 虚机，最小化安装系统服务，如果使用高性能物理机可以适当增加预留
# 另外，集群安装时候apiserver等资源占用会短时较大，建议至少预留1g内存
SYS_RESERVED_ENABLED: "no"


############################
# role:network [flannel,calico,cilium,kube-ovn,kube-router]
############################
# ------------------------------------------- flannel
# [flannel]设置flannel 后端"host-gw","vxlan"等
FLANNEL_BACKEND: "vxlan"
DIRECT_ROUTING: true

# [flannel] 
flannel_ver: "v0.26.0"

# ------------------------------------------- calico
# [calico] IPIP隧道模式可选项有: [Always, CrossSubnet, Never],跨子网可以配置为Always与CrossSubnet(公有云建议使用always比较省事，其他的话需要修改各自公有云的网络配置，具体可以参考各个公有云说明)
# 其次CrossSubnet为隧道+BGP路由混合模式可以提升网络性能，同子网配置为Never即可.
CALICO_IPV4POOL_IPIP: "Never"

# [calico]设置 calico-node使用的host IP，bgp邻居通过该地址建立，可手工指定也可以自动发现
IP_AUTODETECTION_METHOD: "can-reach={{ groups['kube_master'][0] }}"

# [calico]设置calico 网络 backend: bird, vxlan, none
CALICO_NETWORKING_BACKEND: "bird"


# [calico]设置calico 是否使用route reflectors
# 如果集群规模超过50个节点，建议启用该特性
CALICO_RR_ENABLED: false
CALICO_IPV6POOL_DISABLED: true

# CALICO_RR_NODES 配置route reflectors的节点，如果未设置默认使用集群master节点 
# CALICO_RR_NODES: ["192.168.1.1", "192.168.1.2"]


# [calico]更新支持calico 版本: ["3.19", "3.23"]
calico_ver: "v3.31.5"

# [calico]calico 主版本
calico_ver_main: "{{ calico_ver.split('.')[0] }}.{{ calico_ver.split('.')[1] }}"

# ------------------------------------------- cilium
# [cilium]镜像版本
cilium_ver: "1.17.4"
cilium_connectivity_check: false
cilium_hubble_enabled: false
cilium_hubble_ui_enabled: false

# ------------------------------------------- kube-ovn
# [kube-ovn]离线镜像tar包
kube_ovn_ver: "v1.11.5"

# ------------------------------------------- kube-router
# [kube-router]公有云上存在限制，一般需要始终开启 ipinip；自有环境可以设置为 "subnet"
OVERLAY_TYPE: "full"

# [kube-router]NetworkPolicy 支持开关
FIREWALL_ENABLE: true

# [kube-router]kube-router 镜像版本
kube_router_ver: "v1.5.4"


############################
# role:cluster-addon
############################
# coredns 自动安装
dns_install: "yes"
corednsVer: "1.12.1"
ENABLE_LOCAL_DNS_CACHE: true
dnsNodeCacheVer: "1.25.0"
# 设置 local dns cache 地址
LOCAL_DNS_CACHE: "169.254.20.10"

# metric server 自动安装
metricsserver_install: "yes"
metricsVer: "v0.7.2"

# dashboard 自动安装
dashboard_install: "no"
dashboardVer: "7.12.0"

# prometheus 自动安装
prom_install: "no"
prom_namespace: "monitor"
prom_chart_ver: "45.23.0"

# kubeapps 自动安装，如果选择安装，默认同时安装local-storage（提供storageClass: "local-path"）
kubeapps_install: "no"
kubeapps_install_namespace: "kubeapps"
kubeapps_working_namespace: "default"
kubeapps_storage_class: "local-path"
kubeapps_chart_ver: "12.4.3"

# local-storage (local-path-provisioner) 自动安装
local_path_provisioner_install: "no"
local_path_provisioner_ver: "v0.0.31"
# 设置默认本地存储路径
local_path_provisioner_dir: "/opt/local-path-provisioner"

# nfs-provisioner 自动安装
nfs_provisioner_install: "no"
nfs_provisioner_namespace: "kube-system"
nfs_provisioner_ver: "v4.0.2"
nfs_storage_class: "managed-nfs-storage"
nfs_server: "192.168.1.10"
nfs_path: "/data/nfs"

# network-check 自动安装
network_check_enabled: false 
network_check_schedule: "*/5 * * * *"

# kubeblocks 自动安装
kubeblocks_install: "no"
kb_addon_mysql_install: "no"
kb_addon_pg_install: "no"
kb_addon_redis_install: "no"
kb_addon_minio_install: "no"
kb_addon_es_install: "no"
kb_addon_mongodb_install: "no"


############################
# role:harbor
############################
# harbor version，完整版本号
HARBOR_VER: "v2.12.4"
HARBOR_DOMAIN: "harbor.yinzhengjie.com"
HARBOR_PATH: /var/data
HARBOR_TLS_PORT: 8443
HARBOR_REGISTRY: "{{ HARBOR_DOMAIN }}:{{ HARBOR_TLS_PORT }}"

# if set 'false', you need to put certs named harbor.pem and harbor-key.pem in directory 'down'
HARBOR_SELF_SIGNED_CERT: true

# install extra component
HARBOR_WITH_TRIVY: false
[root@ansible99 ~]# 

1.5手动添加 calico-v3.31.yaml.j2 模板

	1.kubeasz官方内置的Calico版本模板有限，如下所示，最高支持3.28版本。
[root@ansible99 ~]# cd /etc/kubeasz/roles/calico/templates/
[root@ansible99 templates]# 
[root@ansible99 templates]# ll
total 96
drwxrwxr-x 2 root root  4096 Jun  8  2025 ./
drwxrwxr-x 5 root root  4096 Jun  8  2025 ../
-rw-rw-r-- 1 root root   180 Jun  8  2025 bgp-default.yaml.j2
-rw-rw-r-- 1 root root   162 Jun  8  2025 bgp-rr.yaml.j2
-rw-rw-r-- 1 root root   215 Jun  8  2025 calico-csr.json.j2
-rw-rw-r-- 1 root root   263 Jun  8  2025 calicoctl.cfg.j2
-rw-rw-r-- 1 root root 21593 Jun  8  2025 calico-v3.24.yaml.j2
-rw-rw-r-- 1 root root 22348 Jun  8  2025 calico-v3.26.yaml.j2
-rw-rw-r-- 1 root root 22523 Jun  8  2025 calico-v3.28.yaml.j2
[root@ansible99 templates]# 



	2.下载v3.31的模板
[root@ansible99 templates]# wget https://raw.githubusercontent.com/projectcalico/calico/v3.31.5/manifests/calico.yaml -O calico-v3.31.yaml.j2

[root@ansible99 templates]# ll
total 420
drwxrwxr-x 2 root root   4096 May 31 02:26 ./
drwxrwxr-x 5 root root   4096 Jun  8  2025 ../
-rw-rw-r-- 1 root root    180 Jun  8  2025 bgp-default.yaml.j2
-rw-rw-r-- 1 root root    162 Jun  8  2025 bgp-rr.yaml.j2
-rw-rw-r-- 1 root root    215 Jun  8  2025 calico-csr.json.j2
-rw-rw-r-- 1 root root    263 Jun  8  2025 calicoctl.cfg.j2
-rw-rw-r-- 1 root root  21593 Jun  8  2025 calico-v3.24.yaml.j2
-rw-rw-r-- 1 root root  22348 Jun  8  2025 calico-v3.26.yaml.j2
-rw-rw-r-- 1 root root  22523 Jun  8  2025 calico-v3.28.yaml.j2
-rw-r--r-- 1 root root 331348 May 31 02:31 calico-v3.31.yaml.j2
[root@ansible99 templates]# 

2.开始安装

1.建议配置命令alias，方便执行

[root@ansible99 ~]# echo "alias dk='docker exec -it kubeasz'" >> /root/.bashrc
[root@ansible99 ~]# 
[root@ansible99 ~]# source /root/.bashrc
[root@ansible99 ~]# 

2.修改准备环境的剧本，注释外部负载均衡器和时间服务器。

	
[root@ansible99 ~]# vim /etc/kubeasz/playbooks/90.setup.yml 
...
- hosts:
  - kube_master
  - kube_node
  - etcd
  # 建议将外部的负载均衡器和时间服务器都注释掉哟~这样就不会去部署这两个服务啦!
  # - ex_lb
  # - chrony
	
	

3.一键安装

[root@ansible99 ~]# dk ezctl setup yinzhengjie-k8s all
ansible-playbook -i clusters/yinzhengjie-k8s/hosts -e @clusters/yinzhengjie-k8s/config.yml  playbooks/90.setup.yml
*** Component Version *********************
*******************************************
*   kubernetes: v1.33.11
*   etcd: v3.5.21
*   calico: v3.28.4
*******************************************
...
	


温馨提示: 
[root@ansible99 ~]# dk ezctl setup yinzhengjie-k8s  # 直接单独输入编号01~07即可。
Usage: ezctl setup <cluster> <step>
available steps:
    01  prepare            to prepare CA/certs & kubeconfig & other system settings
    02  etcd               to setup the etcd cluster
    03  container-runtime  to setup the container runtime(docker or containerd)
    04  kube-master        to setup the master nodes
    05  kube-node          to setup the worker nodes
    06  network            to setup the network plugin
    07  cluster-addon      to setup other useful plugins
    90  all                to run 01~07 all at once
    10  ex-lb              to install external loadbalance for accessing k8s from outside
    11  harbor             to install a new harbor server or to integrate with an existed one

examples: ./ezctl setup test-k8s 01  (or ./ezctl setup test-k8s prepare)
	  ./ezctl setup test-k8s 02  (or ./ezctl setup test-k8s etcd)
          ./ezctl setup test-k8s all
          ./ezctl setup test-k8s 04 -t restart_master
[root@ansible99 ~]# 



	温馨提示:
		本质上安装K8S集群是读取的"/etc/kubeasz/playbooks"里面的剧本文件哟~

3.验证网络连通性

1.查看worker节点

[root@master231 ~]# kubectl get pods -A -o wide
NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE   IP               NODE        NOMINATED NODE   READINESS GATES
kube-system   calico-kube-controllers-654fb9bf6f-7q6lr   1/1     Running   0          43s   192.168.65.129   worker66    <none>           <none>
kube-system   calico-node-77ch2                          1/1     Running   0          43s   10.0.0.77        worker77    <none>           <none>
kube-system   calico-node-mf9g9                          1/1     Running   0          42s   10.0.0.66        worker66    <none>           <none>
kube-system   calico-node-t6mnf                          1/1     Running   0          42s   10.0.0.231       master231   <none>           <none>
kube-system   calico-node-wzzrj                          1/1     Running   0          42s   10.0.0.232       master232   <none>           <none>
[root@master231 ~]# 
[root@master231 ~]# kubectl get no -o wide
NAME        STATUS                     ROLES    AGE   VERSION    INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
master231   Ready,SchedulingDisabled   master   1m   v1.33.11   10.0.0.231    <none>        Ubuntu 24.04.4 LTS   6.8.0-117-generic   containerd://2.1.1
master232   Ready,SchedulingDisabled   master   1m   v1.33.11   10.0.0.232    <none>        Ubuntu 24.04.4 LTS   6.8.0-117-generic   containerd://2.1.1
worker66    Ready                      node     1m   v1.33.11   10.0.0.66     <none>        Ubuntu 24.04.4 LTS   6.8.0-117-generic   containerd://2.1.1
worker77    Ready                      node     1m   v1.33.11   10.0.0.77     <none>        Ubuntu 24.04.4 LTS   6.8.0-117-generic   containerd://2.1.1
[root@master231 ~]# 

2.创建Pod

[root@master231 ~]# cat > deploy-apps.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: yinzhengjie-app01
spec:
  replicas: 1
  selector:
    matchLabels:
      apps: v1 
  template:
    metadata:
      labels:
        apps: v1
    spec:
      nodeName: worker66
      containers:
      - name: c1
        image: registry.cn-hangzhou.aliyuncs.com/yinzhengjie-k8s/apps:v1 

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: yinzhengjie-app02
spec:
  replicas: 1
  selector:
    matchLabels:
      apps: v1 
  template:
    metadata:
      labels:
        apps: v1
    spec:
      nodeName: worker77
      containers:
      - name: c1
        image: registry.cn-hangzhou.aliyuncs.com/yinzhengjie-k8s/apps:v2
EOF


[root@master231 ~]# kubectl apply -f   deploy-apps.yaml 
deployment.apps/yinzhengjie-app01 created
deployment.apps/yinzhengjie-app02 created
[root@master231 ~]# 
[root@master231 ~]# kubectl get pods -o wide
NAME                                 READY   STATUS    RESTARTS   AGE   IP               NODE       NOMINATED NODE   READINESS GATES
yinzhengjie-app01-674b688bcc-fkkrj   1/1     Running   0          19s   192.168.65.130   worker66   <none>           <none>
yinzhengjie-app02-58d588dc64-s9hw5   1/1     Running   0          19s   192.168.165.2    worker77   <none>           <none>
[root@master231 ~]#  

3.测试连通性

[root@master231 ~]# curl 192.168.65.130
<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8"/>
    <title>yinzhengjie apps v1</title>
    <style>
       div img {
          width: 900px;
          height: 600px;
          margin: 0;
       }
    </style>
  </head>

  <body>
    <h1 style="color: green">凡人修仙传 v1 </h1>
    <div>
      <img src="1.jpg">
    <div>
  </body>

</html>
[root@master231 ~]# 
[root@master231 ~]# curl 192.168.165.2
<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8"/>
    <title>yinzhengjie apps v2</title>
    <style>
       div img {
          width: 900px;
          height: 600px;
          margin: 0;
       }
    </style>
  </head>

  <body>
    <h1 style="color: red">凡人修仙传 v2 </h1>
    <div>
      <img src="2.jpg">
    <div>
  </body>

</html>
[root@master231 ~]# 

4.master节点配置K8S自动补全

1.master节点执行自动补全代码

echo "source <(kubectl completion bash)" >> ~/.bashrc && source ~/.bashrc 

2.测试效果验证

如上图所示，说明咱们的自动补全功能生效。

三.补录： 可能会遇到的报错

1.HTTP request sent, awaiting response... 403 Forbidden

  报错信息：
[root@ansible99 ~]# ./ezdown -D
2026-06-01 01:19:53 [ezdown:717] INFO Action begin: download_all
2026-06-01 01:19:53 [ezdown:162] INFO downloading docker binaries, arch:x86_64, version:20.10.24
--2026-06-01 01:19:53--  https://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux/static/stable/x86_64/docker-20.10.24.tgz
Resolving mirrors.tuna.tsinghua.edu.cn (mirrors.tuna.tsinghua.edu.cn)... 101.6.15.130, 2402:f000:1:400::2
Connecting to mirrors.tuna.tsinghua.edu.cn (mirrors.tuna.tsinghua.edu.cn)|101.6.15.130|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2026-06-01 01:19:53 ERROR 403: Forbidden.

2026-06-01 01:19:53 [ezdown:164] ERROR downloading docker failed
[root@ansible99 ~]# 

  解决方案:
[root@ansible99 ~]# wget https://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux/static/stable/x86_64/docker-20.10.24.tgz -O /etc/kubeasz/down/docker-20.10.24.tgz

2.Could not find or access 'calico-v3.31.yaml.j2'\nSearched in ...

  报错信息:
TASK [calico : 配置 calico DaemonSet yaml文件] **************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: If you are using a module and expect the file to exist on the remote, see the remote_src option
fatal: [10.0.0.231]: FAILED! => {"changed": false, "msg": "Could not find or access 'calico-v3.31.yaml.j2'\nSearched in:\n\t/etc/kubeasz/roles/calico/templates/calico-v3.31.yaml.j2\n\t/etc/kubeasz/roles/calico/calico-v3.31.yaml.j2\n\t/etc/kubeasz/roles/calico/tasks/templates/calico-v3.31.yaml.j2\n\t/etc/kubeasz/roles/calico/tasks/calico-v3.31.yaml.j2\n\t/etc/kubeasz/playbooks/templates/calico-v3.31.yaml.j2\n\t/etc/kubeasz/playbooks/calico-v3.31.yaml.j2 on the Ansible Controller.\nIf you are using a module and expect the file to exist on the remote, see the remote_src option"}

NO MORE HOSTS LEFT ******************************************************************************************************************************


  解决方案:
    1.下载模板
[root@ansible99 ~]# cd /etc/kubeasz/roles/calico/templates
[root@ansible99 templates]# 
[root@ansible99 templates]# 
[root@ansible99 templates]#  wget https://raw.githubusercontent.com/projectcalico/calico/v3.31.5/manifests/calico.yaml -O calico-v3.31.yaml.j2


    2.重新安装网络。
[root@ansible99 ~]# dk ezctl setup yinzhengjie-k8s  06

3.https://registry-1.docker.io/v2/": dial tcp 128.242.245.93:443: i/o timeout

  报错信息:
3.10: Pulling from easzlab/pause
61d9e957431b: Pulling fs layer 
Get "https://registry-1.docker.io/v2/": dial tcp 128.242.245.93:443: i/o timeout
2026-06-02 12:33:57 [ezdown:556] ERROR download easzlab/pause:3.10 failed!
2026-06-02 12:33:57 [ezdown:718] ERROR Action failed: download_all
[root123@vm1 ~]# 


  解决方案:
    - 1.可以多尝试几次;
	- 2.如果拉取不了镜像，需要配置docker的VPN代理，效果如下:
[root@k8s-cluster241 ~]# systemctl cat docker
...
[Service]
...
# 主要添加如下3行代码，仅需要修改你自己的VPN地址即可.
Environment="HTTP_PROXY=http://10.0.0.1:7890"
Environment="HTTPS_PROXY=http://10.0.0.1:7890"
Environment="NO_PROXY=localhost,127.0.0.1,easzlab.io.local,.docker.internal,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16"
ExecStart=/opt/kube/bin/dockerd
...
[root@k8s-cluster241 ~]#