二进制部署K8S 1.35.0+最新版实战案例

博客园 - 尹正杰

Categraf使用指南 window系统部署Claude Code实战 N9E集群部署 Ubuntu 24.04 LTS部署Codex实战 aiops运维平台从0到1部署实战 aiops平台环境部署 TKE集群LoadBalancer映射案例 k8s集群为每个节点分配不通的ippool实战 ansible剧本实现一键升级K8S集群的NVIDIA GPU驱动 vmagent采集etcd数据并写入victoriametrics集群 VictoriaMetrics集群部署实战 Squid正向代理实战 kubespray生产故障案例之kubelet启动参数案例 filebeat指定名称空间采集pod数据 Calico自定义Ipool实战 ansible剧本实现kibana的部署和卸载 ansible剧本应用案例集合2 ansible剧本实现磁盘格式化 Ansible急速入门实战篇 ansible快速入门篇 k8s集群管理 kubesray实战 kubespray实战案例 kubespray管理k8s的worker集群扩缩容 kubespray快速部署k8s集群实战 Kubeasz使用吐槽博客专题 Kubeasz基于ezctl实现etcd集群的管理实战 Kubeasz基于ezctl实现k8s集群一键升级 Calico启用纯BGP模式+RR实战案例 Calico 底层原理及IPIP（依赖BGP协议）)和vxlan（不依赖BGP）工作模式切换

尹正杰 · 2025-12-13 · via 博客园 - 尹正杰

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正杰

一.k8s高可用架构设计及基础环境准备
二.安装containerd组件及基础使用
- 1.Containerd环境部署
- 2.Containerd极速入门
三.安装K8S程序
- 1.下载软件包
- 2.解压K8S的二进制程序包到PATH环境变量路径
四.etcd高可用集群部署
五.生成k8s组件相关证书
六.启动K8S的master组件
七.高可用组件haproxy+keepalived安装及验证
八.启动K8S的worker组件
九.部署calico组件并验证测试
十.部署k8s集群附加组件

一.k8s高可用架构设计及基础环境准备

1.k8s高可用架构方案

kubernetes的高可用集群架构 (1)

如上图所示。

2.环境准备

主机名/IP地址	角色分配	硬件配置
k8s-cluster251/10.0.0.251	etcd，master，worker，haproxy，keepalived	2C/4G/50G+
k8s-cluster252/10.0.0.252	etcd，master，worker，haproxy，keepalived	2C/4G/50G+
k8s-cluster253/10.0.0.253	etcd，master，worker，haproxy，keepalived	2C/4G/50G+

硬件配置如上所示。

3.k8s版本说明

k8s 1.24重磅改动

k8s1.24+版本官方默认使用Containerd运行时，因此我们不许安装docker环境

4.所有节点安装常用的软件包

apt update
apt -y install bind9-utils expect rsync jq psmisc net-tools lvm2 vim unzip rename tree

5.k8s-master241节点免密钥登录集群并同步数据

	1 设置主机名,各节点参考如下命令修改即可
[root@k8s-cluster251 ~]# hostnamectl set-hostname k8s-cluster251

	2 设置相应的主机名及hosts文件解析
[root@k8s-cluster251 ~]# cat >> /etc/hosts <<'EOF'
10.0.0.240 apiserver-lb
10.0.0.251 k8s-cluster251
10.0.0.252 k8s-cluster252
10.0.0.253 k8s-cluster253
EOF

	3 配置免密码登录其他节点
[root@k8s-cluster251 ~]# cat > password_free_login.sh <<'EOF'
#!/bin/bash
# auther: Jason Yin

# 创建密钥对
ssh-keygen -t rsa -P "" -f /root/.ssh/id_rsa -q

# 声明你服务器密码，建议所有节点的密码均一致，否则该脚本需要再次进行优化
export mypasswd=1

# 定义主机列表
k8s_host_list=(k8s-cluster252 k8s-cluster253)

# 配置免密登录，利用expect工具免交互输入
for i in ${k8s_host_list[@]};do
expect -c "
spawn ssh-copy-id -i /root/.ssh/id_rsa.pub root@$i
  expect {
    \"*yes/no*\" {send \"yes\r\"; exp_continue}
    \"*password*\" {send \"$mypasswd\r\"; exp_continue}
  }"
done
EOF

[root@k8s-cluster251 ~]# bash password_free_login.sh 


	4 编写同步脚本
[root@k8s-cluster251 ~]# cat > /usr/local/sbin/data_rsync.sh <<'EOF'
#!/bin/bash

# Auther: Jason Yin

if  [ $# -lt 1 ];then
   echo "Usage: $0 /path/to/file(绝对路径) [mode: m|w]"
   exit
fi 

if [ ! -e $1 ];then
    echo "[ $1 ] dir or file not find!"
    exit
fi

fullpath=`dirname $1`

basename=`basename $1`

cd $fullpath

case $2 in
    WORKER_NODE|w)
      K8S_NODE=(k8s-cluster252 k8s-cluster253)
      ;;
    MASTER_NODE|m)
      K8S_NODE=(k8s-cluster252 k8s-cluster253)
      ;;
    *)
      K8S_NODE=(k8s-cluster252 k8s-cluster253)
     ;;
esac

for host in ${K8S_NODE[@]};do
  tput setaf 2
    echo ===== rsyncing ${host}: $basename =====
    tput setaf 7
    rsync -az $basename  `whoami`@${host}:$fullpath
    if [ $? -eq 0 ];then
      echo "命令执行成功!"
    fi
done
EOF


[root@k8s-cluster251 ~]# chmod +x /usr/local/sbin/data_rsync.sh 
[root@k8s-cluster251 ~]# 
[root@k8s-cluster251 ~]# ll /usr/local/sbin/data_rsync.sh 
-rwxr-xr-x 1 root root 725 Jan 26 09:28 /usr/local/sbin/data_rsync.sh*
[root@k8s-cluster251 ~]# 


	5 同步"/etc/hosts"文件到集群
[root@k8s-cluster251 ~]# data_rsync.sh /etc/hosts

6.所有节点Linux基础环境优化

	1 所有节点关闭NetworkManager，ufw
systemctl disable --now NetworkManager ufw 

	2 所有节点关闭swap分区，fstab注释swap
swapoff -a && sysctl -w vm.swappiness=0
sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab
free -h
	
	3 手动同步时区和时间
ln -svf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

	4 所有节点配置limit
cat >> /etc/security/limits.conf <<'EOF'
* soft nofile 655360
* hard nofile 131072
* soft nproc 655350
* hard nproc 655350
* soft memlock unlimited
* hard memlock unlimited
EOF

	5.所有节点优化sshd服务
sed -i 's@#UseDNS yes@UseDNS no@g' /etc/ssh/sshd_config
sed -i 's@^GSSAPIAuthentication yes@GSSAPIAuthentication no@g' /etc/ssh/sshd_config

	- UseDNS选项:
打开状态下，当客户端试图登录SSH服务器时，服务器端先根据客户端的IP地址进行DNS PTR反向查询出客户端的主机名，然后根据查询出的客户端主机名进行DNS正向A记录查询，验证与其原始IP地址是否一致，这是防止客户端欺骗的一种措施，但一般我们的是动态IP不会有PTR记录，打开这个选项不过是在白白浪费时间而已，不如将其关闭。

	- GSSAPIAuthentication:
当这个参数开启（ GSSAPIAuthentication  yes ）的时候，通过SSH登陆服务器时候会有些会很慢！这是由于服务器端启用了GSSAPI。登陆的时候客户端需要对服务器端的IP地址进行反解析，如果服务器的IP地址没有配置PTR记录，那么就容易在这里卡住了。


	6.Linux内核调优
cat > /etc/sysctl.d/k8s.conf <<'EOF'
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv6.conf.all.disable_ipv6 = 1
fs.may_detach_mounts = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384
EOF
sysctl --system

	7 修改终端颜色[可选]
cat <<EOF >>  ~/.bashrc 
PS1='[\[\e[34;1m\]\u@\[\e[0m\]\[\e[32;1m\]\H\[\e[0m\]\[\e[31;1m\] \W\[\e[0m\]]# '
EOF
source ~/.bashrc

7.所有节点安装ipvsadm以实现kube-proxy的负载均衡

	1 安装ipvsadm等相关工具
apt -y install ipvsadm ipset sysstat conntrack  

	2 所有节点创建要开机自动加载的模块配置文件
cat > /etc/modules-load.d/ipvs.conf << 'EOF'
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip
EOF

	3 修改ens33网卡名称为eth0【选做，建议修改，新手跳过】
		3.1 修改配置文件
vim /etc/default/grub
...
GRUB_CMDLINE_LINUX="... net.ifnames=0 biosdevname=0"

		3.2 用grub2-mkconfig重新生成配置 
grub-mkconfig -o /boot/grub/grub.cfg 

		3.3 修改网卡配置
[root@k8s-master01 ~]# cat /etc/netplan/00-installer-config.yaml
network:
  ethernets:
    eth0:
      dhcp4: false
      addresses:
        - 10.0.0.251/24
      routes:
        - to: default
          via: 10.0.0.254
      nameservers:
        addresses:
            # 114 DNS
          - 114.114.114.114
          - 114.114.115.115
            # 阿里云DNS
          - 223.5.5.5
          - 223.6.6.6
            # 腾讯云DNS
          - 119.29.29.29
          - 119.28.28.28
            # 百度DNS
          - 180.76.76.76
            # Google DNS
          - 8.8.8.8
          - 4.4.4.4
  version: 2
[root@k8s-master01 ~]# 

	4 重启操作系统即可
reboot 


	5.验证加载的模块
lsmod | grep --color=auto -e ip_vs -e nf_conntrack
uname -r
nproc  # 查看CPU核心数量
free -h
ifconfig

温馨提示:
	Linux kernel 4.19+版本已经将之前的"nf_conntrack_ipv4"模块更名为"nf_conntrack"模块哟~

二.安装containerd组件及基础使用

1.Containerd环境部署

	1.安装必要的一些系统工具
apt-get -y install apt-transport-https ca-certificates curl software-properties-common


	2.安装GPG证书
curl -fsSL https://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg | apt-key add -

	3.写入软件源信息
add-apt-repository "deb [arch=amd64] https://mirrors.aliyun.com/docker-ce/linux/ubuntu $(lsb_release -cs) stable"

	4.更新软件源
apt-get update

	5.安装containerd组件
apt-get -y install containerd.io


	6.配置containerd需要的模块
		6.1 临时手动加载模块
modprobe -- overlay
modprobe -- br_netfilter

		6.2.开机自动加载所需的内核模块
cat > /etc/modules-load.d/containerd.conf <<EOF
overlay
br_netfilter
EOF

	7. 修改containerd的配置文件
		7.1.重新初始化containerd的配置文件
containerd config default | tee /etc/containerd/config.toml 

		7.2 修改Cgroup的管理者为systemd组件
sed -ri 's#(SystemdCgroup = )false#\1true#' /etc/containerd/config.toml 
grep SystemdCgroup /etc/containerd/config.toml

		7.3 修改pause的基础镜像名称
sed -i 's#registry.k8s.io/pause:3.10.1#registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.10.1#' /etc/containerd/config.toml
grep sandbox /etc/containerd/config.toml


		7.4.修改Containerd的API版本兼容性
sed -i '/version/s#3#2#' /etc/containerd/config.toml
grep version /etc/containerd/config.toml


	8.所有节点启动containerd
		8.1 启动containerd服务
systemctl daemon-reload
systemctl enable --now containerd
systemctl status containerd

		8.2 配置crictl客户端连接的运行时位置
cat > /etc/crictl.yaml <<EOF
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF

		8.3 查看containerd的版本
[root@k8s-cluster251 ~]# ctr version
Client:
  Version:  v2.2.1
  Revision: dea7da592f5d1d2b7755e3a161be07f43fad8f75
  Go version: go1.24.11

Server:
  Version:  v2.2.1
  Revision: dea7da592f5d1d2b7755e3a161be07f43fad8f75
  UUID: 2dec7ac6-e2c0-44fe-80fb-cbc74c45d922
[root@k8s-cluster251 ~]# 



[root@k8s-cluster252 ~]# ctr version
Client:
  Version:  v2.2.1
  Revision: dea7da592f5d1d2b7755e3a161be07f43fad8f75
  Go version: go1.24.11

Server:
  Version:  v2.2.1
  Revision: dea7da592f5d1d2b7755e3a161be07f43fad8f75
  UUID: 5617c300-bd4a-4ebc-ad01-c06eea7394f1
[root@k8s-cluster252 ~]# 



[root@k8s-cluster253 ~]# ctr version
Client:
  Version:  v2.2.1
  Revision: dea7da592f5d1d2b7755e3a161be07f43fad8f75
  Go version: go1.24.11

Server:
  Version:  v2.2.1
  Revision: dea7da592f5d1d2b7755e3a161be07f43fad8f75
  UUID: 0d6aa0ca-79d6-4d2b-9b68-13239a618eea
[root@k8s-cluster253 ~]#

2.Containerd极速入门

温馨提示:
	containerd本身并不提供网络，只负责容器的生命周期。
	将来网络部分交给K8S的CNI插件实现即可。
	
	
更多containerd学习资料推荐：
	https://www.cnblogs.com/yinzhengjie/p/18030527
	https://www.cnblogs.com/yinzhengjie/p/18058010

三.安装K8S程序

1.下载软件包

wget https://dl.k8s.io/v1.35.0/kubernetes-server-linux-amd64.tar.gz

2.解压K8S的二进制程序包到PATH环境变量路径

	1 解压软件包
[root@k8s-cluster251 ~]# tar -xf kubernetes-server-linux-amd64-v1.35.0.tar.gz  --strip-components=3 -C /usr/local/bin kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}


	2 查看kubelet的版本
[root@k8s-cluster251 ~]# kubelet --version
Kubernetes v1.35.0
[root@k8s-cluster251 ~]

	3.分发软件包
[root@k8s-cluster251 ~]# scp /usr/local/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} k8s-cluster252:/usr/local/bin


[root@k8s-cluster251 ~]# scp /usr/local/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} k8s-cluster253:/usr/local/bin

四.etcd高可用集群部署

推荐阅读:
	https://www.cnblogs.com/yinzhengjie/p/19316861

五.生成k8s组件相关证书

1.所有节点创建k8s证书存储目录

mkdir -pv /yinzhengjie/certs/kubernetes/

2.k8s-cluster251节点生成kubernetes自建ca证书

	1.生成证书的CSR文件： 证书签发请求文件，配置了一些域名，公司，单位
[root@k8s-cluster251 ~]# cd /yinzhengjie/certs/pki
[root@k8s-cluster251 pki]# 
[root@k8s-cluster251 pki]# cat > k8s-ca-csr.json  <<EOF
{
"CN": "kubernetes",
"key": {
  "algo": "rsa",
  "size": 2048
},
"names": [
  {
    "C": "CN",
    "ST": "Beijing",
    "L": "Beijing",
    "O": "Kubernetes",
    "OU": "Kubernetes-manual"
  }
],
"ca": {
  "expiry": "876000h"
}
}
EOF


	2 生成kubernetes证书
[root@k8s-cluster251 pki]# cfssl gencert -initca k8s-ca-csr.json | cfssljson -bare /yinzhengjie/certs/kubernetes/k8s-ca
2026/01/28 19:18:51 [INFO] generating a new CA key and certificate from CSR
2026/01/28 19:18:51 [INFO] generate received request
2026/01/28 19:18:51 [INFO] received CSR
2026/01/28 19:18:51 [INFO] generating key: rsa-2048
2026/01/28 19:18:51 [INFO] encoded CSR
2026/01/28 19:18:51 [INFO] signed certificate with serial number 575203929559675667578446704322282218656920577309
[root@k8s-cluster251 pki]# 
[root@k8s-cluster251 pki]# ll /yinzhengjie/certs/kubernetes/k8s-ca*
-rw-r--r-- 1 root root 1070 Jan 28 19:18 /yinzhengjie/certs/kubernetes/k8s-ca.csr
-rw------- 1 root root 1679 Jan 28 19:18 /yinzhengjie/certs/kubernetes/k8s-ca-key.pem
-rw-r--r-- 1 root root 1363 Jan 28 19:18 /yinzhengjie/certs/kubernetes/k8s-ca.pem
[root@k8s-cluster251 pki]#

3.k8s-cluster251节点基于自建ca证书颁发apiserver相关证书

	1 生成k8s证书的有效期为100年
[root@k8s-cluster251 pki]# cat > k8s-ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "876000h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "876000h"
      }
    }
  }
}
EOF


	2 生成apiserver证书的CSR文件： 证书签发请求文件，配置了一些域名，公司，单位
[root@k8s-cluster251 pki]# cat > apiserver-csr.json <<EOF
{
  "CN": "kube-apiserver",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "Kubernetes",
      "OU": "Kubernetes-manual"
    }
  ]
}
EOF


	3 基于自建ca证书生成apiServer的证书文件
[root@k8s-cluster251 pki]# cfssl gencert \
  -ca=/yinzhengjie/certs/kubernetes/k8s-ca.pem \
  -ca-key=/yinzhengjie/certs/kubernetes/k8s-ca-key.pem \
  -config=k8s-ca-config.json \
  --hostname=10.200.0.1,10.0.0.240,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.yinzhengjie,kubernetes.default.svc.yinzhengjie.com,10.0.0.251,10.0.0.252,10.0.0.253,apiserver-lb \
  --profile=kubernetes \
   apiserver-csr.json  | cfssljson -bare /yinzhengjie/certs/kubernetes/apiserver
2026/01/28 19:22:10 [INFO] generate received request
2026/01/28 19:22:10 [INFO] received CSR
2026/01/28 19:22:10 [INFO] generating key: rsa-2048
2026/01/28 19:22:10 [INFO] encoded CSR
2026/01/28 19:22:10 [INFO] signed certificate with serial number 508509839273929914832912652928152106393726388433
[root@k8s-cluster251 pki]# 
[root@k8s-cluster251 pki]# ll /yinzhengjie/certs/kubernetes/apiserver*
-rw-r--r-- 1 root root 1318 Jan 28 19:22 /yinzhengjie/certs/kubernetes/apiserver.csr
-rw------- 1 root root 1675 Jan 28 19:22 /yinzhengjie/certs/kubernetes/apiserver-key.pem
-rw-r--r-- 1 root root 1712 Jan 28 19:22 /yinzhengjie/certs/kubernetes/apiserver.pem
[root@k8s-cluster251 pki]# 



温馨提示:
	"10.200.0.1"为咱们的svc网段的第一个地址，您需要根据自己的场景稍作修改。
	"10.0.0.240"是负载均衡器的VIP地址。
	"kubernetes,...,kubernetes.default.svc.yinzhengjie.com"对应的是apiServer解析的A记录。
	"10.0.0.251,...,10.0.0.253"对应的是K8S集群的地址。

4.生成第三方组件与apiServer通信的聚合证书

聚合证书的作用就是让第三方组件(比如metrics-server等)能够拿这个证书文件和apiServer进行通信。
	
	1 生成聚合证书的用于自建ca的CSR文件
[root@k8s-cluster251 pki]# cat > front-proxy-ca-csr.json <<EOF
{
  "CN": "kubernetes",
  "key": {
     "algo": "rsa",
     "size": 2048
  }
}
EOF

	2 生成聚合证书的自建ca证书
[root@k8s-cluster251 pki]# cfssl gencert -initca front-proxy-ca-csr.json | cfssljson -bare /yinzhengjie/certs/kubernetes/front-proxy-ca
2026/01/28 19:22:38 [INFO] generating a new CA key and certificate from CSR
2026/01/28 19:22:38 [INFO] generate received request
2026/01/28 19:22:38 [INFO] received CSR
2026/01/28 19:22:38 [INFO] generating key: rsa-2048
2026/01/28 19:22:38 [INFO] encoded CSR
2026/01/28 19:22:38 [INFO] signed certificate with serial number 30590855247650508796790460015911903490720686144
[root@k8s-cluster251 pki]# 
[root@k8s-cluster251 pki]# ll /yinzhengjie/certs/kubernetes/front-proxy-ca*
-rw-r--r-- 1 root root  891 Jan 28 19:22 /yinzhengjie/certs/kubernetes/front-proxy-ca.csr
-rw------- 1 root root 1679 Jan 28 19:22 /yinzhengjie/certs/kubernetes/front-proxy-ca-key.pem
-rw-r--r-- 1 root root 1094 Jan 28 19:22 /yinzhengjie/certs/kubernetes/front-proxy-ca.pem
[root@k8s-cluster251 pki]# 



	3.生成聚合证书的用于客户端的CSR文件
[root@k8s-cluster251 pki]# cat > front-proxy-client-csr.json <<EOF
{
  "CN": "front-proxy-client",
  "key": {
     "algo": "rsa",
     "size": 2048
  }
}
EOF


	4 基于聚合证书的自建ca证书签发聚合证书的客户端证书
[root@k8s-cluster251 pki]# cfssl gencert \
  -ca=/yinzhengjie/certs/kubernetes/front-proxy-ca.pem \
  -ca-key=/yinzhengjie/certs/kubernetes/front-proxy-ca-key.pem \
  -config=k8s-ca-config.json \
  -profile=kubernetes \
  front-proxy-client-csr.json | cfssljson -bare /yinzhengjie/certs/kubernetes/front-proxy-client
2026/01/28 19:23:04 [INFO] generate received request
2026/01/28 19:23:04 [INFO] received CSR
2026/01/28 19:23:04 [INFO] generating key: rsa-2048
2026/01/28 19:23:04 [INFO] encoded CSR
2026/01/28 19:23:04 [INFO] signed certificate with serial number 208923816469886410948326544551883807194985271473
2026/01/28 19:23:04 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@k8s-cluster251 pki]# 
[root@k8s-cluster251 pki]# ll /yinzhengjie/certs/kubernetes/front-proxy-client*
-rw-r--r-- 1 root root  903 Jan 28 19:23 /yinzhengjie/certs/kubernetes/front-proxy-client.csr
-rw------- 1 root root 1675 Jan 28 19:23 /yinzhengjie/certs/kubernetes/front-proxy-client-key.pem
-rw-r--r-- 1 root root 1188 Jan 28 19:23 /yinzhengjie/certs/kubernetes/front-proxy-client.pem
[root@k8s-cluster251 pki]#

5.生成controller-manager证书及kubeconfig文件

	1 生成controller-manager的CSR文件
[root@k8s-cluster251 pki]# cat > controller-manager-csr.json <<EOF
{
  "CN": "system:kube-controller-manager",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "system:kube-controller-manager",
      "OU": "Kubernetes-manual"
    }
  ]
}
EOF


	2 生成controller-manager证书文件
[root@k8s-cluster251 pki]# cfssl gencert \
  -ca=/yinzhengjie/certs/kubernetes/k8s-ca.pem \
  -ca-key=/yinzhengjie/certs/kubernetes/k8s-ca-key.pem \
  -config=k8s-ca-config.json \
  -profile=kubernetes \
  controller-manager-csr.json | cfssljson -bare /yinzhengjie/certs/kubernetes/controller-manager
2026/01/28 19:23:36 [INFO] generate received request
2026/01/28 19:23:36 [INFO] received CSR
2026/01/28 19:23:36 [INFO] generating key: rsa-2048
2026/01/28 19:23:36 [INFO] encoded CSR
2026/01/28 19:23:36 [INFO] signed certificate with serial number 361208340266004949124452417972720740751507960714
2026/01/28 19:23:36 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@k8s-cluster251 pki]# 
[root@k8s-cluster251 pki]# ll /yinzhengjie/certs/kubernetes/controller-manager*
-rw-r--r-- 1 root root 1082 Jan 28 19:23 /yinzhengjie/certs/kubernetes/controller-manager.csr
-rw------- 1 root root 1675 Jan 28 19:23 /yinzhengjie/certs/kubernetes/controller-manager-key.pem
-rw-r--r-- 1 root root 1501 Jan 28 19:23 /yinzhengjie/certs/kubernetes/controller-manager.pem
[root@k8s-cluster251 pki]# 


	3 创建一个kubeconfig目录
[root@k8s-cluster251 pki]# mkdir -pv /yinzhengjie/certs/kubeconfig


	4 设置一个集群
[root@k8s-cluster251 pki]# kubectl config set-cluster yinzhengjie-k8s \
  --certificate-authority=/yinzhengjie/certs/kubernetes/k8s-ca.pem \
  --embed-certs=true \
  --server=https://10.0.0.240:8443 \
  --kubeconfig=/yinzhengjie/certs/kubeconfig/kube-controller-manager.kubeconfig
		
	5 设置一个用户项
[root@k8s-cluster251 pki]# kubectl config set-credentials system:kube-controller-manager \
  --client-certificate=/yinzhengjie/certs/kubernetes/controller-manager.pem \
  --client-key=/yinzhengjie/certs/kubernetes/controller-manager-key.pem \
  --embed-certs=true \
  --kubeconfig=/yinzhengjie/certs/kubeconfig/kube-controller-manager.kubeconfig
  
	6 设置一个上下文环境
[root@k8s-cluster251 pki]# kubectl config set-context system:kube-controller-manager@kubernetes \
  --cluster=yinzhengjie-k8s \
  --user=system:kube-controller-manager \
  --kubeconfig=/yinzhengjie/certs/kubeconfig/kube-controller-manager.kubeconfig
  
	7 使用默认的上下文
[root@k8s-cluster251 pki]# kubectl config use-context system:kube-controller-manager@kubernetes \
  --kubeconfig=/yinzhengjie/certs/kubeconfig/kube-controller-manager.kubeconfig

6.生成scheduler证书及kubeconfig文件

	1 生成scheduler的CSR文件
[root@k8s-cluster251 pki]# cat > scheduler-csr.json <<EOF
{
  "CN": "system:kube-scheduler",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "system:kube-scheduler",
      "OU": "Kubernetes-manual"
    }
  ]
}
EOF


	2 生成scheduler证书文件
[root@k8s-cluster251 pki]# cfssl gencert \
  -ca=/yinzhengjie/certs/kubernetes/k8s-ca.pem \
  -ca-key=/yinzhengjie/certs/kubernetes/k8s-ca-key.pem \
  -config=k8s-ca-config.json \
  -profile=kubernetes \
  scheduler-csr.json | cfssljson -bare /yinzhengjie/certs/kubernetes/scheduler
2026/01/28 19:25:17 [INFO] generate received request
2026/01/28 19:25:17 [INFO] received CSR
2026/01/28 19:25:17 [INFO] generating key: rsa-2048
2026/01/28 19:25:17 [INFO] encoded CSR
2026/01/28 19:25:17 [INFO] signed certificate with serial number 110476601892341998899236081556261796844273222747
2026/01/28 19:25:17 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@k8s-cluster251 pki]# 
[root@k8s-cluster251 pki]# 
[root@k8s-cluster251 pki]# ll /yinzhengjie/certs/kubernetes/scheduler*
-rw-r--r-- 1 root root 1058 Jan 28 19:25 /yinzhengjie/certs/kubernetes/scheduler.csr
-rw------- 1 root root 1679 Jan 28 19:25 /yinzhengjie/certs/kubernetes/scheduler-key.pem
-rw-r--r-- 1 root root 1476 Jan 28 19:25 /yinzhengjie/certs/kubernetes/scheduler.pem
[root@k8s-cluster251 pki]# 


	3.设置一个集群
[root@k8s-cluster251 pki]# kubectl config set-cluster yinzhengjie-k8s \
  --certificate-authority=/yinzhengjie/certs/kubernetes/k8s-ca.pem \
  --embed-certs=true \
  --server=https://10.0.0.240:8443 \
  --kubeconfig=/yinzhengjie/certs/kubeconfig/kube-scheduler.kubeconfig
  
  
   4 设置一个用户项
[root@k8s-cluster251 pki]# kubectl config set-credentials system:kube-scheduler \
  --client-certificate=/yinzhengjie/certs/kubernetes/scheduler.pem \
  --client-key=/yinzhengjie/certs/kubernetes/scheduler-key.pem \
  --embed-certs=true \
  --kubeconfig=/yinzhengjie/certs/kubeconfig/kube-scheduler.kubeconfig


	5.设置一个上下文环境
[root@k8s-cluster251 pki]# kubectl config set-context system:kube-scheduler@kubernetes \
  --cluster=yinzhengjie-k8s \
  --user=system:kube-scheduler \
  --kubeconfig=/yinzhengjie/certs/kubeconfig/kube-scheduler.kubeconfig
  
	6 使用默认的上下文
[root@k8s-cluster251 pki]# kubectl config use-context system:kube-scheduler@kubernetes \
  --kubeconfig=/yinzhengjie/certs/kubeconfig/kube-scheduler.kubeconfig

7.生成kube-proxy证书及kubeconfig文件

	1.生成kube-proxy的csr文件
[root@k8s-cluster251 pki]# cat > kube-proxy-csr.json  <<EOF
{
"CN": "system:kube-proxy",
"key": {
  "algo": "rsa",
  "size": 2048
},
"names": [
  {
    "C": "CN",
    "ST": "Beijing",
    "L": "Beijing",
    "O": "system:kube-proxy",
    "OU": "Kubernetes-manual"
  }
]
}
EOF


	2.创建kube-proxy需要的证书文件
[root@k8s-cluster251 pki]#  cfssl gencert \
-ca=/yinzhengjie/certs/kubernetes/k8s-ca.pem \
-ca-key=/yinzhengjie/certs/kubernetes/k8s-ca-key.pem \
-config=k8s-ca-config.json \
-profile=kubernetes \
kube-proxy-csr.json | cfssljson -bare /yinzhengjie/certs/kubernetes/kube-proxy
2026/01/28 19:35:04 [INFO] generate received request
2026/01/28 19:35:04 [INFO] received CSR
2026/01/28 19:35:04 [INFO] generating key: rsa-2048
2026/01/28 19:35:04 [INFO] encoded CSR
2026/01/28 19:35:04 [INFO] signed certificate with serial number 546682827339465170367267668689228021194287145271
2026/01/28 19:35:04 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@k8s-cluster251 pki]# 
[root@k8s-cluster251 pki]# ll /yinzhengjie/certs/kubernetes/kube-proxy*
-rw-r--r-- 1 root root 1045 Jan 28 19:35 /yinzhengjie/certs/kubernetes/kube-proxy.csr
-rw------- 1 root root 1679 Jan 28 19:35 /yinzhengjie/certs/kubernetes/kube-proxy-key.pem
-rw-r--r-- 1 root root 1464 Jan 28 19:35 /yinzhengjie/certs/kubernetes/kube-proxy.pem
[root@k8s-cluster251 pki]# 



	3.设置集群
[root@k8s-cluster251 pki]# kubectl config set-cluster yinzhengjie-k8s \
  --certificate-authority=/yinzhengjie/certs/kubernetes/k8s-ca.pem \
  --embed-certs=true \
  --server=https://10.0.0.240:8443 \
  --kubeconfig=/yinzhengjie/certs/kubeconfig/kube-proxy.kubeconfig
  
  
	4.设置一个用户项
[root@k8s-cluster251 pki]# kubectl config set-credentials system:kube-proxy \
  --client-certificate=/yinzhengjie/certs/kubernetes/kube-proxy.pem \
  --client-key=/yinzhengjie/certs/kubernetes/kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=/yinzhengjie/certs/kubeconfig/kube-proxy.kubeconfig

	5.设置一个上下文环境
[root@k8s-cluster251 pki]# kubectl config set-context kube-proxy@kubernetes \
  --cluster=yinzhengjie-k8s \
  --user=system:kube-proxy \
  --kubeconfig=/yinzhengjie/certs/kubeconfig/kube-proxy.kubeconfig

	6.使用默认的上下文
[root@k8s-cluster251 pki]# kubectl config use-context kube-proxy@kubernetes \
  --kubeconfig=/yinzhengjie/certs/kubeconfig/kube-proxy.kubeconfig

8.配置k8s集群管理员证书及kubeconfig文件

	1 生成管理员的CSR文件
[root@k8s-cluster251 pki]#  cat > admin-csr.json <<EOF
{
  "CN": "admin",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "system:masters",
      "OU": "Kubernetes-manual"
    }
  ]
}
EOF

	2.生成k8s集群管理员证书
[root@k8s-cluster251 pki]# cfssl gencert \
  -ca=/yinzhengjie/certs/kubernetes/k8s-ca.pem \
  -ca-key=/yinzhengjie/certs/kubernetes/k8s-ca-key.pem \
  -config=k8s-ca-config.json \
  -profile=kubernetes \
  admin-csr.json | cfssljson -bare /yinzhengjie/certs/kubernetes/admin
2026/01/28 19:26:20 [INFO] generate received request
2026/01/28 19:26:20 [INFO] received CSR
2026/01/28 19:26:20 [INFO] generating key: rsa-2048
2026/01/28 19:26:20 [INFO] encoded CSR
2026/01/28 19:26:20 [INFO] signed certificate with serial number 577503158797464741908387723052964628687800336068
2026/01/28 19:26:20 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@k8s-cluster251 pki]# 
[root@k8s-cluster251 pki]# 
[root@k8s-cluster251 pki]# ll /yinzhengjie/certs/kubernetes/admin*
-rw-r--r-- 1 root root 1025 Jan 28 19:26 /yinzhengjie/certs/kubernetes/admin.csr
-rw------- 1 root root 1679 Jan 28 19:26 /yinzhengjie/certs/kubernetes/admin-key.pem
-rw-r--r-- 1 root root 1444 Jan 28 19:26 /yinzhengjie/certs/kubernetes/admin.pem
[root@k8s-cluster251 pki]# 

	3.设置一个集群
[root@k8s-cluster251 pki]# kubectl config set-cluster yinzhengjie-k8s \
  --certificate-authority=/yinzhengjie/certs/kubernetes/k8s-ca.pem \
  --embed-certs=true \
  --server=https://10.0.0.240:8443 \
  --kubeconfig=/yinzhengjie/certs/kubeconfig/kube-admin.kubeconfig
  
	4 设置一个用户项
[root@k8s-cluster251 pki]#  kubectl config set-credentials kube-admin \
  --client-certificate=/yinzhengjie/certs/kubernetes/admin.pem \
  --client-key=/yinzhengjie/certs/kubernetes/admin-key.pem \
  --embed-certs=true \
  --kubeconfig=/yinzhengjie/certs/kubeconfig/kube-admin.kubeconfig

	5 设置一个上下文环境
[root@k8s-cluster251 pki]# kubectl config set-context kube-admin@kubernetes \
  --cluster=yinzhengjie-k8s \
  --user=kube-admin \
  --kubeconfig=/yinzhengjie/certs/kubeconfig/kube-admin.kubeconfig
  
	6.使用默认的上下文
[root@k8s-cluster251 pki]# kubectl config use-context kube-admin@kubernetes \
  --kubeconfig=/yinzhengjie/certs/kubeconfig/kube-admin.kubeconfig

9.创建ServiceAccount

	1.ServiceAccount是k8s一种认证方式，创建ServiceAccount的时候会创建一个与之绑定的secret，这个secret会生成一个token
[root@k8s-master01 pki]# openssl genrsa -out /yinzhengjie/certs/kubernetes/sa.key 2048


	2.基于sa.key创建sa.pub
[root@k8s-cluster251 pki]# openssl rsa -in /yinzhengjie/certs/kubernetes/sa.key -pubout -out /yinzhengjie/certs/kubernetes/sa.pub
writing RSA key
[root@k8s-cluster251 pki]# 
[root@k8s-cluster251 pki]# ll /yinzhengjie/certs/kubernetes/sa*
-rw------- 1 root root 1704 Jan 28 19:27 /yinzhengjie/certs/kubernetes/sa.key
-rw-r--r-- 1 root root  451 Jan 28 19:27 /yinzhengjie/certs/kubernetes/sa.pub
[root@k8s-cluster251 pki]#

10.k8s-master01节点K8S组件证书拷贝到其他两个master节点

	1 k8s-master01节点将etcd证书拷贝到其他两个master节点
[root@k8s-cluster251 pki]# data_rsync.sh  /yinzhengjie/certs/kubernetes
===== rsyncing k8s-cluster252: kubernetes =====
命令执行成功!
===== rsyncing k8s-cluster253: kubernetes =====
命令执行成功!
[root@k8s-cluster251 pki]# 
[root@k8s-cluster251 pki]# data_rsync.sh  /yinzhengjie/certs/kubeconfig/
===== rsyncing k8s-cluster252: kubeconfig =====
命令执行成功!
===== rsyncing k8s-cluster253: kubeconfig =====
命令执行成功!
[root@k8s-cluster251 pki]# 


	2 其他两个节点验证文件数量是否正确
[root@k8s-cluster252 ~]# ls /yinzhengjie/certs/kubernetes  | wc -l
26
[root@k8s-cluster252 ~]# 
[root@k8s-cluster252 ~]# ls /yinzhengjie/certs/kubeconfig/  | wc -l
4
[root@k8s-cluster252 ~]# 



[root@k8s-cluster253 ~]# ls /yinzhengjie/certs/kubernetes  | wc -l
26
[root@k8s-cluster253 ~]# 
[root@k8s-cluster253 ~]# ls /yinzhengjie/certs/kubeconfig/  | wc -l
4
[root@k8s-cluster253 ~]#

六.启动K8S的master组件

1.部署ApiServer组件

温馨提示:
  - "--advertise-address"是对应的master节点的IP地址;
  - "--service-cluster-ip-range"对应的是svc的网段
  - "--service-node-port-range"对应的是svc的NodePort端口范围;
  - "--etcd-servers"指定的是etcd集群地址

配置文件参考链接:
	https://kubernetes.io/zh-cn/docs/reference/command-line-tools-reference/kube-apiserver/
	

具体实操:
	1 创建k8s-cluster251节点的配置文件
[root@k8s-cluster251 ~]# cat > /usr/lib/systemd/system/kube-apiserver.service << 'EOF'
[Unit]
Description=Jason Yin's Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-apiserver \
      --v=2  \
      --bind-address=0.0.0.0  \
      --secure-port=6443  \
      --allow_privileged=true \
      --advertise-address=10.0.0.251 \
      --service-cluster-ip-range=10.200.0.0/16  \
      --service-node-port-range=3000-50000  \
      --etcd-servers= https://10.0.0.251:2379,https://10.0.0.252:2379 ,https://10.0.0.253:2379 \
      --etcd-cafile=/yinzhengjie/certs/etcd/etcd-ca.pem  \
      --etcd-certfile=/yinzhengjie/certs/etcd/etcd-server.pem  \
      --etcd-keyfile=/yinzhengjie/certs/etcd/etcd-server-key.pem  \
      --client-ca-file=/yinzhengjie/certs/kubernetes/k8s-ca.pem  \
      --tls-cert-file=/yinzhengjie/certs/kubernetes/apiserver.pem  \
      --tls-private-key-file=/yinzhengjie/certs/kubernetes/apiserver-key.pem  \
      --kubelet-client-certificate=/yinzhengjie/certs/kubernetes/apiserver.pem  \
      --kubelet-client-key=/yinzhengjie/certs/kubernetes/apiserver-key.pem  \
      --service-account-key-file=/yinzhengjie/certs/kubernetes/sa.pub  \
      --service-account-signing-key-file=/yinzhengjie/certs/kubernetes/sa.key \
      --service-account-issuer=https://kubernetes.default.svc.yinzhengjie.com \
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname  \
      --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota  \
      --authorization-mode=Node,RBAC  \
      --enable-bootstrap-token-auth=true  \
      --requestheader-client-ca-file=/yinzhengjie/certs/kubernetes/front-proxy-ca.pem  \
      --proxy-client-cert-file=/yinzhengjie/certs/kubernetes/front-proxy-client.pem  \
      --proxy-client-key-file=/yinzhengjie/certs/kubernetes/front-proxy-client-key.pem  \
      --requestheader-allowed-names=aggregator,front-proxy-client  \
      --requestheader-group-headers=X-Remote-Group  \
      --requestheader-extra-headers-prefix=X-Remote-Extra-  \
      --requestheader-username-headers=X-Remote-User

Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target
EOF


	2 创建k8s-cluster252节点的配置文件
[root@k8s-cluster252 ~]# cat > /usr/lib/systemd/system/kube-apiserver.service << 'EOF'
[Unit]
Description=Jason Yin's Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-apiserver \
      --v=2  \
      --bind-address=0.0.0.0  \
      --secure-port=6443  \
      --allow_privileged=true \
      --advertise-address=10.0.0.252 \
      --service-cluster-ip-range=10.200.0.0/16  \
      --service-node-port-range=3000-50000  \
      --etcd-servers= https://10.0.0.251:2379,https://10.0.0.252:2379 ,https://10.0.0.253:2379 \
      --etcd-cafile=/yinzhengjie/certs/etcd/etcd-ca.pem  \
      --etcd-certfile=/yinzhengjie/certs/etcd/etcd-server.pem  \
      --etcd-keyfile=/yinzhengjie/certs/etcd/etcd-server-key.pem  \
      --client-ca-file=/yinzhengjie/certs/kubernetes/k8s-ca.pem  \
      --tls-cert-file=/yinzhengjie/certs/kubernetes/apiserver.pem  \
      --tls-private-key-file=/yinzhengjie/certs/kubernetes/apiserver-key.pem  \
      --kubelet-client-certificate=/yinzhengjie/certs/kubernetes/apiserver.pem  \
      --kubelet-client-key=/yinzhengjie/certs/kubernetes/apiserver-key.pem  \
      --service-account-key-file=/yinzhengjie/certs/kubernetes/sa.pub  \
      --service-account-signing-key-file=/yinzhengjie/certs/kubernetes/sa.key \
      --service-account-issuer=https://kubernetes.default.svc.yinzhengjie.com \
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname  \
      --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota  \
      --authorization-mode=Node,RBAC  \
      --enable-bootstrap-token-auth=true  \
      --requestheader-client-ca-file=/yinzhengjie/certs/kubernetes/front-proxy-ca.pem  \
      --proxy-client-cert-file=/yinzhengjie/certs/kubernetes/front-proxy-client.pem  \
      --proxy-client-key-file=/yinzhengjie/certs/kubernetes/front-proxy-client-key.pem  \
      --requestheader-allowed-names=aggregator,front-proxy-client  \
      --requestheader-group-headers=X-Remote-Group  \
      --requestheader-extra-headers-prefix=X-Remote-Extra-  \
      --requestheader-username-headers=X-Remote-User

Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target
EOF



	3 创建k8s-cluster253节点的配置文件
[root@k8s-cluster253 ~]# cat > /usr/lib/systemd/system/kube-apiserver.service << 'EOF'
[Unit]
Description=Jason Yin's Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-apiserver \
      --v=2  \
      --bind-address=0.0.0.0  \
      --secure-port=6443  \
      --allow_privileged=true \
      --advertise-address=10.0.0.253 \
      --service-cluster-ip-range=10.200.0.0/16  \
      --service-node-port-range=3000-50000  \
      --etcd-servers= https://10.0.0.251:2379,https://10.0.0.252:2379 ,https://10.0.0.253:2379 \
      --etcd-cafile=/yinzhengjie/certs/etcd/etcd-ca.pem  \
      --etcd-certfile=/yinzhengjie/certs/etcd/etcd-server.pem  \
      --etcd-keyfile=/yinzhengjie/certs/etcd/etcd-server-key.pem  \
      --client-ca-file=/yinzhengjie/certs/kubernetes/k8s-ca.pem  \
      --tls-cert-file=/yinzhengjie/certs/kubernetes/apiserver.pem  \
      --tls-private-key-file=/yinzhengjie/certs/kubernetes/apiserver-key.pem  \
      --kubelet-client-certificate=/yinzhengjie/certs/kubernetes/apiserver.pem  \
      --kubelet-client-key=/yinzhengjie/certs/kubernetes/apiserver-key.pem  \
      --service-account-key-file=/yinzhengjie/certs/kubernetes/sa.pub  \
      --service-account-signing-key-file=/yinzhengjie/certs/kubernetes/sa.key \
      --service-account-issuer=https://kubernetes.default.svc.yinzhengjie.com \
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname  \
      --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota  \
      --authorization-mode=Node,RBAC  \
      --enable-bootstrap-token-auth=true  \
      --requestheader-client-ca-file=/yinzhengjie/certs/kubernetes/front-proxy-ca.pem  \
      --proxy-client-cert-file=/yinzhengjie/certs/kubernetes/front-proxy-client.pem  \
      --proxy-client-key-file=/yinzhengjie/certs/kubernetes/front-proxy-client-key.pem  \
      --requestheader-allowed-names=aggregator,front-proxy-client  \
      --requestheader-group-headers=X-Remote-Group  \
      --requestheader-extra-headers-prefix=X-Remote-Extra-  \
      --requestheader-username-headers=X-Remote-User

Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target
EOF


	4 所有节点启动服务
systemctl daemon-reload && systemctl enable --now kube-apiserver
systemctl status kube-apiserver
ss -ntl | grep 6443

2.部署ControlerManager组件

温馨提示:

- "--cluster-cidr"是Pod的网段地址，我们可以自行修改。

配置文件参考链接:
	https://kubernetes.io/zh-cn/docs/reference/command-line-tools-reference/kube-controller-manager/
	 

	1.所有节点的controller-manager组件配置文件相同: (前提是证书文件存放的位置也要相同哟!)
cat > /usr/lib/systemd/system/kube-controller-manager.service << 'EOF'
[Unit]
Description=Jason Yin's Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-controller-manager \
      --v=2 \
      --root-ca-file=/yinzhengjie/certs/kubernetes/k8s-ca.pem \
      --cluster-signing-cert-file=/yinzhengjie/certs/kubernetes/k8s-ca.pem \
      --cluster-signing-key-file=/yinzhengjie/certs/kubernetes/k8s-ca-key.pem \
      --service-account-private-key-file=/yinzhengjie/certs/kubernetes/sa.key \
      --kubeconfig=/yinzhengjie/certs/kubeconfig/kube-controller-manager.kubeconfig \
      --leader-elect=true \
      --use-service-account-credentials=true \
      --node-monitor-grace-period=40s \
      --node-monitor-period=5s \
      --controllers=*,bootstrapsigner,tokencleaner \
      --allocate-node-cidrs=true \
      --cluster-cidr=10.100.0.0/16 \
      --requestheader-client-ca-file=/yinzhengjie/certs/kubernetes/front-proxy-ca.pem \
      --node-cidr-mask-size=24 \
      --cluster-signing-duration=87600h0m0s \
      --feature-gates=RotateKubeletServerCertificate=true
      
Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target
EOF


	2.启动controller-manager服务
systemctl daemon-reload
systemctl enable --now kube-controller-manager
systemctl  status kube-controller-manager
ss -ntl | grep 10257

3.部署Scheduler组件

	1 所有节点创建配置文件
配置文件参考链接:
https://kubernetes.io/zh-cn/docs/reference/command-line-tools-reference/kube-scheduler/

所有节点的controller-manager组件配置文件相同: (前提是证书文件存放的位置也要相同哟!)
cat > /usr/lib/systemd/system/kube-scheduler.service <<'EOF'
[Unit]
Description=Jason Yin's Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-scheduler \
      --v=2 \
      --leader-elect=true \
      --kubeconfig=/yinzhengjie/certs/kubeconfig/kube-scheduler.kubeconfig

Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target
EOF


	2 启动scheduler服务
systemctl daemon-reload
systemctl enable --now kube-scheduler
systemctl  status kube-scheduler
ss -ntl | grep 10259

4.测试组件是否正常

	1.临时修改api-server的地址和端口
[root@k8s-cluster251 ~]# grep server  /yinzhengjie/certs/kubeconfig/kube-admin.kubeconfig
    # server: https://10.0.0.240:8443
    server: https://10.0.0.251:6443
[root@k8s-cluster251 ~]# 


	2.临时测试集群是否可用
[root@k8s-cluster251 ~]# kubectl get cs --kubeconfig=/yinzhengjie/certs/kubeconfig/kube-admin.kubeconfig
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE   ERROR
scheduler            Healthy   ok        
controller-manager   Healthy   ok        
etcd-0               Healthy   ok        
[root@k8s-cluster251 ~]# 
[root@k8s-cluster251 ~]# kubectl get nodes --kubeconfig=/yinzhengjie/certs/kubeconfig/kube-admin.kubeconfig
No resources found
[root@k8s-cluster251 ~]# 



	3.将配置还原
[root@k8s-cluster251 ~]# grep server  /yinzhengjie/certs/kubeconfig/kube-admin.kubeconfig
    server: https://10.0.0.240:8443
    # server: https://10.0.0.251:6443
[root@k8s-cluster251 ~]# 
[root@k8s-cluster251 ~]# kubectl get cs --kubeconfig=/yinzhengjie/certs/kubeconfig/kube-admin.kubeconfig
E0126 12:12:16.875493    4103 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://10.0.0.240:8443/api?timeout=32s\": dial tcp 10.0.0.240:8443: connect: no route to host"
E0126 12:12:19.943112    4103 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://10.0.0.240:8443/api?timeout=32s\": dial tcp 10.0.0.240:8443: connect: no route to host"
E0126 12:12:23.015217    4103 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://10.0.0.240:8443/api?timeout=32s\": dial tcp 10.0.0.240:8443: connect: no route to host"
E0126 12:12:26.086947    4103 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://10.0.0.240:8443/api?timeout=32s\": dial tcp 10.0.0.240:8443: connect: no route to host"
E0126 12:12:29.158978    4103 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://10.0.0.240:8443/api?timeout=32s\": dial tcp 10.0.0.240:8443: connect: no route to host"
Unable to connect to the server: dial tcp 10.0.0.240:8443: connect: no route to host
[root@k8s-cluster251 ~]#

七.高可用组件haproxy+keepalived安装及验证

1.所有master【k8s-cluster24[1-3]】节点安装高可用组件

温馨提示:
    - 对于高可用组件，其实我们也可以单独找两台虚拟机来部署，但我为了节省2台机器，就直接在master节点复用了。
    - 如果在云上安装K8S则无安装高可用组件了，毕竟公有云大部分都是不支持keepalived的，可以直接使用云产品，比如阿里的"SLB"，腾讯的"ELB"等SAAS产品;
    - 推荐使用ELB，SLB有回环的问题，也就是SLB代理的服务器不能反向访问SLB，但是腾讯云修复了这个问题;

具体实操:
	apt-get -y install keepalived haproxy

2.所有master节点配置haproxy

温馨提示:
	- haproxy的负载均衡器监听地址我配置是8443，你可以修改为其他端口，haproxy会用来反向代理各个master组件的地址;
	- 如果你真的修改晴一定注意上面的证书配置的kubeconfig文件，也要一起修改，否则就会出现链接集群失败的问题;
	
具体实操:
	1 备份配置文件
cp /etc/haproxy/haproxy.cfg{,`date +%F`}

	2 所有节点的配置文件内容相同
cat > /etc/haproxy/haproxy.cfg <<'EOF'
global
  maxconn  2000
  ulimit-n  16384
  log  127.0.0.1 local0 err
  stats timeout 30s

defaults
  log global
  mode  http
  option  httplog
  timeout connect 5000
  timeout client  50000
  timeout server  50000
  timeout http-request 15s
  timeout http-keep-alive 15s

frontend monitor-haproxy
  bind *:9999
  mode http
  option httplog
  monitor-uri /ruok

frontend yinzhengjie-k8s
  bind 0.0.0.0:8443
  bind 127.0.0.1:8443
  mode tcp
  option tcplog
  tcp-request inspect-delay 5s
  default_backend yinzhengjie-k8s

backend yinzhengjie-k8s
  mode tcp
  option tcplog
  option tcp-check
  balance roundrobin
  default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
  server k8s-cluster251   10.0.0.251:6443  check
  server k8s-cluster252   10.0.0.252:6443  check
  server k8s-cluster253   10.0.0.253:6443  check
EOF

3.所有master节点配置keepalived

温馨提示:
	- 注意"interface"字段为你的物理网卡的名称，如果你的网卡是ens33，请将"eth0"修改为"ens33"哟;
	- 注意"mcast_src_ip"各master节点的配置均不相同，修改根据实际环境进行修改哟;
	- 注意"virtual_ipaddress"指定的是负载均衡器的VIP地址，这个地址也要和kubeconfig文件的Apiserver地址要一致哟;
	- 注意"script"字段的脚本用于检测后端的apiServer是否健康;
	- 注意"router_id"字段为节点ip，master每个节点配置自己的IP
	
具体实操:
	1."k8s-cluster251"节点创建配置文件
[root@k8s-cluster251 ~]# ifconfig 
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.0.251  netmask 255.255.255.0  broadcast 10.0.0.255
        ether 00:0c:29:32:73:ac  txqueuelen 1000  (Ethernet)
        RX packets 324292  bytes 234183010 (223.3 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 242256  bytes 31242156 (29.7 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

...

[root@k8s-cluster251 ~]# cat > /etc/keepalived/keepalived.conf <<'EOF'
! Configuration File for keepalived
global_defs {
   router_id 10.0.0.251
}
vrrp_script chk_nginx {
    script "/etc/keepalived/check_port.sh 8443"
    interval 2
    weight -20
}
vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 251
    priority 100
    advert_int 1
    mcast_src_ip 10.0.0.251
    nopreempt
    authentication {
        auth_type PASS
        auth_pass yinzhengjie_k8s
    }
    track_script {
         chk_nginx
    }
    virtual_ipaddress {
        10.0.0.240
    }
}
EOF


	2."k8s-cluster252"节点创建配置文件
[root@k8s-cluster252 ~]# ifconfig 
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.0.252  netmask 255.255.255.0  broadcast 10.0.0.255
        ether 00:0c:29:cf:ad:0a  txqueuelen 1000  (Ethernet)
        RX packets 256743  bytes 42628265 (40.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 252589  bytes 34277384 (32.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

...

[root@k8s-cluster252 ~]# cat > /etc/keepalived/keepalived.conf <<EOF
! Configuration File for keepalived
global_defs {
   router_id 10.0.0.252
}
vrrp_script chk_nginx {
    script "/etc/keepalived/check_port.sh 8443"
    interval 2
    weight -20
}
vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 251
    priority 100
    advert_int 1
    mcast_src_ip 10.0.0.252
    nopreempt
    authentication {
        auth_type PASS
        auth_pass yinzhengjie_k8s
    }
    track_script {
         chk_nginx
    }
    virtual_ipaddress {
        10.0.0.240
    }
}
EOF


	3."k8s-cluster253"节点创建配置文件
[root@k8s-cluster253 ~]# ifconfig 
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.0.253  netmask 255.255.255.0  broadcast 10.0.0.255
        ether 00:0c:29:5f:f7:4f  txqueuelen 1000  (Ethernet)
        RX packets 178577  bytes 34808750 (33.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 171025  bytes 26471309 (25.2 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

...

[root@k8s-cluster253 ~]# cat > /etc/keepalived/keepalived.conf <<EOF
! Configuration File for keepalived
global_defs {
   router_id 10.0.0.253
}
vrrp_script chk_nginx {
    script "/etc/keepalived/check_port.sh 8443"
    interval 2
    weight -20
}
vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 251
    priority 100
    advert_int 1
    mcast_src_ip 10.0.0.253
    nopreempt
    authentication {
        auth_type PASS
        auth_pass yinzhengjie_k8s
    }
    track_script {
         chk_nginx
    }
    virtual_ipaddress {
        10.0.0.240
    }
}
EOF


	4.所有keepalived节点均需要创建健康检查脚本
cat > /etc/keepalived/check_port.sh <<'EOF'
#!/bin/bash
CHK_PORT=$1
if [ -n "$CHK_PORT" ];then
    PORT_PROCESS=`ss -lt|grep $CHK_PORT|wc -l`
    if [ $PORT_PROCESS -eq 0 ];then
        echo "Port $CHK_PORT Is Not Used,End."
        systemctl stop keepalived
    fi
else
    echo "Check Port Cant Be Empty!"
fi
EOF
chmod +x /etc/keepalived/check_port.sh

4.验证haproxy服务并验证

	1 所有节点启动haproxy服务
systemctl enable --now haproxy 
systemctl restart haproxy 
systemctl status haproxy 
ss -ntl | grep 8443

	2 所有节点启动keepalived 
systemctl start keepalived

	3 基于telnet验证haporxy是否正常
[root@k8s-cluster253 ~]# telnet 10.0.0.240 8443
Trying 10.0.0.240...
Connected to 10.0.0.240.
Escape character is '^]'.

	4 基于webUI进行验证
[root@k8s-cluster252 ~]# curl http://10.0.0.240:9999/ruok
<html><body><h1>200 OK</h1>
Service ready.
</body></html>
[root@k8s-cluster252 ~]#

5.kubectl测试验证

	1.各个master节点测试
[root@k8s-cluster251 ~]# kubectl get cs --kubeconfig=/yinzhengjie/certs/kubeconfig/kube-admin.kubeconfig
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE   ERROR
controller-manager   Healthy   ok        
scheduler            Healthy   ok        
etcd-0               Healthy   ok        
[root@k8s-cluster251 ~]# 
	
	
[root@k8s-cluster252 ~]# kubectl get cs --kubeconfig=/yinzhengjie/certs/kubeconfig/kube-admin.kubeconfig
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE   ERROR
controller-manager   Healthy   ok        
scheduler            Healthy   ok        
etcd-0               Healthy   ok        
[root@k8s-cluster252 ~]# 


[root@k8s-cluster253 ~]# kubectl get cs --kubeconfig=/yinzhengjie/certs/kubeconfig/kube-admin.kubeconfig
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE   ERROR
scheduler            Healthy   ok        
controller-manager   Healthy   ok        
etcd-0               Healthy   ok        
[root@k8s-cluster253 ~]# 



	2.准备默认的文件
[root@k8s-cluster251 ~]# mkdir -p  ~/.kube
[root@k8s-cluster251 ~]# cp /yinzhengjie/certs/kubeconfig/kube-admin.kubeconfig ~/.kube/config
[root@k8s-cluster251 ~]# kubectl get cs 
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE   ERROR
controller-manager   Healthy   ok        
scheduler            Healthy   ok        
etcd-0               Healthy   ok        
[root@k8s-cluster251 ~]# 


[root@k8s-cluster252 ~]# mkdir -p  ~/.kube
[root@k8s-cluster252 ~]# cp /yinzhengjie/certs/kubeconfig/kube-admin.kubeconfig ~/.kube/config
[root@k8s-cluster252 ~]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE   ERROR
controller-manager   Healthy   ok        
scheduler            Healthy   ok        
etcd-0               Healthy   ok        
[root@k8s-cluster252 ~]# 


[root@k8s-cluster253 ~]# mkdir -p  ~/.kube
[root@k8s-cluster253 ~]# cp /yinzhengjie/certs/kubeconfig/kube-admin.kubeconfig ~/.kube/config
[root@k8s-cluster253 ~]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE   ERROR
controller-manager   Healthy   ok        
scheduler            Healthy   ok        
etcd-0               Healthy   ok        
[root@k8s-cluster253 ~]# 


	3.各节点配置自动补全功能
kubectl completion bash > ~/.kube/completion.bash.inc
echo source '$HOME/.kube/completion.bash.inc' >> ~/.bashrc 
source ~/.bashrc
kubectl  # 连续按2次tab键，观察是否自动补全，如上图所示。

八.启动K8S的worker组件

1.创建Bootstrapping自动颁发kubelet证书配置

K8S 集群扩容kubelet的bootstrap-token阶段图解

温馨提示:
	- "--server"只想的是负载均衡器的IP地址，由负载均衡器对master节点进行反向代理哟。
	- "--token"也可以自定义，但也要同时修改"bootstrap"的Secret的"token-id"和"token-secret"对应值哟;


	1 设置集群
[root@k8s-cluster251 ~]# kubectl config set-cluster yinzhengjie-k8s \
  --certificate-authority=/yinzhengjie/certs/kubernetes/k8s-ca.pem \
  --embed-certs=true \
  --server=https://10.0.0.240:8443 \
  --kubeconfig=/yinzhengjie/certs/kubeconfig/bootstrap-kubelet.kubeconfig

	2 创建用户
[root@k8s-cluster251 ~]# kubectl config set-credentials tls-bootstrap-token-user  \
  --token=yindao.jasonyinzhengjie \
  --kubeconfig=/yinzhengjie/certs/kubeconfig/bootstrap-kubelet.kubeconfig
  
	3 将集群和用户进行绑定
[root@k8s-cluster251 ~]# kubectl config set-context tls-bootstrap-token-user@kubernetes \
  --cluster=yinzhengjie-k8s \
  --user=tls-bootstrap-token-user \
  --kubeconfig=/yinzhengjie/certs/kubeconfig/bootstrap-kubelet.kubeconfig
  
  	4.配置默认的上下文
[root@k8s-cluster251 ~]# kubectl config use-context tls-bootstrap-token-user@kubernetes \
  --kubeconfig=/yinzhengjie/certs/kubeconfig/bootstrap-kubelet.kubeconfig


	5 创建配bootstrap-secret文件用于授权
[root@k8s-cluster251 ~]# cat > bootstrap-secret.yaml <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: bootstrap-token-yindao
  namespace: kube-system
type: bootstrap.kubernetes.io/token
stringData:
  description: "The default bootstrap token generated by 'kubelet '."
  token-id: yindao
  token-secret: jasonyinzhengjie
  usage-bootstrap-authentication: "true"
  usage-bootstrap-signing: "true"
  auth-extra-groups:  system:bootstrappers:default-node-token,system:bootstrappers:worker,system:bootstrappers:ingress

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubelet-bootstrap
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:node-bootstrapper
subjects:

- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:bootstrappers:default-node-token

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: node-autoapprove-bootstrap
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
subjects:

- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:bootstrappers:default-node-token

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: node-autoapprove-certificate-rotation
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
subjects:

- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:nodes

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-apiserver-to-kubelet
rules:
  - apiGroups:
    - ""
    resources:
    - nodes/proxy
    - nodes/stats
    - nodes/log
    - nodes/spec
    - nodes/metrics
    verbs:
    - "*"

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver
  namespace: ""
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:

  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: kube-apiserver
EOF


	6.应用bootstrap-secret配置文件
[root@k8s-cluster251 ~]# kubectl apply -f bootstrap-secret.yaml 
secret/bootstrap-token-yindao created
clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created
clusterrolebinding.rbac.authorization.k8s.io/node-autoapprove-bootstrap created
clusterrolebinding.rbac.authorization.k8s.io/node-autoapprove-certificate-rotation created
clusterrole.rbac.authorization.k8s.io/system:kube-apiserver-to-kubelet created
clusterrolebinding.rbac.authorization.k8s.io/system:kube-apiserver created
[root@k8s-cluster251 ~]#

2.部署worker节点之kubelet启动实战

	1 同步kubelet的kubeconfig认证文件
[root@k8s-cluster251 ~]# data_rsync.sh /yinzhengjie/certs/kubeconfig/bootstrap-kubelet.kubeconfig 
===== rsyncing k8s-cluster252: bootstrap-kubelet.kubeconfig =====
命令执行成功!
===== rsyncing k8s-cluster253: bootstrap-kubelet.kubeconfig =====
命令执行成功!
[root@k8s-cluster251 ~]# 


	2.所有节点创建工作目录
mkdir -p /var/lib/kubelet /var/log/kubernetes /etc/systemd/system/kubelet.service.d /etc/kubernetes/manifests/


	3.所有节点创建kubelet的配置文件
	
温馨提示:
	- 在"10-kubelet.con"文件中使用"--kubeconfig"指定的"kubelet.kubeconfig"文件并不存在，这个证书文件后期会自动生成;
	- 对于"clusterDNS"是NDS地址，我们可以自定义，比如"10.200.0.254";
	- “clusterDomain”对应的是域名信息，要和我们设计的集群保持一致，比如"yinzhengjie.com";
	- "10-kubelet.conf"文件中的"ExecStart="需要写2次，否则可能无法启动kubelet;
	
cat > /etc/kubernetes/kubelet-conf.yml <<'EOF'
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /yinzhengjie/certs/kubernetes/k8s-ca.pem
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
cgroupDriver: systemd
cgroupsPerQOS: true
clusterDNS:
- 10.200.0.254
clusterDomain: yinzhengjie.com
containerLogMaxFiles: 5
containerLogMaxSize: 10Mi
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableControllerAttachDetach: true
enableDebuggingHandlers: true
enforceNodeAllocatable:
- pods
eventBurst: 10
eventRecordQPS: 5
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true
fileCheckFrequency: 20s
hairpinMode: promiscuous-bridge
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 20s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kubeAPIBurst: 10
kubeAPIQPS: 5
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
registryBurst: 10
registryPullQPS: 5
resolvConf: /etc/kubernetes/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 2m0s
serializeImagePulls: true
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
volumeStatsAggPeriod: 1m0s
EOF


	4.所有节点配置kubelet service
cat > /usr/lib/systemd/system/kubelet.service <<'EOF'
[Unit]
Description=JasonYin's Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=containerd.service
Requires=containerd.service

[Service]
ExecStart=/usr/local/bin/kubelet
Restart=always
StartLimitInterval=0
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF


	5 所有节点配置kubelet service的配置文件
cat > /etc/systemd/system/kubelet.service.d/10-kubelet.conf <<'EOF'
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/yinzhengjie/certs/kubeconfig/bootstrap-kubelet.kubeconfig --kubeconfig=/yinzhengjie/certs/kubeconfig/kubelet.kubeconfig"
Environment="KUBELET_CONFIG_ARGS=--config=/etc/kubernetes/kubelet-conf.yml"
Environment="KUBELET_SYSTEM_ARGS=--container-runtime-endpoint=unix:///run/containerd/containerd.sock"
Environment="KUBELET_EXTRA_ARGS=--node-labels=node.kubernetes.io/node='' "
ExecStart=
ExecStart=/usr/local/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_SYSTEM_ARGS $KUBELET_EXTRA_ARGS
EOF


	6.所有节点准备宿主机DNS解析文件
cat > /etc/kubernetes/resolv.conf <<EOF
nameserver 223.5.5.5
options edns0 trust-ad
search .
EOF

	
	7.启动所有节点kubelet
systemctl daemon-reload
systemctl enable  kubelet
systemctl restart kubelet containerd  # 同时重启containerd和kubelet服务。
systemctl status kubelet


	8.在所有master节点上查看nodes信息。
[root@k8s-cluster251 ~]# kubectl get nodes -o wide
NAME             STATUS     ROLES    AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
k8s-cluster251   NotReady   <none>   2m13s   v1.35.0   10.0.0.251    <none>        Ubuntu 22.04.4 LTS   5.15.0-164-generic   containerd://1.7.20
k8s-cluster252   NotReady   <none>   2m8s    v1.35.0   10.0.0.252    <none>        Ubuntu 22.04.4 LTS   5.15.0-164-generic   containerd://1.7.20
k8s-cluster253   NotReady   <none>   2m6s    v1.35.0   10.0.0.253    <none>        Ubuntu 22.04.4 LTS   5.15.0-164-generic   containerd://1.7.20
[root@k8s-cluster251 ~]# 



	9.可以查看到有相应的csr用户客户端的证书请求
[root@k8s-cluster251 ~]# kubectl get csr
NAME        AGE    SIGNERNAME                                    REQUESTOR                 REQUESTEDDURATION   CONDITION
csr-5j4xx   110s   kubernetes.io/kube-apiserver-client-kubelet   system:bootstrap:yindao   <none>              Approved,Issued
csr-9cmsh   110s   kubernetes.io/kube-apiserver-client-kubelet   system:bootstrap:yindao   <none>              Approved,Issued
csr-ght4f   110s   kubernetes.io/kube-apiserver-client-kubelet   system:bootstrap:yindao   <none>              Approved,Issued
csr-v6sbq   111s   kubernetes.io/kube-apiserver-client-kubelet   system:bootstrap:yindao   <none>              Approved,Issued
csr-xcq44   110s   kubernetes.io/kube-apiserver-client-kubelet   system:bootstrap:yindao   <none>              Approved,Issued
[root@k8s-cluster251 ~]#

  10.检查worker节点证书有效期
[root@k8s-cluster251 ~]#  openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem  -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            dc:e7:47:fa:62:ed:8e:ca:d0:14:47:90:4e:81:6c:c7
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = CN, ST = Beijing, L = Beijing, O = Kubernetes, OU = Kubernetes-manual, CN = kubernetes
        Validity
            Not Before: Jan 28 12:05:32 2026 GMT
            Not After : Jan 26 12:05:32 2036 GMT

    ...


[root@k8s-cluster252 ~]# openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem  -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            62:50:30:24:61:f4:5b:5f:63:81:10:21:83:04:56:71
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = CN, ST = Beijing, L = Beijing, O = Kubernetes, OU = Kubernetes-manual, CN = kubernetes
        Validity
            Not Before: Jan 28 12:05:30 2026 GMT
            Not After : Jan 26 12:05:30 2036 GMT
    ....


[root@k8s-cluster253 ~]# openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem  -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            5d:92:f3:23:96:9e:0f:af:c8:70:63:4b:c4:10:2a:09
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = CN, ST = Beijing, L = Beijing, O = Kubernetes, OU = Kubernetes-manual, CN = kubernetes
        Validity
            Not Before: Jan 28 12:05:38 2026 GMT
            Not After : Jan 26 12:05:38 2036 GMT

    ....

3.部署worker节点之kube-proxy服务

	1.所有节点创建kube-proxy.conf配置文件
cat > /etc/kubernetes/kube-proxy.yml << EOF
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
bindAddress: 0.0.0.0
metricsBindAddress: 127.0.0.1:10249
clientConnection:
  acceptConnection: ""
  burst: 10
  contentType: application/vnd.kubernetes.protobuf
  kubeconfig: /yinzhengjie/certs/kubeconfig/kube-proxy.kubeconfig
  qps: 5
clusterCIDR: 10.100.0.0/16
configSyncPeriod: 15m0s
conntrack:
  max: null
  maxPerCore: 32768
  min: 131072
  tcpCloseWaitTimeout: 1h0m0s
  tcpEstablishedTimeout: 24h0m0s
enableProfiling: false
healthzBindAddress: 0.0.0.0:10256
hostnameOverride: ""
iptables:
  masqueradeAll: false
  masqueradeBit: 14
  minSyncPeriod: 0s
ipvs:
  masqueradeAll: true
  minSyncPeriod: 5s
  scheduler: "rr"
  syncPeriod: 30s
mode: "ipvs"
nodeProtAddress: null
oomScoreAdj: -999
portRange: ""
udpIdelTimeout: 250ms
EOF


	2.所有节点使用systemd管理kube-proxy
cat > /usr/lib/systemd/system/kube-proxy.service << EOF
[Unit]
Description=Jason Yin's Kubernetes Proxy
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-proxy \
  --config=/etc/kubernetes/kube-proxy.yml \
  --v=2 
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF


	3.所有节点启动kube-proxy
systemctl daemon-reload && systemctl enable --now kube-proxy
systemctl status kube-proxy
ss -ntl |grep 10249

九.部署calico组件并验证测试

1.calico和k8s版本兼容性说明

QQ_1769675541941

参考链接:
	https://docs.tigera.io/calico/latest/getting-started/kubernetes/requirements

2.部署calico

温馨提示:
	做本实验之前，建议你先拍个快照，因为我们k8s版本是最新版本，calico目前还不支持最新版本。
	经测试，尽管能部署成功，但是会有部分pod会自动重启，稳定性待进一步调优测试。
	当然，我们也可以考虑使用Flannel网络插件来解决此问题。
	

	1.安装Tigera Operator和自定义资源定义
[root@k8s-cluster251 ~]# wget https://raw.githubusercontent.com/projectcalico/calico/v3.31.3/manifests/tigera-operator.yaml

	
[root@k8s-cluster251 ~]# kubectl create -f tigera-operator.yaml 
namespace/tigera-operator created
serviceaccount/tigera-operator created
clusterrole.rbac.authorization.k8s.io/tigera-operator-secrets created
clusterrole.rbac.authorization.k8s.io/tigera-operator created
clusterrolebinding.rbac.authorization.k8s.io/tigera-operator created
rolebinding.rbac.authorization.k8s.io/tigera-operator-secrets created
deployment.apps/tigera-operator created
[root@k8s-cluster251 ~]# 
[root@k8s-cluster251 ~]# kubectl get pods -A
NAMESPACE         NAME                               READY   STATUS    RESTARTS   AGE
tigera-operator   tigera-operator-6447996989-nskk8   1/1     Running   0          1m37s
[root@k8s-cluster251 ~]# 


	2.通过创建必要的自定义资源来安装Calico
[root@k8s-cluster251 ~]# wget https://raw.githubusercontent.com/projectcalico/calico/v3.31.3/manifests/custom-resources.yaml


[root@k8s-cluster251 ~]# grep cidr custom-resources.yaml
        cidr: 192.168.0.0/16
[root@k8s-cluster251 ~]# 
[root@k8s-cluster251 ~]# sed -i '/cidr/s#192.168#10.100#' custom-resources.yaml 
[root@k8s-cluster251 ~]# 
[root@k8s-cluster251 ~]# grep cidr custom-resources.yaml
        cidr: 10.100.0.0/16
[root@k8s-cluster251 ~]# 
[root@k8s-cluster251 ~]# kubectl create -f custom-resources.yaml 
installation.operator.tigera.io/default created
apiserver.operator.tigera.io/default created
goldmane.operator.tigera.io/default created
whisker.operator.tigera.io/default created
[root@k8s-cluster251 ~]# 
[root@k8s-cluster251 ~]# kubectl get pod -A
NAMESPACE         NAME                                       READY   STATUS    RESTARTS   AGE
calico-system     calico-apiserver-84cf6f57f5-hmk9s          1/1     Running   0          49s
calico-system     calico-apiserver-84cf6f57f5-wr4j8          1/1     Running   0          49s
calico-system     calico-kube-controllers-56647c4dd5-vg8l7   1/1     Running   0          48s
calico-system     calico-node-4tvrw                          1/1     Running   0          49s
calico-system     calico-node-7zwv8                          1/1     Running   0          49s
calico-system     calico-node-g6wjx                          1/1     Running   0          49s
calico-system     calico-typha-65cfc6d749-6qxqp              1/1     Running   0          48s
calico-system     calico-typha-65cfc6d749-8fbpl              1/1     Running   0          49s
calico-system     csi-node-driver-9kqn7                      2/2     Running   0          49s
calico-system     csi-node-driver-nkbmn                      2/2     Running   0          49s
calico-system     csi-node-driver-r7sm6                      2/2     Running   0          49s
calico-system     goldmane-5b885c5555-rn9h7                  1/1     Running   0          49s
calico-system     whisker-696875685d-m62dt                   2/2     Running   0          44s
tigera-operator   tigera-operator-6447996989-nskk8           1/1     Running   0          2m
[root@k8s-cluster251 ~]#

3.测试calico的可用性

温馨提示:
	如果往网络不通，可以尝试删除calico-node相关的pod测试观察。
		kubectl -n calico-system delete pods -l  app.kubernetes.io/name=calico-node


实战案例:
	1.编写资源清单
[root@k8s-cluster251 ~]# cat > deploy-apps.yaml  <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: yinzhengjie-app01
spec:
  replicas: 1
  selector:
    matchLabels:
      apps: v1 
  template:
    metadata:
      labels:
        apps: v1
    spec:
      nodeName: k8s-cluster252
      containers:
      - name: c1
        image: registry.cn-hangzhou.aliyuncs.com/yinzhengjie-k8s/apps:v1 

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: yinzhengjie-app02
spec:
  replicas: 1
  selector:
    matchLabels:
      apps: v1 
  template:
    metadata:
      labels:
        apps: v1
    spec:
      nodeName: k8s-cluster253
      containers:
      - name: c1
        image: registry.cn-hangzhou.aliyuncs.com/yinzhengjie-k8s/apps:v2
EOF


	2.创建资源
[root@k8s-cluster251 ~]# kubectl  apply -f deploy-apps.yaml 
deployment.apps/yinzhengjie-app01 created
deployment.apps/yinzhengjie-app02 created
[root@k8s-cluster251 ~]# 


	3.测试验证
[root@k8s-cluster251 ~]# kubectl get pods -o wide
NAME                                 READY   STATUS    RESTARTS   AGE    IP              NODE             NOMINATED NODE   READINESS GATES
yinzhengjie-app01-859798856d-stxxb   1/1     Running   0          2m6s   10.100.126.84   k8s-cluster252   <none>           <none>
yinzhengjie-app02-6c68b6f784-vjw22   1/1     Running   0          2m6s   10.100.22.8     k8s-cluster253   <none>           <none>
[root@k8s-cluster251 ~]# 
[root@k8s-cluster251 ~]# curl 10.100.126.84
<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8"/>
    <title>yinzhengjie apps v1</title>
    <style>
       div img {
          width: 900px;
          height: 600px;
          margin: 0;
       }
    </style>
  </head>

  <body>
    <h1 style="color: green">凡人修仙传 v1 </h1>
    <div>
      <img src="1.jpg">
    <div>
  </body>

</html>
[root@k8s-cluster251 ~]# 
[root@k8s-cluster251 ~]# curl 10.100.22.8
<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8"/>
    <title>yinzhengjie apps v2</title>
    <style>
       div img {
          width: 900px;
          height: 600px;
          margin: 0;
       }
    </style>
  </head>

  <body>
    <h1 style="color: red">凡人修仙传 v2 </h1>
    <div>
      <img src="2.jpg">
    <div>
  </body>

</html>
[root@k8s-cluster251 ~]#

十.部署k8s集群附加组件

推荐阅读:
	https://www.cnblogs.com/yinzhengjie/p/19530681

此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。原文来自 — 版权归原作者所有。

推荐订阅源

博客园 - 尹正杰

一.k8s高可用架构设计及基础环境准备

1.k8s高可用架构方案

2.环境准备

3.k8s版本说明

4.所有节点安装常用的软件包

5.k8s-master241节点免密钥登录集群并同步数据

6.所有节点Linux基础环境优化

7.所有节点安装ipvsadm以实现kube-proxy的负载均衡

二.安装containerd组件及基础使用

1.Containerd环境部署

2.Containerd极速入门

三.安装K8S程序

1.下载软件包

2.解压K8S的二进制程序包到PATH环境变量路径

四.etcd高可用集群部署

五.生成k8s组件相关证书

1.所有节点创建k8s证书存储目录

2.k8s-cluster251节点生成kubernetes自建ca证书

3.k8s-cluster251节点基于自建ca证书颁发apiserver相关证书

4.生成第三方组件与apiServer通信的聚合证书

5.生成controller-manager证书及kubeconfig文件

6.生成scheduler证书及kubeconfig文件

7.生成kube-proxy证书及kubeconfig文件

8.配置k8s集群管理员证书及kubeconfig文件

9.创建ServiceAccount

10.k8s-master01节点K8S组件证书拷贝到其他两个master节点

六.启动K8S的master组件

1.部署ApiServer组件

2.部署ControlerManager组件

3.部署Scheduler组件

4.测试组件是否正常

七.高可用组件haproxy+keepalived安装及验证

1.所有master【k8s-cluster24[1-3]】节点安装高可用组件

2.所有master节点配置haproxy

3.所有master节点配置keepalived

4.验证haproxy服务并验证

5.kubectl测试验证

八.启动K8S的worker组件

1.创建Bootstrapping自动颁发kubelet证书配置

2.部署worker节点之kubelet启动实战

3.部署worker节点之kube-proxy服务

九.部署calico组件并验证测试

1.calico和k8s版本兼容性说明

2.部署calico

3.测试calico的可用性

十.部署k8s集群附加组件