背景
最近工作在内网环境，需要从头搭建一个K8s集群并进行微服务迁移。
前期调研的K8S离线部署的方案，可以参考其他博主的博文——万字长文详解 PaaS toB 场景下 K8s 离线部署方案
| Item |
Language |
Star |
Fork |
离线部署支持情况 |
| kops |
Golang |
13.2k |
4.1k |
不支持 |
| kubespray |
Ansible |
11.1k |
4.7k |
支持，需自行构建安装包 |
| kubeasz |
Ansible |
7.2k |
2.7k |
支持，需自行构建安装包 |
| sealos |
Golang |
4.1k |
790 |
支持，需付费充值会员 |
| RKE |
Golang |
2.5k |
480 |
不支持，需自行安装 docker |
| sealer |
Golang |
503 |
112 |
支持，源自 sealos |
| kubekey |
Golang |
471 |
155 |
部分支持，仅镜像可离线 |
以上的方案我尝试了sealos、kubekey，但是由于最初给到我的机器本身网络环境不干净，导致cni一直无法正常拉起，而这些部署工具在简化部署的同时，也屏蔽了大量细节，导致定位问题比较麻烦，后续还是使用kubeadm自行构建。
部署物料准备
服务器：
docker：
1
2
3
| wget -P /home/deploy/deb/docker/ https://download.docker.com/linux/ubuntu/dists/bionic/pool/stable/amd64/docker-ce_19.03.13~3-0~ubuntu-bionic_amd64.deb
wget -P /home/deploy/deb/docker/ https://download.docker.com/linux/ubuntu/dists/bionic/pool/stable/amd64/containerd.io_1.3.7-1_amd64.deb
wget -P /home/deploy/deb/docker/ https://download.docker.com/linux/ubuntu/dists/bionic/pool/stable/amd64/docker-ce-cli_19.03.13~3-0~ubuntu-bionic_amd64.deb
|
K8S：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
| apt-get update && apt-get install -y apt-transport-httpscurl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add - cat > /etc/apt/sources.list.d/kubernetes.list << ERIC
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
ERIC apt-get update apt-cache madison kubeadm VERSION=1.19.16-00
###### 将包下载到本地
apt-get install -y --download-only -o dir::cache::archives=/home/deploy/deb/k8s kubelet=$VERSION kubeadm=$VERSION kubectl=$VERSION
|
镜像准备
国内镜像准备
1
| registry.cn-hangzhou.aliyuncs.com/google_containers
|
在联网环境下准备以下镜像：
1
2
3
4
5
6
7
8
| ➜ ~ kubeadm config images --kubernetes-version=v1.19.16 --image-repository registry.cn-hangzhou.aliyuncs.com/google_containers
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.19.16
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.19.16
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.19.16
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.19.16
registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2
registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.13-0
registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.7.0
|
拉取
1
2
3
4
5
6
7
| docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.19.16
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.19.16
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.19.16
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.19.16
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.13-0
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.7.0
|
保存
1
2
3
4
5
6
7
| docker save -o k8s.tar registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.19.16
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.19.16
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.19.16
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.19.16
registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2
registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.13-0
registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.7.0
|
Flannel
下载：https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
实施
关闭swap分区
关闭防火墙
1
| systemctl disable ufw && systemctl stop ufw
|
安装docker
1
| dpkg -i /home/deploy/deb/docker/*.deb
|
安装后默认cgroups驱动使用cgroupfs ，需要调整为systemd，因此，编辑docker配置文件，执行：
1
| sudo vi /etc/docker/daemon.json
|
添加如下内容：
1
2
3
| {
"exec-opts": ["native.cgroupdriver=systemd"]
}
|
重启docker，执行：
1
| sudo systemctl daemon-reload && sudo systemctl restart docker
|
安装kubeadm、kubelet 和 kubectl
1
2
| ###### 离线安装 k8s
dpkg -i /home/deploy/deb/k8s/*.deb
|
导入镜像
启动
1
2
| sudo kubeadm init --pod-network-cidr 10.244.0.0/16 \
--image-repository registry.cn-hangzhou.aliyuncs.com/google_containers
|
执行init成功后，记录下以“kubeadm join”开头的最后两行：
1
2
3
| kubeadm join 192.168.20.104:6443 --token 0mj488.h6v5r010bfhlq9b1 \
--discovery-token-ca-cert-hash sha256:3ea2cc19ceb0f109834f82bde13f5d29c534aba115cd41f8d3719db6b8ec074b
root@master01:/home/deploy/deb/yaml
|
最后依次执行：
1
2
3
4
5
| mkdir -p $HOME/.kubesudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config
|
启用Flannel网络
执行
1
| kubectl apply -f ./kube-flannel.yml
|
执行成功后，稍等3、5分钟，再次执行kubectl get nodes 和 kubectl get pods –all-namespaces，会看到状态正常了
1
2
3
4
5
6
7
8
9
10
| yance@yance-ub:~$ kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-6c76c8bb89-vjghr 1/1 Running 0 46m
coredns-6c76c8bb89-zswv9 1/1 Running 0 46m
etcd-yance-ub 1/1 Running 0 46m
kube-apiserver-yance-ub 1/1 Running 0 46m
kube-controller-manager-yance-ub 1/1 Running 0 46m
kube-flannel-ds-dlxgv 1/1 Running 0 23m
kube-proxy-nhdwj 1/1 Running 0 46m
kube-scheduler-yance-ub 1/1 Running 0 46m
|
加入worker节点
在每个worker节点上，执行
1
2
3
| kubeadm join 192.168.20.104:6443 --token 0mj488.h6v5r010bfhlq9b1 \
--discovery-token-ca-cert-hash sha256:3ea2cc19ceb0f109834f82bde13f5d29c534aba115cd41f8d3719db6b8ec074b
root@master01:/home/deploy/deb/yaml
|
在master上执行：
1
| kubectl label node node_name node-role.kubernetes.io/worker=worker
|
安装 Kuboard v3.x
1
2
3
4
5
6
7
8
9
| sudo docker run -d \
--restart=unless-stopped \
--name=kuboard \
-p 80:80/tcp \
-p 10081:10081/tcp \
-e KUBOARD_ENDPOINT="http://内网IP:80" \
-e KUBOARD_AGENT_SERVER_TCP_PORT="10081" \
-v /root/kuboard-data:/data \
eipwork/kuboard:v3
|
在浏览器输入 http://your-host-ip:80 即可访问 Kuboard v3.x 的界面，登录方式：
- 用户名：
admin
- 密 码：
Kuboard123
