k8s底层基于不同运行时集成harbor企业级私有仓库实战

博客园 - 尹正杰

Categraf使用指南 window系统部署Claude Code实战 N9E集群部署 Ubuntu 24.04 LTS部署Codex实战 aiops运维平台从0到1部署实战 aiops平台环境部署 TKE集群LoadBalancer映射案例 k8s集群为每个节点分配不通的ippool实战 ansible剧本实现一键升级K8S集群的NVIDIA GPU驱动 vmagent采集etcd数据并写入victoriametrics集群 VictoriaMetrics集群部署实战 Squid正向代理实战 kubespray生产故障案例之kubelet启动参数案例 filebeat指定名称空间采集pod数据 Calico自定义Ipool实战 ansible剧本实现kibana的部署和卸载 ansible剧本应用案例集合2 ansible剧本实现磁盘格式化 Ansible急速入门实战篇 ansible快速入门篇 k8s集群管理 kubesray实战 kubespray实战案例 kubespray管理k8s的worker集群扩缩容 kubespray快速部署k8s集群实战 Kubeasz使用吐槽博客专题 Kubeasz基于ezctl实现etcd集群的管理实战 Kubeasz基于ezctl实现k8s集群一键升级 Calico启用纯BGP模式+RR实战案例 Calico 底层原理及IPIP（依赖BGP协议）)和vxlan（不依赖BGP）工作模式切换

尹正杰 · 2026-01-30 · via 博客园 - 尹正杰

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正杰

一.harbor自建证书启用https实战案例
二.k8s集成harbor企业级仓库
- 1.K8S节点配置docker运行时证书实战
- 2.K8S节点配置containerd运行时证书实战

一.harbor自建证书启用https实战案例

1.环境准备

服务器名称	IP地址	硬件配置	操作系统
harbor250.yinzhengjie.com	10.0.0.250	1C+，2G+，100GB+	Ubuntu 22.04 LTS

具体虚拟机硬件配置最低要求如上表所示。

2.安装docker和docker-compose

[root@harbor250.yinzhengjie.com ~]# wget http://192.168.21.253/Resources/Docker/scripts/yinzhengjie-autoinstall-docker-docker-compose.tar.gz
[root@harbor250.yinzhengjie.com ~]# 
[root@harbor250.yinzhengjie.com ~]# tar xf yinzhengjie-autoinstall-docker-docker-compose.tar.gz 
[root@harbor250.yinzhengjie.com ~]# 
[root@harbor250.yinzhengjie.com ~]# ./install-docker.sh i

3.安装并解压harbor安装包

	1 下载harbor软件包
[root@harbor250.yinzhengjie.com ~]# wget https://github.com/goharbor/harbor/releases/download/v2.14.2/harbor-offline-installer-v2.14.2.tgz

		
	2 解压harbor安装包
[root@harbor250.yinzhengjie.com ~]# tar xf harbor-offline-installer-v2.14.2.tgz -C /usr/local/

4.配置CA证书

	1 进入到harbor程序的根目录
[root@harbor250.yinzhengjie.com ~]# cd /usr/local/harbor/
[root@harbor250.yinzhengjie.com harbor]# 
[root@harbor250.yinzhengjie.com harbor]# ll
total 696580
drwxr-xr-x  2 root root      4096 Feb 12 10:35 ./
drwxr-xr-x 11 root root      4096 Feb 12 10:35 ../
-rw-r--r--  1 root root      3646 Jan 12 17:13 common.sh
-rw-r--r--  1 root root 713241468 Jan 12 17:14 harbor.v2.14.2.tar.gz
-rw-r--r--  1 root root     14688 Jan 12 17:13 harbor.yml.tmpl
-rwxr-xr-x  1 root root      1975 Jan 12 17:13 install.sh*
-rw-r--r--  1 root root     11347 Jan 12 17:13 LICENSE
-rwxr-xr-x  1 root root      2211 Jan 12 17:13 prepare*
[root@harbor250.yinzhengjie.com harbor]# 


	2 创建证书存放目录
[root@harbor250.yinzhengjie.com harbor]# apt -y install tree
[root@harbor250.yinzhengjie.com harbor]# 
[root@harbor250.yinzhengjie.com harbor]# mkdir -pv certs/{ca,harbor-server,docker-client}
mkdir: created directory 'certs'
mkdir: created directory 'certs/ca'
mkdir: created directory 'certs/harbor-server'
mkdir: created directory 'certs/docker-client'
[root@harbor250.yinzhengjie.com harbor]#
[root@harbor250.yinzhengjie.com harbor]# tree certs/
certs/
├── ca
├── docker-client
└── harbor-server

3 directories, 0 files
[root@harbor250.yinzhengjie.com harbor]# 
[root@harbor250.yinzhengjie.com harbor]# cd certs/
[root@harbor250.yinzhengjie.com certs]# 



	3 创建CA的私钥
[root@harbor250.yinzhengjie.com harbor]# cd certs/
[root@harbor250.yinzhengjie.com certs]# 
[root@harbor250.yinzhengjie.com certs]# openssl genrsa -out ca/ca.key 4096
[root@harbor250.yinzhengjie.com certs]# 
[root@harbor250.yinzhengjie.com certs]# tree 
.
├── ca
│   └── ca.key
├── docker-client
└── harbor-server

3 directories, 1 file
[root@harbor250.yinzhengjie.com certs]# 



	4 基于自建的CA私钥创建CA证书(注意，证书签发的域名范围)
[root@harbor250.yinzhengjie.com certs]# openssl req -x509 -new -nodes \
 -sha512 -days 3650 \
 -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=yinzhengjie.com" \
 -key ca/ca.key \
 -out ca/ca.crt
 

[root@harbor250.yinzhengjie.com certs]# tree 
.
├── ca
│   ├── ca.crt
│   └── ca.key
├── docker-client
└── harbor-server

3 directories, 2 files
[root@harbor250.yinzhengjie.com certs]# 


	5 查看自建证书信息
[root@harbor250.yinzhengjie.com certs]# openssl  x509 -in ca/ca.crt -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            02:ea:3b:33:6a:55:85:d9:0e:76:7f:cd:6c:67:1e:57:bf:0e:7f:f4
        Signature Algorithm: sha512WithRSAEncryption
        Issuer: C = CN, ST = Beijing, L = Beijing, O = example, OU = Personal, CN = yinzhengjie.com
        Validity
            Not Before: May 22 01:07:50 2025 GMT
            Not After : May 20 01:07:50 2035 GMT
        Subject: C = CN, ST = Beijing, L = Beijing, O = example, OU = Personal, CN = yinzhengjie.com
...

5.配置harbor服务端证书

	1 生成harbor服务器的私钥
[root@harbor250.yinzhengjie.com certs]# openssl genrsa -out harbor-server/harbor250.yinzhengjie.com.key 4096
[root@harbor250.yinzhengjie.com certs]# 
[root@harbor250.yinzhengjie.com certs]# tree 
.
├── ca
│   ├── ca.crt
│   └── ca.key
├── docker-client
└── harbor-server
    └── harbor250.yinzhengjie.com.key

3 directories, 3 files
[root@harbor250.yinzhengjie.com certs]# 


	2 harbor服务器基于私钥签发证书认证请求（csr文件），让自建CA认证
[root@harbor250.yinzhengjie.com certs]#  openssl req -sha512 -new \
    -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=harbor250.yinzhengjie.com" \
    -key harbor-server/harbor250.yinzhengjie.com.key \
    -out harbor-server/harbor250.yinzhengjie.com.csr


[root@harbor250.yinzhengjie.com certs]# tree 
.
├── ca
│   ├── ca.crt
│   └── ca.key
├── docker-client
└── harbor-server
    ├── harbor250.yinzhengjie.com.csr
    └── harbor250.yinzhengjie.com.key

3 directories, 4 files
[root@harbor250.yinzhengjie.com certs]# 


	3 生成 x509 v3 的扩展文件用于认证
[root@harbor250.yinzhengjie.com certs]#  cat > harbor-server/v3.ext <<-EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[alt_names]
DNS.1=harbor250.yinzhengjie.com
EOF


[root@harbor250.yinzhengjie.com certs]# tree 
.
├── ca
│   ├── ca.crt
│   └── ca.key
├── docker-client
└── harbor-server
    ├── harbor250.yinzhengjie.com.csr
    ├── harbor250.yinzhengjie.com.key
    └── v3.ext

3 directories, 5 files
[root@harbor250.yinzhengjie.com certs]# 


	4 基于 x509 v3 的扩展文件认证签发harbor server证书
[root@harbor250.yinzhengjie.com certs]#  openssl x509 -req -sha512 -days 3650 \
    -extfile harbor-server/v3.ext \
    -CA ca/ca.crt -CAkey ca/ca.key -CAcreateserial \
    -in harbor-server/harbor250.yinzhengjie.com.csr \
    -out harbor-server/harbor250.yinzhengjie.com.crt


[root@harbor250.yinzhengjie.com certs]# tree 
.
├── ca
│   ├── ca.crt
│   └── ca.key
├── docker-client
└── harbor-server
    ├── harbor250.yinzhengjie.com.crt
    ├── harbor250.yinzhengjie.com.csr
    ├── harbor250.yinzhengjie.com.key
    └── v3.ext

3 directories, 6 files
[root@harbor250.yinzhengjie.com certs]# 



	5 修改harbor的配置文件使用自建证书
[root@harbor250.yinzhengjie.com certs]# cp ../harbor.yml{.tmpl,}
[root@harbor250.yinzhengjie.com certs]# 
[root@harbor250.yinzhengjie.com certs]# vim ../harbor.yml
...
hostname: harbor250.yinzhengjie.com
https:
  ...
  certificate: /usr/local/harbor/certs/harbor-server/harbor250.yinzhengjie.com.crt
  private_key: /usr/local/harbor/certs/harbor-server/harbor250.yinzhengjie.com.key
...
harbor_admin_password: 1
...
data_volume: /var/lib/harbor
...

6 安装harbor服务

[root@harbor250.yinzhengjie.com certs]# ../install.sh
 
[Step 0]: checking if docker is installed ...

Note: docker version: 20.10.24

[Step 1]: checking docker-compose is installed ...

Note: docker-compose version: 2.23.0

[Step 2]: loading Harbor images ...
...

[Step 5]: starting Harbor ...
[+] Building 0.0s (0/0)                                                                                                 docker:default
[+] Running 10/10
 ✔ Network harbor_harbor        Created                                                                                           0.0s 
 ✔ Container harbor-log         Started                                                                                           0.0s 
 ✔ Container harbor-db          Started                                                                                           0.0s 
 ✔ Container registryctl        Started                                                                                           0.1s 
 ✔ Container harbor-portal      Started                                                                                           0.1s 
 ✔ Container redis              Started                                                                                           0.1s 
 ✔ Container registry           Started                                                                                           0.1s 
 ✔ Container harbor-core        Started                                                                                           0.0s 
 ✔ Container nginx              Started                                                                                           0.0s 
 ✔ Container harbor-jobservice  Started                                                                                           0.0s 
✔ ----Harbor has been installed and started successfully.----
[root@harbor250 certs]# 
[root@harbor250 certs]# ss -ntl | grep 80
LISTEN 0      4096         0.0.0.0:80        0.0.0.0:*          
LISTEN 0      4096            [::]:80           [::]:*          
[root@harbor250 certs]#

7. 访问harbor的WebUI

QQ_1770864873935

	1 在windows添加hosts文件解析如下:
10.0.0.250 harbor250.yinzhengjie.com

		6.2 访问测试:【如上图所示】
https://harbor250.yinzhengjie.com/harbor/projects/1/repositories

二.k8s集成harbor企业级仓库

1.K8S节点配置docker运行时证书实战

QQ_1770865312484

	1.生成harbor客户端证书
[root@harbor250.yinzhengjie.com certs]# cp ca/ca.crt harbor-server/harbor250.yinzhengjie.com.key docker-client/
[root@harbor250.yinzhengjie.com certs]# 
[root@harbor250.yinzhengjie.com certs]# cp harbor-server/harbor250.yinzhengjie.com.crt docker-client/harbor250.yinzhengjie.com.cert
[root@harbor250.yinzhengjie.com certs]# 
[root@harbor250.yinzhengjie.com certs]# 
[root@harbor250.yinzhengjie.com certs]# tree 
.
├── ca
│   ├── ca.crt
│   └── ca.key
├── docker-client
│   ├── ca.crt
│   ├── harbor250.yinzhengjie.com.cert
│   └── harbor250.yinzhengjie.com.key
└── harbor-server
    ├── harbor250.yinzhengjie.com.crt
    ├── harbor250.yinzhengjie.com.csr
    ├── harbor250.yinzhengjie.com.key
    └── v3.ext

3 directories, 9 files
[root@harbor250.yinzhengjie.com certs]# 



	2.k8s所有节点添加hosts文件解析
echo 10.0.0.250 harbor250.yinzhengjie.com >> /etc/hosts
 

	3.K8S所有节点拷贝证书文件 【如果你的证书是官方认证的，可以跳过此步骤，我的环境是自建证书。】
mkdir -pv /etc/docker/certs.d/harbor250.yinzhengjie.com/
scp 10.0.0.250:/usr/local/harbor/certs/docker-client/* /etc/docker/certs.d/harbor250.yinzhengjie.com/
	
	
	4.测试登录
[root@master231 ~]# docker login -u admin -p 1 harbor250.yinzhengjie.com
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
[root@master231 ~]# 

	
	
[root@worker232 ~]# docker login -u admin -p 1 harbor250.yinzhengjie.com
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
[root@worker232 ~]# 

	
[root@worker233 ~]# docker login -u admin -p 1 harbor250.yinzhengjie.com
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
[root@worker233 ~]# 


	5.导入测试镜像【需要再harbor仓库创建一个名为"yinzhengjie-db"的"公开"项目】
[root@worker233 ~]# docker image ls mysql
REPOSITORY   TAG             IMAGE ID       CREATED         SIZE
mysql        8.0.36-oracle   f5f171121fa3   21 months ago   603MB
[root@worker233 ~]# 
[root@worker233 ~]# docker tag mysql:8.0.36-oracle harbor250.yinzhengjie.com/yinzhengjie-db/mysql:8.0.36-oracle

[root@worker233 ~]# docker push harbor250.yinzhengjie.com/yinzhengjie-db/mysql:8.0.36-oracle
The push refers to repository [harbor250.yinzhengjie.com/yinzhengjie-db/mysql]
318dde184d61: Pushed 
1c0ff7ed67c4: Pushed 
876b8cd855eb: Pushed 
84d659420bad: Pushed 
9513d2aedd12: Pushed 
eaa1e85de732: Pushed 
a6909c467615: Pushed 
5b76076a2dd4: Pushed 
fb5c92e924ab: Pushed 
152c1ecea280: Pushed 
fc037c17567d: Pushed 
8.0.36-oracle: digest: sha256:c57363379dee26561c2e554f82e70704be4c8129bd0d10e29252cc0a34774004 size: 2618
[root@worker233 ~]# 


	6.服务端验证查看
如上图所示。

	7.拉取镜像测试
[root@worker232 ~]# docker pull harbor250.yinzhengjie.com/yinzhengjie-db/mysql:8.0.36-oracle
8.0.36-oracle: Pulling from yinzhengjie-db/mysql
bd37f6d99203: Pull complete 
d2433cba0951: Pull complete 
13702d9fe3c3: Pull complete 
83bcc87284a1: Pull complete 
c38d8660e1fa: Pull complete 
7e1bc321f421: Pull complete 
bddd54b9c549: Pull complete 
4eaae1e844ac: Pull complete 
5196e1e87d8f: Pull complete 
6586d096303c: Pull complete 
cf55ff1c80af: Pull complete 
Digest: sha256:c57363379dee26561c2e554f82e70704be4c8129bd0d10e29252cc0a34774004
Status: Downloaded newer image for harbor250.yinzhengjie.com/yinzhengjie-db/mysql:8.0.36-oracle
harbor250.yinzhengjie.com/yinzhengjie-db/mysql:8.0.36-oracle
[root@worker232 ~]# 
[root@worker232 ~]# docker image ls harbor250.yinzhengjie.com/yinzhengjie-db/mysql
REPOSITORY                                   TAG             IMAGE ID       CREATED         SIZE
harbor250.yinzhengjie.com/yinzhengjie-db/mysql   8.0.36-oracle   f5f171121fa3   20 months ago   603MB
[root@worker232 ~]#

2.K8S节点配置containerd运行时证书实战

QQ_1770867373130


  1.k8s默认无法识别自建证书
如上图所示，k8s的worker节点无法识别harbor自建的认证证书。需要Linux主机信任证书。


  2.将harbor服务器的自建ca证书拷贝到k8s所有worker节点解决证书信任问题
scp harbor250.yinzhengjie.com:/usr/local/harbor/certs/docker-client/ca.crt /etc/ssl/certs/
systemctl restart containerd.service 


温馨提示:
  上面的方式我在线下教学时有的同学机器并不好使，于是给出如下的方案，最终得以解决。
mkdir -pv /etc/containerd/certs.d/harbor250.yinzhengjie.com
scp harbor250.yinzhengjie.com:/usr/local/harbor/certs/docker-client/* /etc/containerd/certs.d/harbor250.yinzhengjie.com/
cp /etc/containerd/certs.d/harbor250.yinzhengjie.com/ca.crt /usr/local/share/ca-certificates/
update-ca-certificates 
systemctl restart containerd.service 


  3.测试验证
[root@k8s-cluster251 manifests]# cat 01-deploy-mysql.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: deploy-mysql-server
spec:
  replicas: 1
  selector:
    matchLabels:
      apps: mysql
  template:
    metadata:
      labels:
        apps: mysql
    spec:
      containers:
      - image: harbor250.yinzhengjie.com/yinzhengjie-db/mysql:8.0.36-oracle
        name: mysql
        ports:
        - containerPort: 80
          name: db
        env:
        - name: MYSQL_ALLOW_EMPTY_PASSWORD
          value: "yes"
        resources:
          requests:
            cpu: 0.2
            memory: 200Mi
          limits:
            cpu: 500m
            memory: 500Mi
[root@k8s-cluster251 manifests]# 
[root@k8s-cluster251 manifests]# kubectl apply -f  01-deploy-mysql.yaml 
deployment.apps/deploy-mysql-server created
[root@k8s-cluster251 manifests]# 
[root@k8s-cluster251 manifests]# kubectl get pods -o wide
NAME                                   READY   STATUS    RESTARTS   AGE   IP              NODE             NOMINATED NODE   READINESS GATES
deploy-mysql-server-67d44f7889-zzb5j   1/1     Running   0          4s    10.100.105.25   k8s-cluster253   <none>           <none>
[root@k8s-cluster251 manifests]# 
[root@k8s-cluster251 manifests]# kubectl get nodes -o wide
NAME             STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
k8s-cluster251   Ready    <none>   14d   v1.35.0   10.0.0.251    <none>        Ubuntu 22.04.4 LTS   5.15.0-164-generic   containerd://2.2.1
k8s-cluster252   Ready    <none>   14d   v1.35.0   10.0.0.252    <none>        Ubuntu 22.04.4 LTS   5.15.0-164-generic   containerd://2.2.1
k8s-cluster253   Ready    <none>   14d   v1.35.0   10.0.0.253    <none>        Ubuntu 22.04.4 LTS   5.15.0-164-generic   containerd://2.2.1
[root@k8s-cluster251 manifests]#

此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。原文来自 — 版权归原作者所有。

推荐订阅源

博客园 - 尹正杰

一.harbor自建证书启用https实战案例

1.环境准备

2.安装docker和docker-compose

3.安装并解压harbor安装包

4.配置CA证书

5.配置harbor服务端证书

6 安装harbor服务

7. 访问harbor的WebUI

二.k8s集成harbor企业级仓库

1.K8S节点配置docker运行时证书实战

2.K8S节点配置containerd运行时证书实战