作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
推荐阅读：
https://github.com/etcd-io/etcd
https://etcd.io/docs/
etcd是分布式系统中最关键数据的分布式可靠键值存储，重点是：
简单：定义明确、面向用户的API（gRPC）
安全：自动TLS，可选客户端证书身份验证
快速：以每秒10000次写入为基准
可靠：使用Raft正确分布
etcd是用Go编写的，并使用Raft共识算法来管理高可用的复制日志。
etcd被许多公司用于生产环境，开发团队在关键部署场景中支持它，其中etcd经常与Kubernetes、locksmith、vulcan、Doorman等应用程序合作。严格的稳健性测试进一步确保了可靠性。
etcd使用http协议工作，支持tls，生产环境中一般使用的是https访问。
etcdctl是一个用于与etcd服务器交互的命令行工具，用户大多通过输入或获取键的值与etcd进行交互。
etcdctl用来与etcd对话的API版本可以通过etcdctl_API环境变量设置为版本2或3。
默认情况下，"etcdctl 3.4+"的etcdctl使用v3 API，早期版本"etcdctl 3.3-"默认为v2 API。
值得注意的是，etcd v2版本写入的数据无法通过v3的版本进行查询，说白了，就是高版本不兼容低版本，相当于v3重写了etcd。
生产环境中，官方推荐使用v3版本。
etcd的双端口作用:
- 2379: HTTP|HTTPS
对客户端使用。
- 2380: TCP
etcd集群内部使用。
安装etcd的方式
包管理方式安装： 安装简单，但是版本低
apt -y install etcd-client
二进制方式安装：可以安装任意版本（参考etcd相关的安装笔记即可）
https://www.cnblogs.com/yinzhengjie/p/18265994
| 主机名 | 操作系统 | IP地址 | 硬件配置 |
|---|---|---|---|
| k8s-cluster251 | Ubuntu 22.04 LTS | 10.0.0.251 | Core: 2c Memroy: 4G Disk:50+ |
| k8s-cluster252 | Ubuntu 22.04 LTS | 10.0.0.252 | Core: 2c Memroy: 4G Disk:50+ |
| k8s-cluster253 | Ubuntu 22.04 LTS | 10.0.0.253 | Core: 2c Memroy: 4G Disk:50+ |
如上所示，需要准备3台服务器。
1.下载etcd的软件包
wget https://github.com/etcd-io/etcd/releases/download/v3.6.7/etcd-v3.6.7-linux-amd64.tar.gz
svip:
[root@k8s-cluster251 ~]# wget http://192.168.21.253/Resources/Prometheus/softwares/Etcd/etcd-v3.6.7-linux-amd64.tar.gz
2.解压etcd的二进制程序包到PATH环境变量路径
[root@k8s-cluster251 ~]# tar xf etcd-v3.6.7-linux-amd64.tar.gz -C /usr/local/bin etcd-v3.6.7-linux-amd64/etcd{,ctl,utl} --strip-components=1
[root@k8s-cluster251 ~]#
[root@k8s-cluster251 ~]# ll /usr/local/bin/etcd*
-rwxr-xr-x 1 yinzhengjie yinzhengjie 26087608 Dec 18 03:41 /usr/local/bin/etcd*
-rwxr-xr-x 1 yinzhengjie yinzhengjie 17207480 Dec 18 03:41 /usr/local/bin/etcdctl*
-rwxr-xr-x 1 yinzhengjie yinzhengjie 17244344 Dec 18 03:41 /usr/local/bin/etcdutl*
[root@k8s-cluster251 ~]#
[root@k8s-cluster251 ~]# etcdctl version
etcdctl version: 3.6.7
API version: 3.6
[root@k8s-cluster251 ~]#
3 将软件包下发到所有节点
[root@k8s-cluster251 ~]# scp /usr/local/bin/etcd* k8s-cluster252:/usr/local/bin
[root@k8s-cluster251 ~]# scp /usr/local/bin/etcd* k8s-cluster253:/usr/local/bin
1.安装cfssl证书管理工具
[root@k8s-cluster251 ~]# wget http://192.168.21.253/Resources/Prometheus/softwares/Etcd/yinzhengjie-cfssl-v1.6.5.zip
[root@k8s-cluster251 ~]# unzip yinzhengjie-cfssl-v1.6.5.zip
[root@k8s-cluster251 ~]# apt install rename
[root@k8s-cluster251 ~]# ll cfssl*
-rw-r--r-- 1 root root 11890840 Jun 15 2024 cfssl_1.6.5_linux_amd64
-rw-r--r-- 1 root root 8413336 Jun 15 2024 cfssl-certinfo_1.6.5_linux_amd64
-rw-r--r-- 1 root root 6205592 Jun 15 2024 cfssljson_1.6.5_linux_amd64
[root@k8s-cluster251 ~]#
[root@k8s-cluster251 ~]# rename -v "s/_1.6.5_linux_amd64//g" cfssl*
[root@k8s-cluster251 ~]#
[root@k8s-cluster251 ~]# ll cfssl*
-rw-r--r-- 1 root root 11890840 Jun 15 2024 cfssl
-rw-r--r-- 1 root root 8413336 Jun 15 2024 cfssl-certinfo
-rw-r--r-- 1 root root 6205592 Jun 15 2024 cfssljson
[root@k8s-cluster251 ~]#
[root@k8s-cluster251 ~]# mv cfssl* /usr/local/bin/
[root@k8s-cluster251 ~]#
[root@k8s-cluster251 ~]# ll /usr/local/bin/cfssl*
-rw-r--r-- 1 root root 11890840 Jun 15 2024 /usr/local/bin/cfssl
-rw-r--r-- 1 root root 8413336 Jun 15 2024 /usr/local/bin/cfssl-certinfo
-rw-r--r-- 1 root root 6205592 Jun 15 2024 /usr/local/bin/cfssljson
[root@k8s-cluster251 ~]#
[root@k8s-cluster251 ~]# chmod +x /usr/local/bin/cfssl*
[root@k8s-cluster251 ~]#
[root@k8s-cluster251 ~]# ll /usr/local/bin/cfssl*
-rwxr-xr-x 1 root root 11890840 Jun 15 2024 /usr/local/bin/cfssl*
-rwxr-xr-x 1 root root 8413336 Jun 15 2024 /usr/local/bin/cfssl-certinfo*
-rwxr-xr-x 1 root root 6205592 Jun 15 2024 /usr/local/bin/cfssljson*
[root@k8s-cluster251 ~]#
2.创建证书存储目录
[root@k8s-cluster251 ~]# mkdir -pv /yinzhengjie/certs/{pki,etcd}
3.生成证书的CSR文件： 证书签发请求文件，配置了一些域名，公司，单位
[root@k8s-cluster251 ~]# cd /yinzhengjie/certs/pki/
[root@k8s-cluster251 pki]#
[root@k8s-cluster251 pki]# cat > etcd-ca-csr.json <<EOF
{
"CN": "etcd",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "Beijing",
"L": "Beijing",
"O": "etcd",
"OU": "Etcd Security"
}
],
"ca": {
"expiry": "876000h"
}
}
EOF
4. 生成etcd CA证书和CA证书的key
[root@k8s-cluster251 pki]# cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare /yinzhengjie/certs/etcd/etcd-ca
2026/01/28 16:20:40 [INFO] generating a new CA key and certificate from CSR
2026/01/28 16:20:40 [INFO] generate received request
2026/01/28 16:20:40 [INFO] received CSR
2026/01/28 16:20:40 [INFO] generating key: rsa-2048
2026/01/28 16:20:40 [INFO] encoded CSR
2026/01/28 16:20:40 [INFO] signed certificate with serial number 404238825125584746046449567957149665907733832507
[root@k8s-cluster251 pki]#
[root@k8s-cluster251 pki]# ll /yinzhengjie/certs/etcd/etcd-ca*
-rw-r--r-- 1 root root 1050 Jan 28 16:20 /yinzhengjie/certs/etcd/etcd-ca.csr
-rw------- 1 root root 1675 Jan 28 16:20 /yinzhengjie/certs/etcd/etcd-ca-key.pem
-rw-r--r-- 1 root root 1318 Jan 28 16:20 /yinzhengjie/certs/etcd/etcd-ca.pem
[root@k8s-cluster251 pki]#
5. 生成etcd证书的有效期为100年
[root@k8s-cluster251 pki]# cat > ca-config.json <<EOF
{
"signing": {
"default": {
"expiry": "876000h"
},
"profiles": {
"kubernetes": {
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
],
"expiry": "876000h"
}
}
}
}
EOF
6.生成证书的CSR文件： 证书签发请求文件，配置了一些域名，公司，单位
[root@k8s-cluster251 pki]# cat > etcd-csr.json <<EOF
{
"CN": "etcd",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "Beijing",
"L": "Beijing",
"O": "etcd",
"OU": "Etcd Security"
}
]
}
EOF
7.基于自建的ectd ca证书生成etcd的证书
[root@k8s-cluster251 pki]# cfssl gencert \
-ca=/yinzhengjie/certs/etcd/etcd-ca.pem \
-ca-key=/yinzhengjie/certs/etcd/etcd-ca-key.pem \
-config=ca-config.json \
--hostname=127.0.0.1,k8s-cluster251,k8s-cluster252,k8s-cluster253,10.0.0.251,10.0.0.252,10.0.0.253 \
--profile=kubernetes \
etcd-csr.json | cfssljson -bare /yinzhengjie/certs/etcd/etcd-server
2026/01/28 16:22:02 [INFO] generate received request
2026/01/28 16:22:02 [INFO] received CSR
2026/01/28 16:22:02 [INFO] generating key: rsa-2048
2026/01/28 16:22:02 [INFO] encoded CSR
2026/01/28 16:22:02 [INFO] signed certificate with serial number 321028122811953383489486818437483615274642338969
[root@k8s-cluster251 pki]#
[root@k8s-cluster251 pki]# ll /yinzhengjie/certs/etcd/etcd-server*
-rw-r--r-- 1 root root 1139 Jan 28 16:22 /yinzhengjie/certs/etcd/etcd-server.csr
-rw------- 1 root root 1675 Jan 28 16:22 /yinzhengjie/certs/etcd/etcd-server-key.pem
-rw-r--r-- 1 root root 1472 Jan 28 16:22 /yinzhengjie/certs/etcd/etcd-server.pem
[root@k8s-cluster251 pki]#
8.将etcd证书拷贝到其他两个master节点
[root@k8s-cluster251 pki]# scp -r /yinzhengjie/ k8s-cluster252:/
[root@k8s-cluster251 pki]# scp -r /yinzhengjie/ k8s-cluster253:/
9.去其他2个节点查看证书文件
[root@k8s-cluster252 ~]# tree /yinzhengjie/
/yinzhengjie/
└── certs
├── etcd
│ ├── etcd-ca.csr
│ ├── etcd-ca-key.pem
│ ├── etcd-ca.pem
│ ├── etcd-server.csr
│ ├── etcd-server-key.pem
│ └── etcd-server.pem
└── pki
├── ca-config.json
├── etcd-ca-csr.json
└── etcd-csr.json
3 directories, 9 files
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]#
[root@k8s-cluster253 ~]# tree /yinzhengjie/
/yinzhengjie/
└── certs
├── etcd
│ ├── etcd-ca.csr
│ ├── etcd-ca-key.pem
│ ├── etcd-ca.pem
│ ├── etcd-server.csr
│ ├── etcd-server-key.pem
│ └── etcd-server.pem
└── pki
├── ca-config.json
├── etcd-ca-csr.json
└── etcd-csr.json
3 directories, 9 files
[root@k8s-cluster253 ~]#
1.k8s-cluster251节点的配置文件
[root@k8s-cluster251 ~]# mkdir -pv /yinzhengjie/softwares/etcd
[root@k8s-cluster251 ~]# cat > /yinzhengjie/softwares/etcd/etcd.config.yml <<'EOF'
name: 'k8s-cluster251'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://10.0.0.251:2380'
listen-client-urls: ' https://10.0.0.251:2379 ,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://10.0.0.251:2380'
advertise-client-urls: 'https://10.0.0.251:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-cluster251= https://10.0.0.251:2380,k8s-cluster252=https://10.0.0.252:2380 ,k8s-cluster253=https://10.0.0.253:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
cert-file: '/yinzhengjie/certs/etcd/etcd-server.pem'
key-file: '/yinzhengjie/certs/etcd/etcd-server-key.pem'
client-cert-auth: true
trusted-ca-file: '/yinzhengjie/certs/etcd/etcd-ca.pem'
auto-tls: true
peer-transport-security:
cert-file: '/yinzhengjie/certs/etcd/etcd-server.pem'
key-file: '/yinzhengjie/certs/etcd/etcd-server-key.pem'
peer-client-cert-auth: true
trusted-ca-file: '/yinzhengjie/certs/etcd/etcd-ca.pem'
auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false
EOF
2.k8s-cluster252节点的配置文件
[root@k8s-cluster252 ~]# mkdir -pv /yinzhengjie/softwares/etcd
[root@k8s-cluster252 ~]# cat > /yinzhengjie/softwares/etcd/etcd.config.yml <<'EOF'
name: 'k8s-cluster252'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://10.0.0.252:2380'
listen-client-urls: ' https://10.0.0.252:2379 ,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://10.0.0.252:2380'
advertise-client-urls: 'https://10.0.0.252:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-cluster251= https://10.0.0.251:2380,k8s-cluster252=https://10.0.0.252:2380 ,k8s-cluster253=https://10.0.0.253:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
cert-file: '/yinzhengjie/certs/etcd/etcd-server.pem'
key-file: '/yinzhengjie/certs/etcd/etcd-server-key.pem'
client-cert-auth: true
trusted-ca-file: '/yinzhengjie/certs/etcd/etcd-ca.pem'
auto-tls: true
peer-transport-security:
cert-file: '/yinzhengjie/certs/etcd/etcd-server.pem'
key-file: '/yinzhengjie/certs/etcd/etcd-server-key.pem'
peer-client-cert-auth: true
trusted-ca-file: '/yinzhengjie/certs/etcd/etcd-ca.pem'
auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false
EOF
3.k8s-cluster253节点的配置文件
[root@k8s-cluster253 ~]# mkdir -pv /yinzhengjie/softwares/etcd
[root@k8s-cluster253 ~]# cat > /yinzhengjie/softwares/etcd/etcd.config.yml <<'EOF'
name: 'k8s-cluster253'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://10.0.0.253:2380'
listen-client-urls: ' https://10.0.0.253:2379 ,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://10.0.0.253:2380'
advertise-client-urls: 'https://10.0.0.253:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-cluster251= https://10.0.0.251:2380,k8s-cluster252=https://10.0.0.252:2380 ,k8s-cluster253=https://10.0.0.253:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
cert-file: '/yinzhengjie/certs/etcd/etcd-server.pem'
key-file: '/yinzhengjie/certs/etcd/etcd-server-key.pem'
client-cert-auth: true
trusted-ca-file: '/yinzhengjie/certs/etcd/etcd-ca.pem'
auto-tls: true
peer-transport-security:
cert-file: '/yinzhengjie/certs/etcd/etcd-server.pem'
key-file: '/yinzhengjie/certs/etcd/etcd-server-key.pem'
peer-client-cert-auth: true
trusted-ca-file: '/yinzhengjie/certs/etcd/etcd-ca.pem'
auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false
EOF
1.所有节点[k8s-cluster251,k8s-cluster252,k8s-cluster253]编写etcd启动脚本
cat > /usr/lib/systemd/system/etcd.service <<'EOF'
[Unit]
Description=Jason Yin's Etcd Service
Documentation=https://coreos.com/etcd/docs/latest/
After=network.target
[Service]
Type=notify
ExecStart=/usr/local/bin/etcd --config-file=/yinzhengjie/softwares/etcd/etcd.config.yml
Restart=on-failure
RestartSec=10
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
Alias=etcd3.service
EOF
2.所有节点启动etcd集群
systemctl daemon-reload && systemctl enable --now etcd
systemctl status etcd
3.查看etcd集群状态
[root@k8s-cluster251 ~]# etcdctl --endpoints=" https://10.0.0.251:2379,https://10.0.0.252:2379 ,https://10.0.0.253:2379" --cacert=/yinzhengjie/certs/etcd/etcd-ca.pem --cert=/yinzhengjie/certs/etcd/etcd-server.pem --key=/yinzhengjie/certs/etcd/etcd-server-key.pem endpoint status --write-out=table
+-------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
+-------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| https://10.0.0.251:2379 | 566d563f3c9274ed | 3.6.7 | 3.6.0 | 20 kB | 20 kB | 0% | 2.1 GB | false | false | 2 | 9 | 9 | | | false |
| https://10.0.0.252:2379 | b83b69ba7d246b29 | 3.6.7 | 3.6.0 | 20 kB | 16 kB | 20% | 2.1 GB | true | false | 2 | 9 | 9 | | | false |
| https://10.0.0.253:2379 | 47b70f9ecb1f200 | 3.6.7 | 3.6.0 | 25 kB | 16 kB | 34% | 2.1 GB | false | false | 2 | 9 | 9 | | | false |
+-------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
[root@k8s-cluster251 ~]#
1 停止leader节点
[root@k8s-cluster252 ~]# ss -ntl | egrep "2379|2380"
LISTEN 0 4096 127.0.0.1:2379 0.0.0.0:*
LISTEN 0 4096 10.0.0.252:2379 0.0.0.0:*
LISTEN 0 4096 10.0.0.252:2380 0.0.0.0:*
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# systemctl stop etcd
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# ss -ntl | egrep "2379|2380"
[root@k8s-cluster252 ~]#
2 查看现有集群环境，发现新leader诞生
[root@k8s-cluster253 ~]# etcdctl --endpoints=" https://10.0.0.251:2379,https://10.0.0.252:2379 ,https://10.0.0.253:2379" --cacert=/oldboyedu/certs/etcd/etcd-ca.pem --cert=/oldboyedu/certs/etcd/etcd-server.pem --key=/oldboyedu/certs/etcd/etcd-server-key.pem endpoint status --write-out=table
{"level":"warn","ts":"2026-01-23T09:46:23.188084+0800","logger":"etcd-client","caller":"v3@v3.6.7/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000336f00/10.0.0.252:2379","method":"/etcdserverpb.Maintenance/Status","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = \"transport: Error while dialing: dial tcp 10.0.0.252:2379: connect: connection refused\""}
Failed to get the status of endpoint https://10.0.0.252:2379 (context deadline exceeded)
+-------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
+-------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| https://10.0.0.251:2379 | 566d563f3c9274ed | 3.6.7 | 3.6.0 | 20 kB | 20 kB | 0% | 2.1 GB | true | false | 3 | 10 | 10 | | | false |
| https://10.0.0.253:2379 | 47b70f9ecb1f200 | 3.6.7 | 3.6.0 | 25 kB | 16 kB | 34% | 2.1 GB | false | false | 3 | 10 | 10 | | | false |
+-------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
[root@k8s-cluster253 ~]#
3.由于242节点挂掉，因此我们在后端的地址链接时可以不指定该节点
[root@k8s-cluster253 ~]# etcdctl --endpoints=" https://10.0.0.251:2379 ,https://10.0.0.253:2379" --cacert=/oldboyedu/certs/etcd/etcd-ca.pem --cert=/oldboyedu/certs/etcd/etcd-server.pem --key=/oldboyedu/certs/etcd/etcd-server-key.pem endpoint status --write-out=table
+-------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
+-------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| https://10.0.0.251:2379 | 566d563f3c9274ed | 3.6.7 | 3.6.0 | 20 kB | 20 kB | 0% | 2.1 GB | true | false | 3 | 10 | 10 | | | false |
| https://10.0.0.253:2379 | 47b70f9ecb1f200 | 3.6.7 | 3.6.0 | 25 kB | 16 kB | 34% | 2.1 GB | false | false | 3 | 10 | 10 | | | false |
+-------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
[root@k8s-cluster253 ~]#
4. 再将之前的leader起来
[root@k8s-cluster252 ~]# ss -ntl | egrep "2379|2380"
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# systemctl start etcd
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# ss -ntl | egrep "2379|2380"
LISTEN 0 4096 127.0.0.1:2379 0.0.0.0:*
LISTEN 0 4096 10.0.0.252:2379 0.0.0.0:*
LISTEN 0 4096 10.0.0.252:2380 0.0.0.0:*
[root@k8s-cluster252 ~]#
5.发现并没有变回leader
[root@k8s-cluster252 ~]# etcdctl --endpoints=" https://10.0.0.251:2379,https://10.0.0.252:2379 ,https://10.0.0.253:2379" --cacert=/oldboyedu/certs/etcd/etcd-ca.pem --cert=/oldboyedu/certs/etcd/etcd-server.pem --key=/oldboyedu/certs/etcd/etcd-server-key.pem endpoint status --write-out=table
+-------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
+-------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| https://10.0.0.251:2379 | 566d563f3c9274ed | 3.6.7 | 3.6.0 | 25 kB | 25 kB | 0% | 2.1 GB | true | false | 3 | 11 | 11 | | | false |
| https://10.0.0.252:2379 | b83b69ba7d246b29 | 3.6.7 | 3.6.0 | 20 kB | 20 kB | 0% | 2.1 GB | false | false | 3 | 11 | 11 | | | false |
| https://10.0.0.253:2379 | 47b70f9ecb1f200 | 3.6.7 | 3.6.0 | 25 kB | 16 kB | 34% | 2.1 GB | false | false | 3 | 11 | 11 | | | false |
+-------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
[root@k8s-cluster252 ~]#
1 添加别名
[root@k8s-cluster251 ~]# vim ~/.bashrc
...
alias etcdctl='etcdctl --endpoints=" https://10.0.0.251:2379,https://10.0.0.252:2379 ,https://10.0.0.253:2379" --cacert=/yinzhengjie/certs/etcd/etcd-ca.pem --cert=/yinzhengjie/certs/etcd/etcd-server.pem --key=/yinzhengjie/certs/etcd/etcd-server-key.pem'
...
2.测试验证
[root@k8s-cluster251 ~]# source ~/.bashrc
[root@k8s-cluster251 ~]#
[root@k8s-cluster251 ~]# etcdctl endpoint status --write-out=table
+-------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
+-------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| https://10.0.0.251:2379 | 566d563f3c9274ed | 3.6.7 | 3.6.0 | 25 kB | 25 kB | 0% | 2.1 GB | true | false | 3 | 11 | 11 | | | false |
| https://10.0.0.252:2379 | b83b69ba7d246b29 | 3.6.7 | 3.6.0 | 20 kB | 20 kB | 0% | 2.1 GB | false | false | 3 | 11 | 11 | | | false |
| https://10.0.0.253:2379 | 47b70f9ecb1f200 | 3.6.7 | 3.6.0 | 25 kB | 16 kB | 34% | 2.1 GB | false | false | 3 | 11 | 11 | | | false |
+-------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
[root@k8s-cluster251 ~]#
[root@k8s-cluster251 ~]# etcdctl endpoint status -w table
+-------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
+-------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| https://10.0.0.251:2379 | 566d563f3c9274ed | 3.6.7 | 3.6.0 | 25 kB | 25 kB | 0% | 2.1 GB | true | false | 3 | 11 | 11 | | | false |
| https://10.0.0.252:2379 | b83b69ba7d246b29 | 3.6.7 | 3.6.0 | 20 kB | 20 kB | 0% | 2.1 GB | false | false | 3 | 11 | 11 | | | false |
| https://10.0.0.253:2379 | 47b70f9ecb1f200 | 3.6.7 | 3.6.0 | 25 kB | 16 kB | 34% | 2.1 GB | false | false | 3 | 11 | 11 | | | false |
+-------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
[root@k8s-cluster251 ~]#
3.将配置同步到其他节点
[root@k8s-cluster251 ~]# scp .bashrc k8s-cluster252:~
[root@k8s-cluster251 ~]# scp .bashrc k8s-cluster253:~
4.其他测试验证
[root@k8s-cluster252 ~]# source ~/.bashrc
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# etcdctl endpoint status -w table
+-------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
+-------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| https://10.0.0.251:2379 | 566d563f3c9274ed | 3.6.7 | 3.6.0 | 25 kB | 25 kB | 0% | 2.1 GB | true | false | 3 | 11 | 11 | | | false |
| https://10.0.0.252:2379 | b83b69ba7d246b29 | 3.6.7 | 3.6.0 | 20 kB | 20 kB | 0% | 2.1 GB | false | false | 3 | 11 | 11 | | | false |
| https://10.0.0.253:2379 | 47b70f9ecb1f200 | 3.6.7 | 3.6.0 | 25 kB | 16 kB | 34% | 2.1 GB | false | false | 3 | 11 | 11 | | | false |
+-------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]#
[root@k8s-cluster253 ~]# source ~/.bashrc
[root@k8s-cluster253 ~]#
[root@k8s-cluster253 ~]# etcdctl endpoint status -w table
+-------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
+-------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| https://10.0.0.251:2379 | 566d563f3c9274ed | 3.6.7 | 3.6.0 | 25 kB | 25 kB | 0% | 2.1 GB | true | false | 3 | 11 | 11 | | | false |
| https://10.0.0.252:2379 | b83b69ba7d246b29 | 3.6.7 | 3.6.0 | 20 kB | 20 kB | 0% | 2.1 GB | false | false | 3 | 11 | 11 | | | false |
| https://10.0.0.253:2379 | 47b70f9ecb1f200 | 3.6.7 | 3.6.0 | 25 kB | 16 kB | 34% | 2.1 GB | false | false | 3 | 11 | 11 | | | false |
+-------------------------+------------------+---------+-----------------+---------+--------+-----------------------+--------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
[root@k8s-cluster253 ~]#
etcd的操作和zookeeper，Redis的操作类似，存储数据都是键值对。
推荐阅读：
https://etcd.io/docs/v3.6/op-guide/
https://etcd.io/docs/v3.6/benchmarks/
https://etcd.io/docs/v3.6/upgrades/
1 写入数据KEY的school，value等于oldboyedu
[root@k8s-cluster253 ~]# etcdctl put school oldboyedu
OK
[root@k8s-cluster253 ~]#
[root@k8s-cluster253 ~]# etcdctl put /class linux
OK
[root@k8s-cluster253 ~]#
[root@k8s-cluster253 ~]# etcdctl put classroom 教室3
OK
[root@k8s-cluster253 ~]#
[root@k8s-cluster253 ~]# etcdctl put /etc/hosts 10.0.0.141 ceph141
OK
[root@k8s-cluster253 ~]#
[root@k8s-cluster253 ~]# etcdctl put /oldboyedu/docker/registry 轻量级镜像仓库
OK
[root@k8s-cluster253 ~]#
2 查看数据
[root@k8s-cluster252 ~]# etcdctl get school
school
oldboyedu
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# etcdctl get school --keys-only
school
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# etcdctl get school --print-value-only
oldboyedu
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# etcdctl get / --prefix --keys-only
/class
/etc/hosts
/oldboyedu/docker/registry
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# etcdctl get / --prefix --print-value-only
linux
10.0.0.141
轻量级镜像仓库
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# etcdctl get / --prefix
/class
linux
/etc/hosts
10.0.0.141
/oldboyedu/docker/registry
轻量级镜像仓库
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# etcdctl get "" --prefix --keys-only
/class
/etc/hosts
/oldboyedu/docker/registry
classroom
school
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# etcdctl get "" --prefix --print-value-only
linux
10.0.0.141
轻量级镜像仓库
教室3
oldboyedu
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# etcdctl get "" --prefix
/class
linux
/etc/hosts
10.0.0.141
/oldboyedu/docker/registry
轻量级镜像仓库
classroom
教室3
school
oldboyedu
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]#
3 修改数据
[root@k8s-cluster252 ~]# etcdctl get school --print-value-only
oldboyedu
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# etcdctl put school 老男孩IT教育
OK
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# etcdctl get school --print-value-only
老男孩IT教育
[root@k8s-cluster252 ~]#
4 删除数据
[root@k8s-cluster252 ~]# etcdctl get "" --prefix --keys-only
/class
/etc/hosts
/oldboyedu/docker/registry
classroom
school
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# etcdctl del school
1
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# etcdctl del school
0
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# etcdctl del / --prefix
3
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# etcdctl del / --prefix
0
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# etcdctl get "" --prefix --keys-only
classroom
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# etcdctl del "" --prefix
1
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# etcdctl get "" --prefix --keys-only
[root@k8s-cluster252 ~]#
在生产环境中，K8S的所有数据都存储在etcd中，我们不仅仅要保证etcd的高可用。
还要保证etcd的数据不丢失，而数据备份恢复时保证数据不丢失的有效手段。
推荐阅读:
https://etcd.io/docs/v3.5/op-guide/recovery/
https://etcd.io/docs/v3.5/op-guide/
https://etcd.io/docs/v3.5/learning/
https://etcd.io/docs/v3.5/upgrades/
1 原生数据【数据随机创建即可，用于模拟备份环节】
[root@k8s-cluster252 ~]# etcdctl put /school oldboyedu
OK
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# etcdctl put /school/linux101 教室03
OK
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# etcdctl put /school/linux101/陈康桥 洗脚
OK
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# etcdctl put /school/linux101/马春辉 修脚
OK
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# etcdctl put /school/linux101/张文杰 按摩
OK
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# etcdctl put /school/linux101/冉奥 SPA
OK
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# etcdctl put /school/linux101/李佳琪 带货
OK
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# etcdctl put /school/linux101/顾俊宁 玩手机俯卧撑
OK
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# etcdctl put /school/linux101/王铎 洗澡按摩
OK
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# etcdctl get "" --prefix
/school
oldboyedu
/school/linux101
教室03
/school/linux101/冉奥
SPA
/school/linux101/张文杰
按摩
/school/linux101/李佳琪
带货
/school/linux101/王铎
洗澡按摩
/school/linux101/陈康桥
洗脚
/school/linux101/顾俊宁
玩手机俯卧撑
/school/linux101/马春辉
修脚
[root@k8s-cluster252 ~]#
2 创建快照用于备份数据
[root@k8s-cluster252 ~]# \etcdctl snapshot save /tmp/oldboyedu-etcd-`date +%F`.backup
{"level":"info","ts":"2026-01-23T10:54:35.866164+0800","caller":"snapshot/v3_snapshot.go:83","msg":"created temporary db file","path":"/tmp/oldboyedu-etcd-2026-01-23.backup.part"}
{"level":"info","ts":"2026-01-23T10:54:35.867865+0800","logger":"client","caller":"v3@v3.6.7/maintenance.go:236","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2026-01-23T10:54:35.872632+0800","caller":"snapshot/v3_snapshot.go:96","msg":"fetching snapshot","endpoint":"127.0.0.1:2379"}
{"level":"info","ts":"2026-01-23T10:54:35.872892+0800","logger":"client","caller":"v3@v3.6.7/maintenance.go:302","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2026-01-23T10:54:35.873531+0800","caller":"snapshot/v3_snapshot.go:111","msg":"fetched snapshot","endpoint":"127.0.0.1:2379","size":"25 kB","took":"7.252294ms","etcd-version":"3.6.0"}
{"level":"info","ts":"2026-01-23T10:54:35.873716+0800","caller":"snapshot/v3_snapshot.go:121","msg":"saved","path":"/tmp/oldboyedu-etcd-2026-01-23.backup"}
Snapshot saved at /tmp/oldboyedu-etcd-2026-01-23.backup
Server version 3.6.0
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# ll /tmp/oldboyedu-etcd-2026-01-23.backup
-rw------- 1 root root 24608 Jan 23 10:54 /tmp/oldboyedu-etcd-2026-01-23.backup
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# file /tmp/oldboyedu-etcd-2026-01-23.backup
/tmp/oldboyedu-etcd-2026-01-23.backup: data
[root@k8s-cluster252 ~]#
3.准备快照查看工具
[root@k8s-cluster251 ~]# tar xf etcd-v3.6.7-linux-amd64.tar.gz -C /usr/local/bin etcd-v3.6.7-linux-amd64/etcdutl --strip-components=1
[root@k8s-cluster251 ~]#
[root@k8s-cluster251 ~]# ll /usr/local/bin/etcdutl
-rwxr-xr-x 1 yinzhengjie yinzhengjie 17244344 Dec 18 03:41 /usr/local/bin/etcdutl*
[root@k8s-cluster251 ~]#
[root@k8s-cluster251 ~]# ll /usr/local/bin/
total 85040
drwxr-xr-x 2 root root 4096 Jan 23 11:00 ./
drwxr-xr-x 10 root root 4096 Feb 17 2024 ../
-rwxr-xr-x 1 root root 11890840 Jun 15 2024 cfssl*
-rwxr-xr-x 1 root root 8413336 Jun 15 2024 cfssl-certinfo*
-rwxr-xr-x 1 root root 6205592 Jun 15 2024 cfssljson*
-rwxr-xr-x 1 yinzhengjie yinzhengjie 26087608 Dec 18 03:41 etcd*
-rwxr-xr-x 1 yinzhengjie yinzhengjie 17207480 Dec 18 03:41 etcdctl*
-rwxr-xr-x 1 yinzhengjie yinzhengjie 17244344 Dec 18 03:41 etcdutl*
[root@k8s-cluster251 ~]#
[root@k8s-cluster251 ~]# scp /usr/local/bin/etcdutl 10.0.0.252:/usr/local/bin/
[root@k8s-cluster251 ~]# scp /usr/local/bin/etcdutl 10.0.0.253:/usr/local/bin/
4.查看快照的基本信息
[root@k8s-cluster252 ~]# etcdutl snapshot status /tmp/oldboyedu-etcd-2026-01-23.backup -w table
+----------+----------+------------+------------+---------+
| HASH | REVISION | TOTAL KEYS | TOTAL SIZE | VERSION |
+----------+----------+------------+------------+---------+
| af1ef805 | 19 | 9 | 25 kB | 3.6.0 |
+----------+----------+------------+------------+---------+
[root@k8s-cluster252 ~]#
1.将快照拷贝到其他两个集群节点
[root@k8s-cluster252 ~]# scp /tmp/oldboyedu-etcd-`date +%F`.backup 10.0.0.251:/tmp
[root@k8s-cluster252 ~]# scp /tmp/oldboyedu-etcd-`date +%F`.backup 10.0.0.253:/tmp
2 删除所有数据【搞破坏】
[root@k8s-cluster252 ~]# etcdctl get "" --prefix
/school
oldboyedu
/school/linux101
教室03
/school/linux101/冉奥
SPA
/school/linux101/张文杰
按摩
/school/linux101/李佳琪
带货
/school/linux101/王铎
洗澡按摩
/school/linux101/陈康桥
洗脚
/school/linux101/顾俊宁
玩手机俯卧撑
/school/linux101/马春辉
修脚
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# etcdctl del "" --prefix
9
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# etcdctl get "" --prefix
[root@k8s-cluster252 ~]#
3 停止etcd集群
[root@k8s-cluster251 ~]# systemctl stop etcd
[root@k8s-cluster251 ~]#
[root@k8s-cluster252 ~]# systemctl stop etcd
[root@k8s-cluster252 ~]#
[root@k8s-cluster253 ~]# systemctl stop etcd
[root@k8s-cluster253 ~]#
4.各节点恢复数据 【恢复的数据目录必须为空】
[root@k8s-cluster251 ~]# ll /var/lib/etcd-2026
ls: cannot access '/var/lib/etcd-2026': No such file or directory
[root@k8s-cluster251 ~]#
[root@k8s-cluster251 ~]# etcdutl snapshot restore /tmp/oldboyedu-etcd-`date +%F`.backup --data-dir=/var/lib/etcd-2026
2026-01-23T11:06:51+08:00 info snapshot/v3_snapshot.go:305 restoring snapshot {"path": "/tmp/oldboyedu-etcd-2026-01-23.backup", "wal-dir": "/var/lib/etcd-2026/member/wal", "data-dir": "/var/lib/etcd-2026", "snap-dir": "/var/lib/etcd-2026/member/snap", "initial-memory-map-size": 10737418240}
2026-01-23T11:06:51+08:00 info bbolt backend/backend.go:203 Opening db file (/var/lib/etcd-2026/member/snap/db) with mode -rw------- and with options: {Timeout: 0s, NoGrowSync: false, NoFreelistSync: true, PreLoadFreelist: false, FreelistType: , ReadOnly: false, MmapFlags: 8000, InitialMmapSize: 10737418240, PageSize: 0, NoSync: false, OpenFile: 0x0, Mlock: false, Logger: 0xc000066178}
2026-01-23T11:06:51+08:00 info bbolt bbolt@v1.4.3/db.go:321 Opening bbolt db (/var/lib/etcd-2026/member/snap/db) successfully
2026-01-23T11:06:51+08:00 info schema/membership.go:138 Trimming membership information from the backend...
2026-01-23T11:06:51+08:00 info bbolt backend/backend.go:203 Opening db file (/var/lib/etcd-2026/member/snap/db) with mode -rw------- and with options: {Timeout: 0s, NoGrowSync: false, NoFreelistSync: true, PreLoadFreelist: false, FreelistType: , ReadOnly: false, MmapFlags: 8000, InitialMmapSize: 10737418240, PageSize: 0, NoSync: false, OpenFile: 0x0, Mlock: false, Logger: 0xc0000661c0}
2026-01-23T11:06:51+08:00 info bbolt bbolt@v1.4.3/db.go:321 Opening bbolt db (/var/lib/etcd-2026/member/snap/db) successfully
2026-01-23T11:06:51+08:00 info membership/cluster.go:424 added member {"cluster-id": "cdf818194e3a8c32", "local-member-id": "0", "added-peer-id": "8e9e05c52164694d", "added-peer-peer-urls": ["http://localhost:2380"], "added-peer-is-learner": false}
2026-01-23T11:06:51+08:00 info bbolt backend/backend.go:203 Opening db file (/var/lib/etcd-2026/member/snap/db) with mode -rw------- and with options: {Timeout: 0s, NoGrowSync: false, NoFreelistSync: true, PreLoadFreelist: false, FreelistType: , ReadOnly: false, MmapFlags: 8000, InitialMmapSize: 10737418240, PageSize: 0, NoSync: false, OpenFile: 0x0, Mlock: false, Logger: 0xc000066368}
2026-01-23T11:06:51+08:00 info bbolt bbolt@v1.4.3/db.go:321 Opening bbolt db (/var/lib/etcd-2026/member/snap/db) successfully
2026-01-23T11:06:51+08:00 info snapshot/v3_snapshot.go:333 restored snapshot {"path": "/tmp/oldboyedu-etcd-2026-01-23.backup", "wal-dir": "/var/lib/etcd-2026/member/wal", "data-dir": "/var/lib/etcd-2026", "snap-dir": "/var/lib/etcd-2026/member/snap", "initial-memory-map-size": 10737418240}
[root@k8s-cluster251 ~]#
[root@k8s-cluster251 ~]# ll /var/lib/etcd-2026
total 12
drwx------ 3 root root 4096 Jan 23 11:06 ./
drwxr-xr-x 62 root root 4096 Jan 23 11:06 ../
drwx------ 4 root root 4096 Jan 23 11:06 member/
[root@k8s-cluster251 ~]#
[root@k8s-cluster252 ~]# ll /var/lib/etcd-2026
ls: cannot access '/var/lib/etcd-2026': No such file or directory
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# etcdutl snapshot restore /tmp/oldboyedu-etcd-`date +%F`.backup --data-dir=/var/lib/etcd-2026
2026-01-23T11:06:18+08:00 info snapshot/v3_snapshot.go:305 restoring snapshot {"path": "/tmp/oldboyedu-etcd-2026-01-23.backup", "wal-dir": "/var/lib/etcd-2026/member/wal", "data-dir": "/var/lib/etcd-2026", "snap-dir": "/var/lib/etcd-2026/member/snap", "initial-memory-map-size": 10737418240}
2026-01-23T11:06:18+08:00 info bbolt backend/backend.go:203 Opening db file (/var/lib/etcd-2026/member/snap/db) with mode -rw------- and with options: {Timeout: 0s, NoGrowSync: false, NoFreelistSync: true, PreLoadFreelist: false, FreelistType: , ReadOnly: false, MmapFlags: 8000, InitialMmapSize: 10737418240, PageSize: 0, NoSync: false, OpenFile: 0x0, Mlock: false, Logger: 0xc00011cce0}
2026-01-23T11:06:18+08:00 info bbolt bbolt@v1.4.3/db.go:321 Opening bbolt db (/var/lib/etcd-2026/member/snap/db) successfully
2026-01-23T11:06:18+08:00 info schema/membership.go:138 Trimming membership information from the backend...
2026-01-23T11:06:18+08:00 info bbolt backend/backend.go:203 Opening db file (/var/lib/etcd-2026/member/snap/db) with mode -rw------- and with options: {Timeout: 0s, NoGrowSync: false, NoFreelistSync: true, PreLoadFreelist: false, FreelistType: , ReadOnly: false, MmapFlags: 8000, InitialMmapSize: 10737418240, PageSize: 0, NoSync: false, OpenFile: 0x0, Mlock: false, Logger: 0xc00011c190}
2026-01-23T11:06:18+08:00 info bbolt bbolt@v1.4.3/db.go:321 Opening bbolt db (/var/lib/etcd-2026/member/snap/db) successfully
2026-01-23T11:06:18+08:00 info membership/cluster.go:424 added member {"cluster-id": "cdf818194e3a8c32", "local-member-id": "0", "added-peer-id": "8e9e05c52164694d", "added-peer-peer-urls": ["http://localhost:2380"], "added-peer-is-learner": false}
2026-01-23T11:06:18+08:00 info bbolt backend/backend.go:203 Opening db file (/var/lib/etcd-2026/member/snap/db) with mode -rw------- and with options: {Timeout: 0s, NoGrowSync: false, NoFreelistSync: true, PreLoadFreelist: false, FreelistType: , ReadOnly: false, MmapFlags: 8000, InitialMmapSize: 10737418240, PageSize: 0, NoSync: false, OpenFile: 0x0, Mlock: false, Logger: 0xc00011c828}
2026-01-23T11:06:18+08:00 info bbolt bbolt@v1.4.3/db.go:321 Opening bbolt db (/var/lib/etcd-2026/member/snap/db) successfully
2026-01-23T11:06:18+08:00 info snapshot/v3_snapshot.go:333 restored snapshot {"path": "/tmp/oldboyedu-etcd-2026-01-23.backup", "wal-dir": "/var/lib/etcd-2026/member/wal", "data-dir": "/var/lib/etcd-2026", "snap-dir": "/var/lib/etcd-2026/member/snap", "initial-memory-map-size": 10737418240}
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# ll /var/lib/etcd-2026
total 12
drwx------ 3 root root 4096 Jan 23 11:06 ./
drwxr-xr-x 62 root root 4096 Jan 23 11:06 ../
drwx------ 4 root root 4096 Jan 23 11:06 member/
[root@k8s-cluster252 ~]#
[root@k8s-cluster253 ~]# ll /var/lib/etcd-2026
ls: cannot access '/var/lib/etcd-2026': No such file or directory
[root@k8s-cluster253 ~]#
[root@k8s-cluster253 ~]# etcdutl snapshot restore /tmp/oldboyedu-etcd-`date +%F`.backup --data-dir=/var/lib/etcd-2026
2026-01-23T11:05:22+08:00 info snapshot/v3_snapshot.go:305 restoring snapshot {"path": "/tmp/oldboyedu-etcd-2026-01-23.backup", "wal-dir": "/var/lib/etcd-2026/member/wal", "data-dir": "/var/lib/etcd-2026", "snap-dir": "/var/lib/etcd-2026/member/snap", "initial-memory-map-size": 10737418240}
2026-01-23T11:05:22+08:00 info bbolt backend/backend.go:203 Opening db file (/var/lib/etcd-2026/member/snap/db) with mode -rw------- and with options: {Timeout: 0s, NoGrowSync: false, NoFreelistSync: true, PreLoadFreelist: false, FreelistType: , ReadOnly: false, MmapFlags: 8000, InitialMmapSize: 10737418240, PageSize: 0, NoSync: false, OpenFile: 0x0, Mlock: false, Logger: 0xc00011cce0}
2026-01-23T11:05:22+08:00 info bbolt bbolt@v1.4.3/db.go:321 Opening bbolt db (/var/lib/etcd-2026/member/snap/db) successfully
2026-01-23T11:05:22+08:00 info schema/membership.go:138 Trimming membership information from the backend...
2026-01-23T11:05:22+08:00 info bbolt backend/backend.go:203 Opening db file (/var/lib/etcd-2026/member/snap/db) with mode -rw------- and with options: {Timeout: 0s, NoGrowSync: false, NoFreelistSync: true, PreLoadFreelist: false, FreelistType: , ReadOnly: false, MmapFlags: 8000, InitialMmapSize: 10737418240, PageSize: 0, NoSync: false, OpenFile: 0x0, Mlock: false, Logger: 0xc00011c190}
2026-01-23T11:05:22+08:00 info bbolt bbolt@v1.4.3/db.go:321 Opening bbolt db (/var/lib/etcd-2026/member/snap/db) successfully
2026-01-23T11:05:22+08:00 info membership/cluster.go:424 added member {"cluster-id": "cdf818194e3a8c32", "local-member-id": "0", "added-peer-id": "8e9e05c52164694d", "added-peer-peer-urls": ["http://localhost:2380"], "added-peer-is-learner": false}
2026-01-23T11:05:22+08:00 info bbolt backend/backend.go:203 Opening db file (/var/lib/etcd-2026/member/snap/db) with mode -rw------- and with options: {Timeout: 0s, NoGrowSync: false, NoFreelistSync: true, PreLoadFreelist: false, FreelistType: , ReadOnly: false, MmapFlags: 8000, InitialMmapSize: 10737418240, PageSize: 0, NoSync: false, OpenFile: 0x0, Mlock: false, Logger: 0xc00011c818}
2026-01-23T11:05:22+08:00 info bbolt bbolt@v1.4.3/db.go:321 Opening bbolt db (/var/lib/etcd-2026/member/snap/db) successfully
2026-01-23T11:05:22+08:00 info snapshot/v3_snapshot.go:333 restored snapshot {"path": "/tmp/oldboyedu-etcd-2026-01-23.backup", "wal-dir": "/var/lib/etcd-2026/member/wal", "data-dir": "/var/lib/etcd-2026", "snap-dir": "/var/lib/etcd-2026/member/snap", "initial-memory-map-size": 10737418240}
[root@k8s-cluster253 ~]#
[root@k8s-cluster253 ~]# ll /var/lib/etcd-2026
total 12
drwx------ 3 root root 4096 Jan 23 11:05 ./
drwxr-xr-x 62 root root 4096 Jan 23 11:05 ../
drwx------ 4 root root 4096 Jan 23 11:05 member/
[root@k8s-cluster253 ~]#
5 将恢复后的数据目录作为新的数据目录
[root@k8s-cluster251 ~]# grep "/var/lib/etcd" /oldboyedu/softwares/etcd/etcd.config.yml
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
[root@k8s-cluster251 ~]#
[root@k8s-cluster251 ~]# sed -ri "s#(/var/lib/etcd)#\1-2026#g" /oldboyedu/softwares/etcd/etcd.config.yml
[root@k8s-cluster251 ~]#
[root@k8s-cluster251 ~]# grep "/var/lib/etcd" /oldboyedu/softwares/etcd/etcd.config.yml
data-dir: /var/lib/etcd-2026
wal-dir: /var/lib/etcd-2026/wal
[root@k8s-cluster251 ~]#
[root@k8s-cluster252 ~]# sed -ri "s#(/var/lib/etcd)#\1-2026#g" /oldboyedu/softwares/etcd/etcd.config.yml
[root@k8s-cluster252 ~]#
[root@k8s-cluster252 ~]# grep "/var/lib/etcd" /oldboyedu/softwares/etcd/etcd.config.yml
data-dir: /var/lib/etcd-2026
wal-dir: /var/lib/etcd-2026/wal
[root@k8s-cluster252 ~]#
[root@k8s-cluster253 ~]# sed -ri "s#(/var/lib/etcd)#\1-2026#g" /oldboyedu/softwares/etcd/etcd.config.yml
[root@k8s-cluster253 ~]#
[root@k8s-cluster253 ~]# grep "/var/lib/etcd" /oldboyedu/softwares/etcd/etcd.config.yml
data-dir: /var/lib/etcd-2026
wal-dir: /var/lib/etcd-2026/wal
[root@k8s-cluster253 ~]#
6 启动etcd集群
[root@k8s-cluster251 ~]# systemctl start etcd
[root@k8s-cluster251 ~]#
[root@k8s-cluster252 ~]# systemctl start etcd
[root@k8s-cluster252 ~]#
[root@k8s-cluster253 ~]# systemctl start etcd
[root@k8s-cluster253 ~]#
7.验证数据是否恢复
[root@k8s-cluster251 ~]# etcdctl get "" --prefix
/school
oldboyedu
/school/linux101
教室03
/school/linux101/冉奥
SPA
/school/linux101/张文杰
按摩
/school/linux101/李佳琪
带货
/school/linux101/王铎
洗澡按摩
/school/linux101/陈康桥
洗脚
/school/linux101/顾俊宁
玩手机俯卧撑
/school/linux101/马春辉
修脚
[root@k8s-cluster251 ~]#
8.测试数据是否可以正常读写
[root@k8s-cluster251 ~]# etcdctl put /school/linux101/崔雅栋 游戏
OK
[root@k8s-cluster251 ~]#
[root@k8s-cluster251 ~]# etcdctl get "" --prefix
/school
oldboyedu
/school/linux101
教室03
/school/linux101/冉奥
SPA
/school/linux101/崔雅栋
游戏
/school/linux101/张文杰
按摩
/school/linux101/李佳琪
带货
/school/linux101/王铎
洗澡按摩
/school/linux101/陈康桥
洗脚
/school/linux101/顾俊宁
玩手机俯卧撑
/school/linux101/马春辉
修脚
[root@k8s-cluster251 ~]#
请使用k8s的控制器，实现周期性备份etcd数据，并进行还原测试。
要求每分钟备份一次数据。
1.准备etcdctl和etcdutl程序
[root@master231 ~]# mkdir etcd-backup
[root@master231 ~]#
[root@master231 ~]# cd etcd-backup
[root@master231 etcd-backup]#
[root@master231 etcd-backup]# scp 10.0.0.251:/usr/local/bin/etcd{utl,ctl} ./
2.推送基础镜像到harbor仓库
[root@master231 ~]# wget http://192.168.21.253/Resources/Docker/images/Linux/alpine-v3.20.2.tar.gz
[root@master231 ~]# docker load -i alpine-v3.20.2.tar.gz
78561cef0761: Loading layer [==================================================>] 8.082MB/8.082MB
Loaded image: alpine:3.20.2
[root@master231 ~]#
[root@master231 ~]# docker tag alpine:3.20.2 harbor250.oldboyedu.com/oldboyedu-linux/alpine:3.20.2
[root@master231 ~]#
[root@master231 ~]# docker push harbor250.oldboyedu.com/oldboyedu-linux/alpine:3.20.2
The push refers to repository [harbor250.oldboyedu.com/oldboyedu-linux/alpine]
78561cef0761: Pushed
3.20.2: digest: sha256:eddacbc7e24bf8799a4ed3cdcfa50d4b88a323695ad80f317b6629883b2c2a78 size: 528
[root@master231 ~]#
3.编写Dockerfile
[root@master231 etcd-backup]# cat Dockerfile
FROM harbor250.oldboyedu.com/oldboyedu-linux/alpine:3.20.2
MAINTAINER auther=JasonYin
LABEL school=oldboyedu \
class=linux101
COPY etcdutl /usr/local/bin
COPY etcdctl /usr/local/bin
CMD ["tail","-f","/etc/hosts"]
[root@master231 etcd-backup]#
4.编译镜像
[root@master231 etcd-backup]# docker build -t harbor250.oldboyedu.com/oldboyedu-case-demo/etcdutl:v0.1 .
5.将镜像推送到harbor仓库
[root@master231 etcd-backup]# docker push harbor250.oldboyedu.com/oldboyedu-case-demo/etcdutl:v0.1
1.准备etcd证书文件
[root@master231 etcd-backup]# scp 10.0.0.251:/oldboyedu/certs/etcd/etcd-{ca,server,server-key}.pem ./
2.将证书文件存储为secret
[root@master231 etcd-backup]# ll *.pem
-rw-r--r-- 1 root root 1318 Jan 23 12:20 etcd-ca.pem
-rw------- 1 root root 1675 Jan 23 12:20 etcd-server-key.pem
-rw-r--r-- 1 root root 1472 Jan 23 12:20 etcd-server.pem
[root@master231 etcd-backup]#
[root@master231 etcd-backup]# kubectl create secret generic etcd-certs --from-file=etcd-ca=etcd-ca.pem --from-file=etcd-server-key=etcd-server-key.pem --from-file=etcd-server=etcd-server.pem
secret/etcd-certs created
[root@master231 etcd-backup]#
[root@master231 etcd-backup]# kubectl get secrets etcd-certs
NAME TYPE DATA AGE
etcd-certs Opaque 3 7s
[root@master231 etcd-backup]#
3.编写资源清单
[root@master231 cronjobs]# cat 02-cj-etcd-backup.yaml
apiVersion: v1
kind: Endpoints
metadata:
name: etcd-cluster
subsets:
- addresses:
- ip: 10.0.0.251
- ip: 10.0.0.252
- ip: 10.0.0.253
ports:
- port: 2379
---
apiVersion: v1
kind: Service
metadata:
name: etcd-cluster
spec:
type: ClusterIP
ports:
- protocol: TCP
port: 2379
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-etcd
spec:
accessModes:
- ReadWriteMany
resources:
limits:
storage: 2G
requests:
storage: 1G
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: cj-etcdutl
spec:
schedule: "* * * * *"
jobTemplate:
spec:
template:
spec:
volumes:
- name: dt
hostPath:
path: /etc/localtime
- name: certs
secret:
secretName: etcd-certs
- name: backup
persistentVolumeClaim:
claimName: pvc-etcd
restartPolicy: OnFailure
containers:
- name: c1
image: harbor250.oldboyedu.com/oldboyedu-case-demo/etcdutl:v0.1
command:
- /bin/sh
- -c
- etcdctl --endpoints=https://10.0.0.251:2379 --cacert=/data/etcd-ca --cert=/data/etcd-server --key=/data/etcd-server-key snapshot save /tmp/oldboyedu-etcd-`date +%F-%T`.backup
volumeMounts:
- name: dt
mountPath: /etc/localtime
- name: certs
mountPath: /data
- name: backup
mountPath: /tmp
[root@master231 cronjobs]#
[root@master231 cronjobs]#
4.创建资源
[root@master231 cronjobs]# kubectl apply -f 02-cj-etcd-backup.yaml
endpoints/etcd-cluster created
service/etcd-cluster created
persistentvolumeclaim/pvc-etcd created
cronjob.batch/cj-etcdutl created
[root@master231 cronjobs]#
[root@master231 cronjobs]# kubectl get cj,jobs,po -o wide
NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE CONTAINERS IMAGES SELECTOR
cronjob.batch/cj-etcdutl * * * * * False 0 25s 109s c1 harbor250.oldboyedu.com/oldboyedu-case-demo/etcdutl:v0.1 <none>
NAME COMPLETIONS DURATION AGE CONTAINERS IMAGES SELECTOR
job.batch/cj-etcdutl-29485725 1/1 3s 85s c1 harbor250.oldboyedu.com/oldboyedu-case-demo/etcdutl:v0.1 controller-uid=561600d1-d691-49d1-bfa4-f78cc62c893e
job.batch/cj-etcdutl-29485726 1/1 3s 25s c1 harbor250.oldboyedu.com/oldboyedu-case-demo/etcdutl:v0.1 controller-uid=ba8dbaa3-3db0-4c48-9082-349cc86c9eb3
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/cj-etcdutl-29485725-dqwbc 0/1 Completed 0 85s 10.100.2.35 worker233 <none> <none>
pod/cj-etcdutl-29485726-hk9zk 0/1 Completed 0 25s 10.100.2.36 worker233 <none> <none>
[root@master231 cronjobs]#
5.验证后端数据
[root@master231 cronjobs]# kubectl get pvc pvc-etcd
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
pvc-etcd Bound pvc-c2852f05-7dbb-4112-a37d-eddad14dc7eb 1G RWX oldboyedu-sc-haha 60s
[root@master231 cronjobs]#
[root@master231 cronjobs]# kubectl describe pv pvc-c2852f05-7dbb-4112-a37d-eddad14dc7eb | grep Source -A 10
Source:
Type: CSI (a Container Storage Interface (CSI) volume source)
Driver: nfs.csi.k8s.io
FSType:
VolumeHandle: 10.0.0.231#yinzhengjie/data/nfs-server/sc-haha#pvc-c2852f05-7dbb-4112-a37d-eddad14dc7eb##
ReadOnly: false
VolumeAttributes: csi.storage.k8s.io/pv/name=pvc-c2852f05-7dbb-4112-a37d-eddad14dc7eb
csi.storage.k8s.io/pvc/name=pvc-etcd
csi.storage.k8s.io/pvc/namespace=default
server=10.0.0.231
share=/yinzhengjie/data/nfs-server/sc-haha
[root@master231 cronjobs]#
[root@master231 cronjobs]# ll /yinzhengjie/data/nfs-server/sc-haha/pvc-c2852f05-7dbb-4112-a37d-eddad14dc7eb/
total 64
drwxr-xr-x 2 root root 4096 Jan 23 12:46 ./
drwxr-xr-x 6 root root 4096 Jan 23 12:44 ../
-rw------- 1 root root 24608 Jan 23 12:45 oldboyedu-etcd-2026-01-23-12:45:01.backup
-rw------- 1 root root 24608 Jan 23 12:46 oldboyedu-etcd-2026-01-23-12:46:00.backup
[root@master231 cronjobs]#
[root@master231 cronjobs]#
1.编写资源清单
[root@master231 jobs]# cat 02-jobs-etcd-recover.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-recover
spec:
accessModes:
- ReadWriteMany
resources:
limits:
storage: 2G
requests:
storage: 1G
---
apiVersion: batch/v1
kind: Job
metadata:
name: jobs-recover-etcd
spec:
template:
spec:
volumes:
- name: dt
hostPath:
path: /etc/localtime
- name: certs
secret:
secretName: etcd-certs
- name: backup
persistentVolumeClaim:
claimName: pvc-etcd
- name: recover
persistentVolumeClaim:
claimName: pvc-recover
containers:
- name: c1
image: harbor250.oldboyedu.com/oldboyedu-case-demo/etcdutl:v0.1
volumeMounts:
- name: dt
mountPath: /etc/localtime
- name: certs
mountPath: /data
- name: backup
mountPath: /tmp
- name: recover
mountPath: /recover-etcd-data
env:
- name: FILENAME
value: oldboyedu-etcd-2026-01-23-15:57:00.backup
command:
- /bin/sh
- -c
- etcdutl snapshot restore /tmp/$FILENAME --data-dir=/recover-etcd-data
restartPolicy: Never
backoffLimit: 4
[root@master231 jobs]#
2.创建资源
[root@master231 jobs]# kubectl apply -f 02-jobs-etcd-recover.yaml
persistentvolumeclaim/pvc-recover created
job.batch/jobs-recover-etcd created
[root@master231 jobs]#
[root@master231 jobs]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
cj-etcdutl-29485927-54nv9 0/1 Completed 0 2m10s 10.100.2.238 worker233 <none> <none>
cj-etcdutl-29485928-bqzft 0/1 Completed 0 70s 10.100.2.240 worker233 <none> <none>
cj-etcdutl-29485929-cqk29 0/1 Completed 0 10s 10.100.2.241 worker233 <none> <none>
jobs-recover-etcd-pw5x2 0/1 Completed 0 3s 10.100.2.242 worker233 <none> <none>
[root@master231 jobs]#
3.验证后台数据
[root@master231 jobs]# kubectl get pvc pvc-recover
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
pvc-recover Bound pvc-53f32ff8-9b41-49a6-be2b-17b0f7622354 1G RWX oldboyedu-sc-haha 14s
[root@master231 jobs]#
[root@master231 jobs]# kubectl describe pv pvc-53f32ff8-9b41-49a6-be2b-17b0f7622354 | grep VolumeHandle
VolumeHandle: 10.0.0.231#yinzhengjie/data/nfs-server/sc-haha#pvc-53f32ff8-9b41-49a6-be2b-17b0f7622354##
[root@master231 jobs]#
[root@master231 jobs]# ll /yinzhengjie/data/nfs-server/sc-haha/pvc-53f32ff8-9b41-49a6-be2b-17b0f7622354/
total 12
drwxr-xr-x 3 root root 4096 Jan 23 16:09 ./
drwxr-xr-x 7 root root 4096 Jan 23 16:09 ../
drwx------ 4 root root 4096 Jan 23 16:09 member/
[root@master231 jobs]#
1.清空etcd集群数据
[root@k8s-cluster251 ~]# etcdctl get "" --prefix
/school
oldboyedu
/school/linux101
教室03
/school/linux101/冉奥
SPA
/school/linux101/崔雅栋
游戏
/school/linux101/张文杰
按摩
/school/linux101/李佳琪
带货
/school/linux101/王铎
洗澡按摩
/school/linux101/陈康桥
洗脚
/school/linux101/顾俊宁
玩手机俯卧撑
/school/linux101/马春辉
修脚
[root@k8s-cluster251 ~]#
[root@k8s-cluster251 ~]#
[root@k8s-cluster251 ~]# etcdctl del "" --prefix
10
[root@k8s-cluster251 ~]#
[root@k8s-cluster251 ~]# etcdctl get "" --prefix
[root@k8s-cluster251 ~]#
2.停止服务
[root@k8s-cluster251 ~]# systemctl stop etcd
[root@k8s-cluster252 ~]# systemctl stop etcd
[root@k8s-cluster253 ~]# systemctl stop etcd
3.将个节点的数据目录清除
[root@k8s-cluster251 ~]# rm -rf /var/lib/etcd-2026
[root@k8s-cluster251 ~]#
[root@k8s-cluster252 ~]# rm -rf /var/lib/etcd-2026
[root@k8s-cluster252 ~]#
[root@k8s-cluster253 ~]# rm -rf /var/lib/etcd-2026
[root@k8s-cluster253 ~]#
4.将恢复数据重新拷贝到etcd集群
[root@master231 jobs]# scp -r /yinzhengjie/data/nfs-server/sc-haha/pvc-53f32ff8-9b41-49a6-be2b-17b0f7622354/ 10.0.0.251:/var/lib/etcd-2026
[root@master231 jobs]# scp -r /yinzhengjie/data/nfs-server/sc-haha/pvc-53f32ff8-9b41-49a6-be2b-17b0f7622354/ 10.0.0.252:/var/lib/etcd-2026
[root@master231 jobs]# scp -r /yinzhengjie/data/nfs-server/sc-haha/pvc-53f32ff8-9b41-49a6-be2b-17b0f7622354/ 10.0.0.253:/var/lib/etcd-2026
5.etcd集群启动测试
[root@k8s-cluster251 ~]# systemctl start etcd
[root@k8s-cluster252 ~]# systemctl start etcd
[root@k8s-cluster253 ~]# systemctl start etcd
6.验证数据是否恢复
[root@k8s-cluster252 ~]# etcdctl get "" --prefix
/school
oldboyedu
/school/linux101
教室03
/school/linux101/冉奥
SPA
/school/linux101/崔雅栋
游戏
/school/linux101/张文杰
按摩
/school/linux101/李佳琪
带货
/school/linux101/王铎
洗澡按摩
/school/linux101/陈康桥
洗脚
/school/linux101/顾俊宁
玩手机俯卧撑
/school/linux101/马春辉
修脚
[root@k8s-cluster252 ~]#
7.测试写入数据
[root@k8s-cluster252 ~]# etcdctl put /school/linux101/郭开庆 俯卧撑
OK
[root@k8s-cluster252 ~]#
8.其他节点查看数据
[root@k8s-cluster253 ~]# etcdctl get "" --prefix
/school
oldboyedu
/school/linux101
教室03
/school/linux101/冉奥
SPA
/school/linux101/崔雅栋
游戏
/school/linux101/张文杰
按摩
/school/linux101/李佳琪
带货
/school/linux101/王铎
洗澡按摩
/school/linux101/郭开庆
俯卧撑
/school/linux101/陈康桥
洗脚
/school/linux101/顾俊宁
玩手机俯卧撑
/school/linux101/马春辉
修脚
[root@k8s-cluster253 ~]#
1.查看etcd的静态Pod资源清单
[root@master231 ~]# cat /etc/kubernetes/manifests/etcd.yaml
apiVersion: v1
kind: Pod
metadata:
...
labels:
component: etcd
tier: control-plane
name: etcd
namespace: kube-system
spec:
containers:
- command:
- etcd
- --advertise-client-urls=https://10.0.0.231:2379
- --cert-file=/etc/kubernetes/pki/etcd/server.crt
- --client-cert-auth=true
- --data-dir=/var/lib/etcd
- --experimental-initial-corrupt-check=true
- --initial-advertise-peer-urls=https://10.0.0.231:2380
- --initial-cluster=master231=https://10.0.0.231:2380
- --key-file=/etc/kubernetes/pki/etcd/server.key
- --listen-client-urls= https://127.0.0.1:2379 ,https://10.0.0.231:2379
- --listen-metrics-urls=http://127.0.0.1:2381
- --listen-peer-urls=https://10.0.0.231:2380
- --name=master231
- --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
- --peer-client-cert-auth=true
- --peer-key-file=/etc/kubernetes/pki/etcd/peer.key
- --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
- --snapshot-count=10000
- --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
image: registry.aliyuncs.com/google_containers/etcd:3.5.6-0
...
name: etcd
...
volumeMounts:
- mountPath: /var/lib/etcd
name: etcd-data
- mountPath: /etc/kubernetes/pki/etcd
name: etcd-certs
hostNetwork: true
...
volumes:
- hostPath:
path: /etc/kubernetes/pki/etcd
type: DirectoryOrCreate
name: etcd-certs
- hostPath:
path: /var/lib/etcd
type: DirectoryOrCreate
name: etcd-data
status: {}
[root@master231 ~]#
[root@master231 ~]# ss -ntl | grep 2379
LISTEN 0 16384 10.0.0.231:2379 0.0.0.0:*
LISTEN 0 16384 127.0.0.1:2379 0.0.0.0:*
[root@master231 ~]#
2.验证数据
[root@master231 ~]# ll /etc/kubernetes/pki/etcd/
total 40
drwxr-xr-x 2 root root 4096 Jan 6 11:29 ./
drwxr-xr-x 3 root root 4096 Jan 14 10:35 ../
-rw-r--r-- 1 root root 1086 Jan 6 11:29 ca.crt
-rw------- 1 root root 1679 Jan 6 11:29 ca.key
-rw-r--r-- 1 root root 1159 Jan 6 11:29 healthcheck-client.crt
-rw------- 1 root root 1675 Jan 6 11:29 healthcheck-client.key
-rw-r--r-- 1 root root 1200 Jan 6 11:29 peer.crt
-rw------- 1 root root 1679 Jan 6 11:29 peer.key
-rw-r--r-- 1 root root 1200 Jan 6 11:29 server.crt
-rw------- 1 root root 1679 Jan 6 11:29 server.key
[root@master231 ~]#
[root@master231 ~]# ll /var/lib/etcd/
total 12
drwx------ 3 root root 4096 Jan 6 12:30 ./
drwxr-xr-x 66 root root 4096 Jan 19 10:36 ../
drwx------ 4 root root 4096 Jan 6 12:30 member/
[root@master231 ~]#
3.安装etcdctl客户端工具
[root@master231 ~]# scp 10.0.0.251:/usr/local/bin/etcd{ctl,utl} /usr/local/bin/
4.访问etcd数据
[root@master231 ~]# etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key get / --prefix --keys-only
[root@master231 ~]# etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key get / --prefix --keys-only | wc -l
1940
[root@master231 ~]#
5.验证k8s的数据存储在etcd中
[root@master231 ~]# etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key get / --prefix --keys-only | grep pods
/registry/pods/default/cj-etcdutl-29485979-8vjll
/registry/pods/default/cj-etcdutl-29485980-rszbl
/registry/pods/default/cj-etcdutl-29485981-h8dbb
/registry/pods/default/jobs-recover-etcd-pw5x2
/registry/pods/kube-flannel/kube-flannel-ds-h24tn
/registry/pods/kube-flannel/kube-flannel-ds-rqhp7
/registry/pods/kube-flannel/kube-flannel-ds-x5fjx
/registry/pods/kube-public/xiuxian-oldboyedu-linux-6847ff7955-9jq45
/registry/pods/kube-system/coredns-6d8c4cb4d-fspfl
/registry/pods/kube-system/coredns-6d8c4cb4d-jjwhx
/registry/pods/kube-system/csi-nfs-controller-5c5c695fb-678qj
/registry/pods/kube-system/csi-nfs-node-5jssl
/registry/pods/kube-system/csi-nfs-node-7rdh5
/registry/pods/kube-system/csi-nfs-node-bxsx9
/registry/pods/kube-system/etcd-master231
/registry/pods/kube-system/kube-apiserver-master231
/registry/pods/kube-system/kube-controller-manager-master231
/registry/pods/kube-system/kube-proxy-r9jxv
/registry/pods/kube-system/kube-proxy-ts66f
/registry/pods/kube-system/kube-proxy-xm74s
/registry/pods/kube-system/kube-scheduler-master231
/registry/pods/kube-system/metrics-server-6b4f784878-cd585
/registry/pods/kube-system/metrics-server-6b4f784878-rsp6r
/registry/pods/kubernetes-dashboard/dashboard-metrics-scraper-9d986c98c-nrkwx
/registry/pods/kubernetes-dashboard/kubernetes-dashboard-5ccf77bb87-6z5dr
/registry/pods/metallb-system/controller-644c958987-mllg5
/registry/pods/metallb-system/speaker-8xpkb
/registry/pods/metallb-system/speaker-kxlx2
/registry/pods/metallb-system/speaker-pghbm
/registry/pods/traefik/traefik-server-6df7595d6f-ttslc
[root@master231 ~]#
[root@master231 ~]#
[root@master231 ~]# kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
default cj-etcdutl-29485979-8vjll 0/1 Completed 0 2m15s
default cj-etcdutl-29485980-rszbl 0/1 Completed 0 75s
default cj-etcdutl-29485981-h8dbb 0/1 Completed 0 15s
default jobs-recover-etcd-pw5x2 0/1 Completed 0 52m
kube-flannel kube-flannel-ds-h24tn 1/1 Running 0 24h
kube-flannel kube-flannel-ds-rqhp7 1/1 Running 0 2d4h
kube-flannel kube-flannel-ds-x5fjx 1/1 Running 0 2d4h
kube-public xiuxian-oldboyedu-linux-6847ff7955-9jq45 1/1 Running 0 30h
kube-system coredns-6d8c4cb4d-fspfl 1/1 Running 0 30h
kube-system coredns-6d8c4cb4d-jjwhx 1/1 Running 0 30h
kube-system csi-nfs-controller-5c5c695fb-678qj 4/4 Running 0 6d
kube-system csi-nfs-node-5jssl 3/3 Running 0 6d
kube-system csi-nfs-node-7rdh5 3/3 Running 0 24h
kube-system csi-nfs-node-bxsx9 3/3 Running 0 6d
kube-system etcd-master231 1/1 Running 2 (17d ago) 17d
kube-system kube-apiserver-master231 1/1 Running 0 9d
kube-system kube-controller-manager-master231 1/1 Running 4 (9d ago) 17d
kube-system kube-proxy-r9jxv 1/1 Running 0 11d
kube-system kube-proxy-ts66f 1/1 Running 0 11d
kube-system kube-proxy-xm74s 1/1 Running 0 24h
kube-system kube-scheduler-master231 1/1 Running 4 (9d ago) 17d
kube-system metrics-server-6b4f784878-cd585 1/1 Running 0 25h
kube-system metrics-server-6b4f784878-rsp6r 0/1 CrashLoopBackOff 686 (2m15s ago) 8d
kubernetes-dashboard dashboard-metrics-scraper-9d986c98c-nrkwx 1/1 Running 926 (26s ago) 8d
kubernetes-dashboard kubernetes-dashboard-5ccf77bb87-6z5dr 1/1 Running 0 30h
metallb-system controller-644c958987-mllg5 1/1 Running 0 30h
metallb-system speaker-8xpkb 1/1 Running 0 11d
metallb-system speaker-kxlx2 1/1 Running 0 11d
metallb-system speaker-pghbm 1/1 Running 0 24h
traefik traefik-server-6df7595d6f-ttslc 1/1 Running 0 30h
[root@master231 ~]#
[root@master231 ~]# etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
get /registry/pods/default/cj-etcdutl-29485981-h8dbb --prefix --print-value-only # 我们可以通过查看某个key的信息获取到该pod的详细信息。
1.停止k8s集群（跳过）
二进制管理起来很方便，但是kubeadm停止Kubelet进程即可。
值得注意的是，kubeadm的master节点的Kubelet暂时不要停止，因为etcd基于静态pod技术启动的。
2.备份旧etcd的数据
[root@master231 ~]# etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save /tmp/oldboyedu-etcd-`date +%F+%T`.backup
[root@master231 ~]# ll -h /tmp/oldboyedu-etcd-2026-01-23+17:16:43.backup
-rw------- 1 root root 18M Jan 23 17:16 /tmp/oldboyedu-etcd-2026-01-23+17:16:43.backup
[root@master231 ~]#
4.将备份数据拷贝到etcd集群节点
[root@master231 ~]# scp /tmp/oldboyedu-etcd-2026-01-23+17:16:43.backup 10.0.0.251:/tmp
[root@master231 ~]# scp /tmp/oldboyedu-etcd-2026-01-23+17:16:43.backup 10.0.0.252:/tmp
[root@master231 ~]# scp /tmp/oldboyedu-etcd-2026-01-23+17:16:43.backup 10.0.0.253:/tmp
5.停止etcd集群并清除原有数据
[root@k8s-cluster251 ~]# systemctl stop etcd
[root@k8s-cluster251 ~]# rm -rf /var/lib/etcd-2026/
[root@k8s-cluster251 ~]#
[root@k8s-cluster252 ~]# systemctl stop etcd
[root@k8s-cluster252 ~]# rm -rf /var/lib/etcd-2026/
[root@k8s-cluster252 ~]#
[root@k8s-cluster253 ~]# systemctl stop etcd
[root@k8s-cluster253 ~]# rm -rf /var/lib/etcd-2026/
[root@k8s-cluster253 ~]#
6.恢复数据
[root@k8s-cluster251 ~]# etcdutl snapshot restore /tmp/oldboyedu-etcd-2026-01-23+17\:16\:43.backup --data-dir=/var/lib/etcd-2026
[root@k8s-cluster252 ~]# etcdutl snapshot restore /tmp/oldboyedu-etcd-2026-01-23+17\:16\:43.backup --data-dir=/var/lib/etcd-2026
[root@k8s-cluster253 ~]# etcdutl snapshot restore /tmp/oldboyedu-etcd-2026-01-23+17\:16\:43.backup --data-dir=/var/lib/etcd-2026
7.启动etcd集群
[root@k8s-cluster251 ~]# systemctl start etcd
[root@k8s-cluster252 ~]# systemctl start etcd
[root@k8s-cluster253 ~]# systemctl start etcd
8.验证数据是否导入成功
[root@k8s-cluster251 ~]# etcdctl get "" --prefix --keys-only | wc -l
1902
[root@k8s-cluster251 ~]#
9.停止旧环境的etcd服务
[root@master231 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master231 Ready control-plane,master 17d v1.23.17
worker232 Ready <none> 17d v1.23.17
worker233 Ready <none> 25h v1.23.17
[root@master231 ~]#
[root@master231 ~]# mv /etc/kubernetes/manifests/etcd.yaml /opt/
[root@master231 ~]#
[root@master231 ~]# ss -ntl | grep 2379
[root@master231 ~]#
[root@master231 ~]# kubectl get nodes
Error from server (Timeout): the server was unable to return a response in the time allotted, but may still be processing the request (get nodes)
[root@master231 ~]#
10.拷贝新etcd集群的证书文件到本地
[root@master231 ~]# scp -r 10.0.0.251:/oldboyedu/certs/ /oldboyedu/
[root@master231 ~]#
[root@master231 ~]# ll /oldboyedu/certs/etcd/
total 44
drwxr-xr-x 2 root root 4096 Jan 23 17:29 ./
drwxr-xr-x 3 root root 4096 Jan 23 17:29 ../
-rw-r--r-- 1 root root 294 Jan 23 17:29 ca-config.json
-rw-r--r-- 1 root root 1050 Jan 23 17:29 etcd-ca.csr
-rw-r--r-- 1 root root 249 Jan 23 17:29 etcd-ca-csr.json
-rw------- 1 root root 1679 Jan 23 17:29 etcd-ca-key.pem
-rw-r--r-- 1 root root 1318 Jan 23 17:29 etcd-ca.pem
-rw-r--r-- 1 root root 210 Jan 23 17:29 etcd-csr.json
-rw-r--r-- 1 root root 1139 Jan 23 17:29 etcd-server.csr
-rw------- 1 root root 1675 Jan 23 17:29 etcd-server-key.pem
-rw-r--r-- 1 root root 1472 Jan 23 17:29 etcd-server.pem
[root@master231 ~]#
11.api-server指向新的etcd集群
[root@master231 ~]# vim /etc/kubernetes/manifests/kube-apiserver.yaml
...
13 containers:
14 - command:
15 - kube-apiserver
16 ...
24 #- --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
25 #- --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
26 #- --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
27 #- --etcd-servers=https://127.0.0.1:2379
28 - --etcd-cafile=/certs/etcd-ca.pem
29 - --etcd-certfile=/certs/etcd-server.pem
30 - --etcd-keyfile=/certs/etcd-server-key.pem
31 - --etcd-servers= https://10.0.0.251:2379,https://10.0.0.252:2379 ,https://10.0.0.253:2379
...
80 volumeMounts:
81 - mountPath: /certs
82 name: new-etcd-certs
83 readOnly: true
...
107 volumes:
108 - hostPath:
109 path: /oldboyedu/certs/etcd/
110 name: new-etcd-certs
12.重启Kubelet自动重启api-server等组件
[root@master231 ~]# systemctl restart kubelet
[root@master231 ~]#
[root@master231 ~]# ss -ntl| grep 6443
LISTEN 0 16384 *:6443 *:*
[root@master231 ~]#
[root@master231 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master231 Ready control-plane,master 17d v1.23.17
worker232 Ready <none> 17d v1.23.17
worker233 Ready <none> 25h v1.23.17
[root@master231 ~]#
[root@master231 ~]#
[root@master231 ~]# ss -ntl| grep 2379
[root@master231 ~]#
参考链接:
https://tzfun.github.io/etcd-workbench/
https://github.com/tzfun/etcd-workbench/blob/master/README_ZH.md
https://github.com/tzfun/etcd-workbench-web/blob/master/server/src/main/resources/etcd-workbench.conf
1.拉取镜像
docker pull tzfun/etcd-workbench:1.1.4
SVIP:
[root@harbor250.oldboyedu.com ~]# wget http://192.168.21.253/Resources/Prometheus/images/etcd-workbench/oldboyedu-etcd-workbench-v1.1.4.tar.gz
[root@harbor250.oldboyedu.com ~]# docker load -i oldboyedu-etcd-workbench-v1.1.4.tar.gz
2.准备配置文件
[root@harbor250.oldboyedu.com ~]# cat etcd-workbench.conf
[server]
# 服务监听的端口
port = 8002
# 链接超时时间
etcdExecuteTimeoutMillis = 3000
# 数据存储目录
dataDir = ./data
[auth]
# 启用认证功能
enable = true
# 指定用户名和密码
user = admin:yinzhengjie
[log]
# 指定日志的级别
level = INFO
# 日志存储目录
file = ./logs
# 日志文件的名称
fileName = etcd-workbench
# 指定日志的滚动大小
fileLimitSize = 100
# 日志打印的位置
printers = std,file
[root@harbor250.oldboyedu.com ~]#
3.运行etcd-workbench
[root@harbor250.oldboyedu.com ~]# docker run -d \
-v /root/etcd-workbench.conf:/usr/tzfun/etcd-workbench/etcd-workbench.conf \
--name etcd-workbench \
--network host \
tzfun/etcd-workbench:1.1.4
[root@harbor250.oldboyedu.com ~]# ss -ntl | grep 8002
LISTEN 0 4096 *:8002 *:*
[root@harbor250.oldboyedu.com ~]#
[root@harbor250.oldboyedu.com ~]# docker ps -l
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4327ecceeb18 tzfun/etcd-workbench:1.1.4 "/bin/sh -c 'java …" 8 seconds ago Up 7 seconds etcd-workbench
[root@harbor250.oldboyedu.com ~]#
1.访问etcd-workbench的webUI
http://10.0.0.250:8002/
用户名和密码使用你自定义的即可。
2.拷贝证书到windows系统
trusted-ca-file: '/oldboyedu/certs/etcd/etcd-ca.pem'
cert-file: '/oldboyedu/certs/etcd/etcd-server.pem'
key-file: '/oldboyedu/certs/etcd/etcd-server-key.pem'
3.上传证书并测试
略，见视频。
略。
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。