PostgreSQL 高可用集群 patroni 自动故障转移测试

博客园_首页

Plist 二进制格式 Milvus 和 PGVector，哪个更好？ OpenClaw 已过时？在 VS Code 中运行 Hermes Agent！第30篇文章：一个大三计科生的自白 Manim如何在数学公式中完美显示中文？ Docker 部署 RocketMQ 5 并发编程核心概念辨析 C#事务处理最佳实践：别再让“主表存了、明细丢了”的破事发生 CLI 是什么？为什么大厂突然集体卷命令行？【从0到1构建一个ClaudeAgent】协作-自主Agent UIImageView 设置图片不生效的原因排查最小二乘问题详解20：无先验约束下的增量式SFM自由网平差痞子衡嵌入式：大话双核i.MXRT1180之XIP应用里借助MU实现可靠Flash IAP的方法 AI Chat 封装， SemanticKerne.AiProvider.Unified 已发布 Windows下右键编辑js文件无法打开记事本——在注册表中使用环境变量在后台服务中使用 Scoped 服务，为什么总是报错？ H200 安装驱动并使用sglang启动模型 wireshark 抓包Trap上报告警内容我用 AI 辅助开发了一系列小工具（2）：图片压缩工具 [A Primer On MC and CC] 2.1 Memory Consistency 1 - 指令重排序和 SC 模型 Oracle数据库SCN推进技术详解与实践指南玩转控件:封装个带图片的Label控件 Claude Code 4.7 真正该升级的不是模型，而是你的工作流前端小白一句话，AI 帮我做了个颜值拉满的桌面媒体播放器。当代码不再是门槛，一句话编程就是现实。 5. WorkBuddy: 小龙虾的灵魂三件套，让你的小龙虾不只是工具 SQLite 分片方案实战：三种分片策略的深度对比告别简陋 UI！一款基于 Fluent Design 和基于 WinUI 的开源免费、现代化的 Avalonia UI 控件库关于二进制排列组合枚举的总结 AI开发-python-LangGraph框架（3-27-LangGraph从零实现大模型智能决策工作流） ElasticSearch主分片和副本分片概念详解【002】HTTPS 粗解：证书、TLS 握手与对后端配置的影响 Hermes Agent 一周暴涨五万 Star，但我劝你别急着追明明连接的是Redis的DB0，为什么能查到DB3的数据？【从0到1构建一个ClaudeAgent】协作-Agent团队熟悉电子元器件之后，电子小白下一步该怎么走？ MAF快速入门（23）通过C#类定义Skills .NET 高级开发 | 手写一个对象映射框架 FastAPI数据库ORM怎么选？我肝了三个Demo后，终于不再纠结了 mysqldump 参数拾遗：在遗忘与铭记之间 C# .NET 周刊｜2026年3月5期 Claude code入门 - 陈彦斌一文学习入门 ThingsBoard 开源物联网平台 GitHub 热门项目 | 2026年04月16日如何为GIT设置全局勾子，为每次提交追加信息 Number.isFinite和isFinite与isNaN()和Number.isNaN的区别 PortSwigger SQL注入LAB2 推荐一个测试人必备的Skills，从功能到性能全搞定（附详细实操和安装下载方式）筑基期：掌握Odoo基础核心知识点02（Odoo XML 开发方式详解） GLM模型这么火，咱们用vllm也咧一个呗！深入理解 AbortController：从底层原理到跨语言设计哲学字符串学习笔记多租户系统框架的基础模块设计和分析设计 Apache SeaTunnel Zeta 为什么能做到“又快又稳”？ AI开发-python-LangGraph框架（3-26-LangGraph基本概念及第一个简单样例） Vue 3 组件通信，别只会用 Props 和 Emits 了，这几个狠活儿你得看看 ElasticSearch7.X版本配置密码用Manim实现动态交点计算--从一个动点问题说起团结引擎+Addressable+Instant Game打包抖音小游戏 function call 实战：让 LLM 自动判断 pod 异常、调用日志工具并完成故障分析 bubseek —— 让 Agent 的足迹，变成团队的洞察通过 C# 读取并导出 PDF 书签如何用 GitHub Actions 实现 Steam 自动化发布【从0到1构建一个ClaudeAgent】并发-后台任务 .NET 高级开发 | 定制 ASP.NET Core 框架电子小白：什么是运算放大器（运放） zero2Agent：面向大厂面试的 Agent 工程教程，从概念到生产的完整学习路线堆上的ORW HC32F460 USB CDC通信异常：非对齐访问异常排查 20260413-Hyperbridge 攻击事件：发生在默克尔山上的验证绕过那些喊着AI 要淘汰你的人，正在靠你的焦虑赚大钱！深度学习进阶（八）Swin Transformer 最小二乘问题详解19：带先验约束的增量式SFM优化与实现 SnapTranslate 3.0 正式发布：全局划词翻译 + 完整英语学习闭环，一站式搞定查词、记词、复习工作的意义、工作的困难认知再思考 .NET + AI 进阶实战：基于类的技能开发 - 打造可治理的 Agent 能力模块【从0到1构建一个ClaudeAgent】规划与协调-技能上周热点回顾（4.6-4.12）电子小白的工具三件套：面包板、杜邦线、万能板单表五亿数据的查询优化 | Mysql、StarRocks 2. WorkBuddy：从“我是谁”到“帮我干活” C# 如何减少代码运行时间：7 个实战技巧基于HelixToolkit.SharpDX 渲染3D模型 - 笺上知微从零开始的双臂具身VLA起源及现阶段发展综述 - SkyXZ 记对 xonsh shell 的使用, 脚本编写, 迁移及调优 - pluvium27 受够了Vibe Coding的失控？换个起点，让AI事半功倍从开始配置漏洞环境到漏洞复现流程 - 難しい关于10年工作经验的程序员对OpenClaw的实战经验分享以及看法 - 虚无境 Any metadata 的内存布局 C# .NET 周刊｜2026年3月2期 - InCerry 我帮你测过了，测试圈排名第二的 Skill 依然很牛逼 Skill Discovery | 无监督技能发现的经典工作总结 - MoonOut 上下文工程是什么？过时了么？一文讲明白！ - 一枫说码开了 TUN 模式还是直连？90% 的人都踩过这个坑 AScript扩展多种脚本语言 - rockey627 AI 学习笔记：Agent 的记忆机制你能被装进一个文件里吗？——7 万人把同事"蒸馏"成了 AI - 我没有三颗心脏 Claude Code 通关手册（七）：给 AI 装上技能包——Skills 完全指南 - 暮色之狐在浏览器中快速编辑代码：VSCode Web 集成实践 - Newbe36524 蒸馏自己 skill？基于 Deepseek 的蒸馏器，丐版蒸馏方式，简单便捷 - To_Carpe_Diem Spring AI Aliababa和AgentScope，哪个更好？ - 苏三说技术

MSSQL123 · 2026-05-22 · via 博客园_首页

网上有很多关于patroni的文章很多，绝大多数文章是通过手动搭建的方式，仅做出了一个patroni的环境搭建，包括各种微信群等，对于patroni参数的使用，故障转移的原理以及实操都只字未提，本文通过Ubuntu 20 环境下 patroni 自动化安装，一分钟快速搭建 patroni 集群来快速搭建一个集群，以及实操的方式实现故障转移的测试和验证，通过模拟真实的故障以及故障转移的日志，来分析故障转移的实现和效果。

2026-05-21 08:46:37,142 INFO: no action. I am (ubuntu11), the leader with the lock
2026-05-21 08:46:47,145 INFO: no action. I am (ubuntu11), the leader with the lock
2026-05-21 08:46:57,190 INFO: no action. I am (ubuntu11), the leader with the lock
2026-05-21 08:47:07,148 INFO: no action. I am (ubuntu11), the leader with the lock
2026-05-21 08:47:17,145 INFO: no action. I am (ubuntu11), the leader with the lock
2026-05-21 08:47:27,189 INFO: no action. I am (ubuntu11), the leader with the lock
2026-05-21 08:47:37,153 INFO: no action. I am (ubuntu11), the leader with the lock

Ubuntu 12 从节点日志

2026-05-21 08:46:47,628 INFO: no action. I am (ubuntu12), a secondary, and following a leader (ubuntu11)
2026-05-21 08:46:57,628 INFO: no action. I am (ubuntu12), a secondary, and following a leader (ubuntu11)
2026-05-21 08:47:07,671 INFO: no action. I am (ubuntu12), a secondary, and following a leader (ubuntu11)
2026-05-21 08:47:17,632 INFO: no action. I am (ubuntu12), a secondary, and following a leader (ubuntu11)
2026-05-21 08:47:27,675 INFO: no action. I am (ubuntu12), a secondary, and following a leader (ubuntu11)
2026-05-21 08:47:37,139 INFO: no action. I am (ubuntu12), a secondary, and following a leader (ubuntu11)
2026-05-21 08:47:47,233 INFO: no action. I am (ubuntu12), a secondary, and following a leader (ubuntu11)
2026-05-21 08:47:57,681 INFO: no action. I am (ubuntu12), a secondary, and following a leader (ubuntu11)

Ubuntu 13 从节点日志

2026-05-21 08:46:57,643 INFO: no action. I am (ubuntu13), a secondary, and following a leader (ubuntu11)
2026-05-21 08:47:07,696 INFO: no action. I am (ubuntu13), a secondary, and following a leader (ubuntu11)
2026-05-21 08:47:17,647 INFO: no action. I am (ubuntu13), a secondary, and following a leader (ubuntu11)
2026-05-21 08:47:27,688 INFO: no action. I am (ubuntu13), a secondary, and following a leader (ubuntu11)
2026-05-21 08:47:37,155 INFO: no action. I am (ubuntu13), a secondary, and following a leader (ubuntu11)
2026-05-21 08:47:47,255 INFO: no action. I am (ubuntu13), a secondary, and following a leader (ubuntu11)
2026-05-21 08:47:57,696 INFO: no action. I am (ubuntu13), a secondary, and following a leader (ubuntu11)

1，自动故障转移场景1：主节点OS正常，patroni服务异常故障

主节点状态正常，关闭主节点patroni服务模拟主节点故障

root@ubuntu11:/usr/local/patroni_install# patronictl -c /usr/local/pgsql17/patroni/patroni.yml list
+ Cluster: pg_cluster_wy_prod (7641831362696373502) ---------+----+-------------+-----+------------+-----+
| Member   | Host                 | Role         | State     | TL | Receive LSN | Lag | Replay LSN | Lag |
+----------+----------------------+--------------+-----------+----+-------------+-----+------------+-----+
| ubuntu11 | 192.168.152.121:9000 | Leader       | running   |  6 |             |     |            |     |
| ubuntu12 | 192.168.152.122:9000 | Sync Standby | streaming |  6 |   0/F000348 |   0 |  0/F000348 |   0 |
| ubuntu13 | 192.168.152.123:9000 | Replica      | streaming |  6 |   0/F000348 |   0 |  0/F000348 |   0 |
+----------+----------------------+--------------+-----------+----+-------------+-----+------------+-----+
root@ubuntu11:/usr/local/patroni_install# systemctl stop patroni
root@ubuntu11:/usr/local/patroni_install#

从节点Ubuntu12上观察到的集群状态，此时原始主节点已处于停止状态

root@ubuntu12:/usr/local/patroni_install# patronictl -c /usr/local/pgsql17/patroni/patroni.yml list
+ Cluster: pg_cluster_wy_prod (7641831362696373502) ---------+----+-------------+-----+------------+-----+
| Member   | Host                 | Role         | State     | TL | Receive LSN | Lag | Replay LSN | Lag |
+----------+----------------------+--------------+-----------+----+-------------+-----+------------+-----+
| ubuntu11 | 192.168.152.121:9000 | Replica      | stopped   |    |     unknown |     |    unknown |     |
| ubuntu12 | 192.168.152.122:9000 | Leader       | running   |  7 |             |     |            |     |
| ubuntu13 | 192.168.152.123:9000 | Sync Standby | streaming |  7 |  0/100001A8 |   0 | 0/100001A8 |   0 |
+----------+----------------------+--------------+-----------+----+-------------+-----+------------+-----+

原始从节点Ubuntu12，成为新的主节点，日志如下

......
2026-05-21 08:55:27,680 INFO: no action. I am (ubuntu12), a secondary, and following a leader (ubuntu11)
2026-05-21 08:55:37,723 INFO: no action. I am (ubuntu12), a secondary, and following a leader (ubuntu11)
2026-05-21 08:55:47,683 INFO: no action. I am (ubuntu12), a secondary, and following a leader (ubuntu11)
2026-05-21 08:55:57,681 INFO: no action. I am (ubuntu12), a secondary, and following a leader (ubuntu11)
2026-05-21 08:56:06,109 WARNING: Request failed to ubuntu11: GET http://192.168.152.121:8008/patroni (HTTPConnectionPool(host='192.168.152.121', port=8008): Max retries exceeded with url: /patroni (Caused by ProtocolError('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))))
2026-05-21 08:56:06,169 INFO: promoted self to leader by acquiring session lock
2026-05-21 08:56:06,169 INFO: Lock owner: ubuntu12; I am ubuntu12
2026-05-21 08:56:06,172 INFO: updated leader lock during promote
server promoting
2026-05-21 08:56:07,185 INFO: Lock owner: ubuntu12; I am ubuntu12
2026-05-21 08:56:07,195 INFO: Assigning synchronous standby status to ['ubuntu13']
server signaled
2026-05-21 08:56:09,324 INFO: Synchronous standby status assigned to ['ubuntu13']
2026-05-21 08:56:09,369 INFO: no action. I am (ubuntu12), the leader with the lock
2026-05-21 08:56:17,196 INFO: no action. I am (ubuntu12), the leader with the lock
2026-05-21 08:56:27,187 INFO: no action. I am (ubuntu12), the leader with the lock
2026-05-21 08:56:37,242 INFO: no action. I am (ubuntu12), the leader with the lock
......

这种场景下的故障转移的流程：
手动关闭Ubuntu11 Patroni 主节点模拟故障 ———>Ubuntu 11上的patroni主动删除 DCS 中的 leader key———> Ubuntu12 从节点经过loop_wait轮训后检测到DSC无主 ———> 获取锁提升为 Leader———> promote 本地PostgreSQL为主库

2，自动故障转移场景2：主节点服务器断电

Ubuntu11 通过“关机”（而非关闭客户机）来模拟服务器突然断电，这种场景需要深刻理解租约寿命，也就是ttl（默认 30 秒）参数的概念

新的主节点Ubuntu 12上看到的集群状态

root@ubuntu13:/usr/local/patroni_install# patronictl -c /usr/local/pgsql17/patroni/patroni.yml list
+ Cluster: pg_cluster_wy_prod (7642212398676862997) ----+----+-------------+-----+------------+-----+------------------------+
| Member   | Host                 | Role    | State     | TL | Receive LSN | Lag | Replay LSN | Lag | Tags                   |
+----------+----------------------+---------+-----------+----+-------------+-----+------------+-----+------------------------+
| ubuntu11 | 192.168.152.121:9000 | Leader  | running   | 10 |             |     |            |     | failover_priority: 100 |
| ubuntu12 | 192.168.152.122:9000 | Replica | streaming | 10 |   0/C000000 |   0 |  0/C000358 |   0 | failover_priority: 80  |
| ubuntu13 | 192.168.152.123:9000 | Replica | streaming | 10 |   0/C000380 |   0 |  0/C000380 |   0 | failover_priority: 60  |
+----------+----------------------+---------+-----------+----+-------------+-----+------------+-----+------------------------+
root@ubuntu13:/usr/local/patroni_install#
root@ubuntu13:/usr/local/patroni_install# patronictl -c /usr/local/pgsql17/patroni/patroni.yml list
+ Cluster: pg_cluster_wy_prod (7642212398676862997) ---------+----+-------------+-----+------------+-----+-----------------------+
| Member   | Host                 | Role         | State     | TL | Receive LSN | Lag | Replay LSN | Lag | Tags                  |
+----------+----------------------+--------------+-----------+----+-------------+-----+------------+-----+-----------------------+
| ubuntu12 | 192.168.152.122:9000 | Leader       | running   | 11 |             |     |            |     | failover_priority: 80 |
| ubuntu13 | 192.168.152.123:9000 | Sync Standby | streaming | 11 |   0/C000688 |   0 |  0/C000688 |   0 | failover_priority: 60 |
+----------+----------------------+--------------+-----------+----+-------------+-----+------------+-----+-----------------------+
root@ubuntu13:/usr/local/patroni_install#

新的主节点Ubuntu12上patroni的日志

2026-05-22 13:53:59,956 INFO: no action. I am (ubuntu12), a secondary, and following a leader (ubuntu11)
2026-05-22 13:54:10,451 INFO: no action. I am (ubuntu12), a secondary, and following a leader (ubuntu11)
2026-05-22 13:54:20,026 INFO: no action. I am (ubuntu12), a secondary, and following a leader (ubuntu11)
2026-05-22 13:54:30,458 INFO: no action. I am (ubuntu12), a secondary, and following a leader (ubuntu11)
2026-05-22 13:54:40,461 INFO: no action. I am (ubuntu12), a secondary, and following a leader (ubuntu11)
2026-05-22 13:54:50,499 INFO: no action. I am (ubuntu12), a secondary, and following a leader (ubuntu11)
2026-05-22 13:55:00,456 INFO: no action. I am (ubuntu12), a secondary, and following a leader (ubuntu11)
2026-05-22 13:55:10,457 INFO: no action. I am (ubuntu12), a secondary, and following a leader (ubuntu11)######差不多这个点开始对Ubuntu11掉电
2026-05-22 13:55:20,498 INFO: no action. I am (ubuntu12), a secondary, and following a leader (ubuntu11)######为什么这个点，检测到的Ubuntu11还是正常状态？因为Ubuntu11的lease也就是租约还没有过期
2026-05-22 13:55:32,106 WARNING: Request failed to ubuntu11: GET http://192.168.152.121:8008/patroni (HTTPConnectionPool(host='192.168.152.121', port=8008): Max retries exceeded with url: /patroni (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f893566d880>, 'Connection to 192.168.152.121 timed out. (connect timeout=2)')))
2026-05-22 13:55:32,114 INFO: promoted self to leader by acquiring session lock
2026-05-22 13:55:32,114 INFO: Lock owner: ubuntu12; I am ubuntu12
2026-05-22 13:55:32,115 INFO: updated leader lock during promote
2026-05-22 13:55:33,137 INFO: Lock owner: ubuntu12; I am ubuntu12
2026-05-22 13:55:33,193 INFO: Assigning synchronous standby status to ['ubuntu13']
2026-05-22 13:55:35,316 INFO: Synchronous standby status assigned to ['ubuntu13']
2026-05-22 13:55:35,322 INFO: no action. I am (ubuntu12), the leader with the lock
2026-05-22 13:55:35,377 INFO: no action. I am (ubuntu12), the leader with the lock
2026-05-22 13:55:45,324 INFO: no action. I am (ubuntu12), the leader with the lock
2026-05-22 13:55:55,367 INFO: no action. I am (ubuntu12), the leader with the lock
2026-05-22 13:56:05,329 INFO: no action. I am (ubuntu12), the leader with the lock

新的主节点通过psql查看身份状态

postgres=#
postgres=#
postgres=# select now(),pg_is_in_recovery();            ###########################这里开始对原始主节点Ubuntu11 掉电，然后连续查询
              now              | pg_is_in_recovery
-------------------------------+-------------------
 2026-05-22 13:55:10.849473+08 | t
(1 row)

postgres=# select now(),pg_is_in_recovery();
              now              | pg_is_in_recovery
-------------------------------+-------------------
 2026-05-22 13:55:11.665724+08 | t
(1 row)

postgres=# select now(),pg_is_in_recovery();
             now              | pg_is_in_recovery
------------------------------+-------------------
 2026-05-22 13:55:12.32947+08 | t
(1 row)

postgres=# select now(),pg_is_in_recovery();
              now              | pg_is_in_recovery
-------------------------------+-------------------
 2026-05-22 13:55:13.017149+08 | t
(1 row)

postgres=# select now(),pg_is_in_recovery();
              now              | pg_is_in_recovery
-------------------------------+-------------------
 2026-05-22 13:55:13.799962+08 | t
(1 row)

postgres=# select now(),pg_is_in_recovery();
              now              | pg_is_in_recovery
-------------------------------+-------------------
 2026-05-22 13:55:14.902866+08 | t
(1 row)

postgres=# select now(),pg_is_in_recovery();
              now              | pg_is_in_recovery
-------------------------------+-------------------
 2026-05-22 13:55:15.672331+08 | t
(1 row)

postgres=# select now(),pg_is_in_recovery();
              now              | pg_is_in_recovery
-------------------------------+-------------------
 2026-05-22 13:55:16.435662+08 | t
(1 row)

postgres=# select now(),pg_is_in_recovery();
              now              | pg_is_in_recovery
-------------------------------+-------------------
 2026-05-22 13:55:17.070935+08 | t
(1 row)

postgres=# select now(),pg_is_in_recovery();
              now              | pg_is_in_recovery
-------------------------------+-------------------
 2026-05-22 13:55:17.816528+08 | t
(1 row)

postgres=# select now(),pg_is_in_recovery();
              now              | pg_is_in_recovery
-------------------------------+-------------------
 2026-05-22 13:55:18.546785+08 | t
(1 row)

postgres=# select now(),pg_is_in_recovery();
              now              | pg_is_in_recovery
-------------------------------+-------------------
 2026-05-22 13:55:19.393943+08 | t
(1 row)

#......中间省略掉......

postgres=# select now(),pg_is_in_recovery();
              now              | pg_is_in_recovery
-------------------------------+-------------------
 2026-05-22 13:55:29.759037+08 | t
(1 row)

postgres=# select now(),pg_is_in_recovery();
              now              | pg_is_in_recovery
-------------------------------+-------------------
 2026-05-22 13:55:30.417626+08 | t
(1 row)

postgres=# select now(),pg_is_in_recovery();
              now              | pg_is_in_recovery
-------------------------------+-------------------
 2026-05-22 13:55:31.089604+08 | t
(1 row)

postgres=# select now(),pg_is_in_recovery();
              now              | pg_is_in_recovery
-------------------------------+-------------------
 2026-05-22 13:55:31.775459+08 | t
(1 row)

postgres=# select now(),pg_is_in_recovery();            ###########################22秒之后，新的主节点才真正promote起来
              now              | pg_is_in_recovery
-------------------------------+-------------------
 2026-05-22 13:55:32.400935+08 | f
(1 row)

postgres=# select now(),pg_is_in_recovery();
             now              | pg_is_in_recovery
------------------------------+-------------------
 2026-05-22 13:55:33.27183+08 | f
(1 row)

postgres=# select now(),pg_is_in_recovery();
              now              | pg_is_in_recovery
-------------------------------+-------------------
 2026-05-22 13:55:33.950342+08 | f
(1 row)

postgres=# select now(),pg_is_in_recovery();
              now              | pg_is_in_recovery
-------------------------------+-------------------
 2026-05-22 13:55:34.758651+08 | f
(1 row)

postgres=# postgres=#
postgres-# postgres=#

结合上述日志，来理解ttl的概念，从时间的维度来观察：

1，2026-05-22 13:55:10,457，上面提到差不多在这个是时间点开始对原主节点Ubuntu11断电，
2，2026-05-22 13:55:20,498 ，patroni日志中检测到的Ubuntu11还是正常状态？
3， 2026-05-22 13:55:32.400935，通过查询新的主节点的pg_is_in_recovery，发现pg_is_in_recovery才变为f，也即故障转移成功

日志是否与实际操作的不符合，明明Ubuntu11在13:55:10就断电了，为什么13:55:20还在检测的时候还是正常的，但是直到13:55:32，新的主节点才真正开始工作，这是不是矛盾的？

这是因为，在13:55:10断电，在13:55:10前几秒（减去一个loop_wait的时间点，loop_wait默认10秒）， Ubuntu11上的patroni对etcd中的leader key续约，续约一次生效时间为向后推30秒，lease也就是租约还没有过期，其租约大概在13:55:30之后才过期，因此在13:55:20这个时间点，接替它的从节点上的patroni服务，检测到leader key 并没有过期。
直到下一个检测周期，也即13:55:30的时候，这一轮检查的时候才发现“2026-05-22 13:55:32,106 WARNING: Request failed to ubuntu11: GET http://192.168.152.121:8008/patroni (HTTPConnectionPool(host='192.168.152.121', port=8008): Max retries exceeded with url: /patroni (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f893566d880>, 'Connection to 192.168.152.121 timed out. (connect timeout=2)')))”原始主节点异常，为什么日志是13:55:32，在13:55:30的基础上加了2秒呢？因为connect timeout=2

以上才是patroni参数中ttl的真正含义。

这种场景下的故障转移的流程：
关闭Ubuntu11 电源模拟主节点故障 ———>10秒后 Ubuntu 11上的leader 租约扔有效（实际上此时Ubuntu已宕机） ———>10秒后 Ubuntu 11上的leader 租约扔有效（实际上此时Ubuntu已宕机） ———>10秒后 Ubuntu 12检测到leader 失效———> 抢占leader key，promote 本地PostgreSQL为主库

因此如果想提到patroni的故障转移的灵敏性，需要减小ttl的值，也即减少leader key的租约时间，同时也要减小loop_wait，增加判断leader key的频率，来提升故障检测以及转移的灵敏性，但也要意识到，调小这两个参数，可能在网络抖动是会带来的预期之外的故障转移。

3，自动故障转移场景3：主节点网络分区

用iptables -A OUTPUT -d 192.168.152.121 -j DROP

从节点1

root@ubuntu12:/usr/local/patroni_install# sudo iptables -A OUTPUT -d 192.168.152.121 -j DROP
root@ubuntu12:/usr/local/patroni_install# sudo iptables -A INPUT  -s 192.168.152.121 -j DROP
root@ubuntu12:/usr/local/patroni_install#

从节点2

root@ubuntu13:/usr/local/patroni_install# sudo iptables -A OUTPUT -d 192.168.152.121 -j DROP
root@ubuntu13:/usr/local/patroni_install# sudo iptables -A INPUT  -s 192.168.152.121 -j DROP
root@ubuntu13:/usr/local/patroni_install#

网络分区已形成

1，对于新的主节点：Ubuntu12已经成功接管主节点

2026-05-22 14:47:38,980 INFO: no action. I am (ubuntu12), a secondary, and following a leader (ubuntu11)
2026-05-22 14:47:49,402 INFO: no action. I am (ubuntu12), a secondary, and following a leader (ubuntu11)
2026-05-22 14:47:58,941 INFO: no action. I am (ubuntu12), a secondary, and following a leader (ubuntu11)
2026-05-22 14:48:09,491 INFO: no action. I am (ubuntu12), a secondary, and following a leader (ubuntu11)
2026-05-22 14:48:19,441 INFO: no action. I am (ubuntu12), a secondary, and following a leader (ubuntu11)
2026-05-22 14:48:31,104 WARNING: Request failed to ubuntu11: GET http://192.168.152.121:8008/patroni (HTTPConnectionPool(host='192.168.152.121', port=8008): Max retries exceeded with url: /patroni (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f8988fdb1f0>, 'Connection to 192.168.152.121 timed out. (connect timeout=2)')))
2026-05-22 14:48:31,183 INFO: promoted self to leader by acquiring session lock
2026-05-22 14:48:31,187 INFO: Lock owner: ubuntu12; I am ubuntu12
2026-05-22 14:48:31,239 INFO: updated leader lock during promote
2026-05-22 14:48:32,206 INFO: Lock owner: ubuntu12; I am ubuntu12
2026-05-22 14:48:32,214 INFO: Assigning synchronous standby status to ['ubuntu13']
2026-05-22 14:48:34,337 INFO: Synchronous standby status assigned to ['ubuntu13']
2026-05-22 14:48:34,385 INFO: no action. I am (ubuntu12), the leader with the lock
2026-05-22 14:48:42,245 INFO: no action. I am (ubuntu12), the leader with the lock
2026-05-22 14:48:52,256 INFO: no action. I am (ubuntu12), the leader with the lock

需要说明的是，对网络分区的故障转移，与上面主节点断电一样，虽然在新主节点的日志中，从发现到故障转移只用了10秒多，但实际上，在网络分区之后，因为原主节点对于leader key的最后一次续约加上了30秒（ttl），导致网络分区发生后，新的主节点在探测ttl的时候，前2次探测的时候实际上网络分区已经形成，但此时新的主节点尚未接管，直至原主节点的leader key 租约超期，这一点与上面一种情况一样，详细测试过不在赘述。

2，对于原主节点：
此时原主节点日志已无法连接至Ubuntu12 和Ubuntu 13，注意日志
2026-05-22 14:48:17,257 ERROR: Error communicating with DCS
2026-05-22 14:48:17,258 INFO: demoting self because DCS is not accessible and I was a leader
2026-05-22 14:48:17,258 INFO: Demoting self (offline)
原始主节点网络分区之后，自动降级为只读状态，因此不会出现双主或者脑裂的现象。同时会连续不停地尝试连接到Ubuntu12和ubuntu13上的etcd集群（日志在持续生成，没有贴全），以确保网络恢复后自动加入集群

2026-05-22 14:47:38,896 INFO: no action. I am (ubuntu11), the leader with the lock
2026-05-22 14:47:38,963 INFO: no action. I am (ubuntu11), the leader with the lock
2026-05-22 14:47:48,912 INFO: no action. I am (ubuntu11), the leader with the lock
2026-05-22 14:47:58,948 INFO: no action. I am (ubuntu11), the leader with the lock
2026-05-22 14:48:08,903 INFO: Lock owner: ubuntu11; I am ubuntu11
2026-05-22 14:48:12,244 ERROR: Request to server http://192.168.152.121:2379 failed: ReadTimeoutError("HTTPConnectionPool(host='192.168.152.121', port=2379): Read timed out. (read timeout=3.3332171243333355)")
2026-05-22 14:48:12,244 INFO: Reconnection allowed, looking for another server.
2026-05-22 14:48:12,244 INFO: Retrying on http://192.168.152.123:2379
2026-05-22 14:48:13,913 ERROR: Request to server http://192.168.152.123:2379 failed: MaxRetryError("HTTPConnectionPool(host='192.168.152.123', port=2379): Max retries exceeded with url: /v3/lease/keepalive (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f205c04b2b0>, 'Connection to 192.168.152.123 timed out. (connect timeout=1.6666666666666667)'))")
2026-05-22 14:48:13,913 INFO: Reconnection allowed, looking for another server.
2026-05-22 14:48:13,913 INFO: Retrying on http://192.168.152.122:2379
2026-05-22 14:48:15,583 ERROR: Request to server http://192.168.152.122:2379 failed: MaxRetryError("HTTPConnectionPool(host='192.168.152.122', port=2379): Max retries exceeded with url: /v3/lease/keepalive (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f205c04b2e0>, 'Connection to 192.168.152.122 timed out. (connect timeout=1.6666666666666667)'))")
2026-05-22 14:48:15,583 INFO: Reconnection allowed, looking for another server.
2026-05-22 14:48:17,253 ERROR: Failed to get list of machines from http://192.168.152.122:2379/v3: MaxRetryError("HTTPConnectionPool(host='192.168.152.122', port=2379): Max retries exceeded with url: /v3/cluster/member/list (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f205c04b520>, 'Connection to 192.168.152.122 timed out. (connect timeout=1.6666666666666667)'))")
2026-05-22 14:48:17,256 ERROR: watchprefix failed: ProtocolError('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
2026-05-22 14:48:17,257 ERROR: Error communicating with DCS
2026-05-22 14:48:17,258 INFO: demoting self because DCS is not accessible and I was a leader
2026-05-22 14:48:17,258 INFO: Demoting self (offline)
2026-05-22 14:48:18,932 ERROR: Failed to get list of machines from http://192.168.152.122:2379/v3: MaxRetryError("HTTPConnectionPool(host='192.168.152.122', port=2379): Max retries exceeded with url: /v3/cluster/member/list (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f205c04b970>, 'Connection to 192.168.152.122 timed out. (connect timeout=1.6666666666666667)'))")
2026-05-22 14:49:00,355 INFO: postmaster pid=3525
2026-05-22 14:49:01,400 INFO: demoted self because DCS is not accessible and I was a leader
2026-05-22 14:49:01,403 WARNING: Loop time exceeded, rescheduling immediately.
2026-05-22 14:49:01,405 INFO: Lock owner: ubuntu11; I am ubuntu11
2026-05-22 14:49:01,405 INFO: establishing a new patroni heartbeat connection to postgres
2026-05-22 14:49:04,749 ERROR: Request to server http://192.168.152.121:2379 failed: ReadTimeoutError("HTTPConnectionPool(host='192.168.152.121', port=2379): Read timed out. (read timeout=3.33254870033331)")
2026-05-22 14:49:04,749 INFO: Reconnection allowed, looking for another server.
2026-05-22 14:49:04,749 INFO: Retrying on http://192.168.152.123:2379
2026-05-22 14:49:06,419 ERROR: Request to server http://192.168.152.123:2379 failed: MaxRetryError("HTTPConnectionPool(host='192.168.152.123', port=2379): Max retries exceeded with url: /v3/lease/keepalive (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f205c04bfa0>, 'Connection to 192.168.152.123 timed out. (connect timeout=1.6666666666666667)'))")
2026-05-22 14:49:06,419 INFO: Reconnection allowed, looking for another server.
2026-05-22 14:49:06,419 INFO: Retrying on http://192.168.152.122:2379
2026-05-22 14:49:08,089 ERROR: Request to server http://192.168.152.122:2379 failed: MaxRetryError("HTTPConnectionPool(host='192.168.152.122', port=2379): Max retries exceeded with url: /v3/lease/keepalive (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f2027e451c0>, 'Connection to 192.168.152.122 timed out. (connect timeout=1.6666666666666667)'))")
2026-05-22 14:49:08,089 INFO: Reconnection allowed, looking for another server.
2026-05-22 14:49:09,758 ERROR: Failed to get list of machines from http://192.168.152.122:2379/v3: MaxRetryError("HTTPConnectionPool(host='192.168.152.122', port=2379): Max retries exceeded with url: /v3/cluster/member/list (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f205c53fa90>, 'Connection to 192.168.152.122 timed out. (connect timeout=1.6666666666666667)'))")
2026-05-22 14:49:11,417 ERROR: Request to server http://192.168.152.121:2379 failed: ReadTimeoutError("HTTPConnectionPool(host='192.168.152.121', port=2379): Read timed out. (read timeout=1.00350682653891)")
2026-05-22 14:49:11,417 INFO: Reconnection allowed, looking for another server.
2026-05-22 14:49:13,086 ERROR: Failed to get list of machines from http://192.168.152.122:2379/v3: MaxRetryError("HTTPConnectionPool(host='192.168.152.122', port=2379): Max retries exceeded with url: /v3/cluster/member/list (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f205c04ba60>, 'Connection to 192.168.152.122 timed out. (connect timeout=1.6666666666666667)'))")
2026-05-22 14:49:13,088 ERROR: Error communicating with DCS
2026-05-22 14:49:13,088 INFO: DCS is not accessible
2026-05-22 14:49:13,088 ERROR: watchprefix failed: ProtocolError('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
2026-05-22 14:49:13,090 WARNING: Loop time exceeded, rescheduling immediately.
2026-05-22 14:49:14,757 ERROR: Failed to get list of machines from http://192.168.152.122:2379/v3: MaxRetryError("HTTPConnectionPool(host='192.168.152.122', port=2379): Max retries exceeded with url: /v3/cluster/member/list (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f205c04b820>, 'Connection to 192.168.152.122 timed out. (connect timeout=1.6666666666666667)'))")
2026-05-22 14:49:14,763 INFO: Lock owner: ubuntu11; I am ubuntu11
2026-05-22 14:49:18,103 ERROR: Request to server http://192.168.152.121:2379 failed: ReadTimeoutError("HTTPConnectionPool(host='192.168.152.121', port=2379): Read timed out. (read timeout=3.3331819403333234)")
2026-05-22 14:49:18,103 INFO: Reconnection allowed, looking for another server.
2026-05-22 14:49:18,103 INFO: Retrying on http://192.168.152.122:2379
2026-05-22 14:49:19,773 ERROR: Request to server http://192.168.152.122:2379 failed: MaxRetryError("HTTPConnectionPool(host='192.168.152.122', port=2379): Max retries exceeded with url: /v3/lease/keepalive (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f2027e450d0>, 'Connection to 192.168.152.122 timed out. (connect timeout=1.6666666666666667)'))")
2026-05-22 14:49:19,773 INFO: Reconnection allowed, looking for another server.
2026-05-22 14:49:19,773 INFO: Retrying on http://192.168.152.123:2379
2026-05-22 14:49:21,441 ERROR: Request to server http://192.168.152.123:2379 failed: MaxRetryError("HTTPConnectionPool(host='192.168.152.123', port=2379): Max retries exceeded with url: /v3/lease/keepalive (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f2027e45370>, 'Connection to 192.168.152.123 timed out. (connect timeout=1.6666666666666667)'))")
2026-05-22 14:49:21,442 INFO: Reconnection allowed, looking for another server.
2026-05-22 14:49:23,112 ERROR: Failed to get list of machines from http://192.168.152.122:2379/v3: MaxRetryError("HTTPConnectionPool(host='192.168.152.122', port=2379): Max retries exceeded with url: /v3/cluster/member/list (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f2027e45670>, 'Connection to 192.168.152.122 timed out. (connect timeout=1.6666666666666667)'))")
2026-05-22 14:49:23,114 ERROR: Error communicating with DCS
2026-05-22 14:49:23,114 INFO: DCS is not accessible
2026-05-22 14:49:23,114 ERROR: watchprefix failed: ProtocolError('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
2026-05-22 14:49:23,115 WARNING: Loop time exceeded, rescheduling immediately.
2026-05-22 14:49:24,784 ERROR: Failed to get list of machines from http://192.168.152.122:2379/v3: MaxRetryError("HTTPConnectionPool(host='192.168.152.122', port=2379): Max retries exceeded with url: /v3/cluster/member/list (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f2027e45b50>, 'Connection to 192.168.152.122 timed out. (connect timeout=1.6666666666666667)'))")
2026-05-22 14:49:24,790 INFO: Lock owner: ubuntu11; I am ubuntu11
2026-05-22 14:49:28,128 ERROR: Request to server http://192.168.152.121:2379 failed: ReadTimeoutError("HTTPConnectionPool(host='192.168.152.121', port=2379): Read timed out. (read timeout=3.332673626333379)")
2026-05-22 14:49:28,128 INFO: Reconnection allowed, looking for another server.
2026-05-22 14:49:28,128 INFO: Retrying on http://192.168.152.123:2379
2026-05-22 14:49:29,799 ERROR: Request to server http://192.168.152.123:2379 failed: MaxRetryError("HTTPConnectionPool(host='192.168.152.123', port=2379): Max retries exceeded with url: /v3/lease/keepalive (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f2027e5c220>, 'Connection to 192.168.152.123 timed out. (connect timeout=1.6666666666666667)'))")
2026-05-22 14:49:29,799 INFO: Reconnection allowed, looking for another server.
2026-05-22 14:49:29,799 INFO: Retrying on http://192.168.152.122:2379
2026-05-22 14:49:31,469 ERROR: Request to server http://192.168.152.122:2379 failed: MaxRetryError("HTTPConnectionPool(host='192.168.152.122', port=2379): Max retries exceeded with url: /v3/lease/keepalive (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f205c41f460>, 'Connection to 192.168.152.122 timed out. (connect timeout=1.6666666666666667)'))")
2026-05-22 14:49:31,469 INFO: Reconnection allowed, looking for another server.
2026-05-22 14:49:33,138 ERROR: Failed to get list of machines from http://192.168.152.122:2379/v3: MaxRetryError("HTTPConnectionPool(host='192.168.152.122', port=2379): Max retries exceeded with url: /v3/cluster/member/list (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f205c53f9a0>, 'Connection to 192.168.152.122 timed out. (connect timeout=1.6666666666666667)'))")
2026-05-22 14:49:34,794 ERROR: Request to server http://192.168.152.121:2379 failed: ReadTimeoutError("HTTPConnectionPool(host='192.168.152.121', port=2379): Read timed out. (read timeout=1.1904628130222932)")
2026-05-22 14:49:34,794 INFO: Reconnection allowed, looking for another server.
2026-05-22 14:49:36,464 ERROR: Failed to get list of machines from http://192.168.152.122:2379/v3: MaxRetryError("HTTPConnectionPool(host='192.168.152.122', port=2379): Max retries exceeded with url: /v3/cluster/member/list (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f205c04b0a0>, 'Connection to 192.168.152.122 timed out. (connect timeout=1.6666666666666667)'))")
2026-05-22 14:49:36,468 ERROR: watchprefix failed: ProtocolError('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
2026-05-22 14:49:36,468 ERROR: Error communicating with DCS
2026-05-22 14:49:36,468 INFO: DCS is not accessible
2026-05-22 14:49:36,470 WARNING: Loop time exceeded, rescheduling immediately.
2026-05-22 14:49:38,140 ERROR: Failed to get list of machines from http://192.168.152.122:2379/v3: MaxRetryError("HTTPConnectionPool(host='192.168.152.122', port=2379): Max retries exceeded with url: /v3/cluster/member/list (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f205c04b8b0>, 'Connection to 192.168.152.122 timed out. (connect timeout=1.6666666666666667)'))")
2026-05-22 14:49:38,145 INFO: Lock owner: ubuntu11; I am ubuntu11
2026-05-22 14:49:41,485 ERROR: Request to server http://192.168.152.121:2379 failed: ReadTimeoutError("HTTPConnectionPool(host='192.168.152.121', port=2379): Read timed out. (read timeout=3.33319548833335)")
2026-05-22 14:49:41,485 INFO: Reconnection allowed, looking for another server.
2026-05-22 14:49:41,485 INFO: Retrying on http://192.168.152.122:2379
2026-05-22 14:49:43,153 ERROR: Request to server http://192.168.152.122:2379 failed: MaxRetryError("HTTPConnectionPool(host='192.168.152.122', port=2379): Max retries exceeded with url: /v3/lease/keepalive (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f2027e45fa0>, 'Connection to 192.168.152.122 timed out. (connect timeout=1.6666666666666667)'))")
2026-05-22 14:49:43,153 INFO: Reconnection allowed, looking for another server.
2026-05-22 14:49:43,153 INFO: Retrying on http://192.168.152.123:2379
2026-05-22 14:49:44,823 ERROR: Request to server http://192.168.152.123:2379 failed: MaxRetryError("HTTPConnectionPool(host='192.168.152.123', port=2379): Max retries exceeded with url: /v3/lease/keepalive (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f2027e459d0>, 'Connection to 192.168.152.123 timed out. (connect timeout=1.6666666666666667)'))")
2026-05-22 14:49:44,823 INFO: Reconnection allowed, looking for another server.
2026-05-22 14:49:46,493 ERROR: Failed to get list of machines from http://192.168.152.122:2379/v3: MaxRetryError("HTTPConnectionPool(host='192.168.152.122', port=2379): Max retries exceeded with url: /v3/cluster/member/list (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f2027e45ac0>, 'Connection to 192.168.152.122 timed out. (connect timeout=1.6666666666666667)'))")
2026-05-22 14:49:48,150 ERROR: Request to server http://192.168.152.121:2379 failed: ReadTimeoutError("HTTPConnectionPool(host='192.168.152.121', port=2379): Read timed out. (read timeout=1.4432267216546961)")
2026-05-22 14:49:48,150 INFO: Reconnection allowed, looking for another server.
2026-05-22 14:49:49,819 ERROR: Failed to get list of machines from http://192.168.152.122:2379/v3: MaxRetryError("HTTPConnectionPool(host='192.168.152.122', port=2379): Max retries exceeded with url: /v3/cluster/member/list (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f2027e45100>, 'Connection to 192.168.152.122 timed out. (connect timeout=1.6666666666666667)'))")
2026-05-22 14:49:49,821 ERROR: Error communicating with DCS
2026-05-22 14:49:49,821 ERROR: watchprefix failed: ProtocolError('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
2026-05-22 14:49:49,821 INFO: DCS is not accessible

Ubuntu 12解除网络分区，

root@ubuntu12:/usr/local/patroni_install# sudo iptables -D OUTPUT -d 192.168.152.121 -j DROP
root@ubuntu12:/usr/local/patroni_install# sudo iptables -D INPUT  -s 192.168.152.121 -j DROP

Ubuntu 13上也解除网络分区

root@ubuntu13:/usr/local/patroni_install# sudo iptables -D OUTPUT -d 192.168.152.121 -j DROP
root@ubuntu13:/usr/local/patroni_install# sudo iptables -D INPUT  -s 192.168.152.121 -j DROP
root@ubuntu13:/usr/local/patroni_install#
root@ubuntu13:/usr/local/patroni_install#

可以发现被隔离的Ubuntu11自动以从节点身份加入集群。

root@ubuntu12:/usr/local/patroni_install#
root@ubuntu12:/usr/local/patroni_install#
root@ubuntu12:/usr/local/patroni_install# patronictl -c /usr/local/pgsql17/patroni/patroni.yml list
+ Cluster: pg_cluster_wy_prod (7642589780522937440) ---------+----+-------------+-----+------------+-----+
| Member   | Host                 | Role         | State     | TL | Receive LSN | Lag | Replay LSN | Lag |
+----------+----------------------+--------------+-----------+----+-------------+-----+------------+-----+
| ubuntu11 | 192.168.152.121:9000 | Replica      | streaming |  5 |   0/60043F0 |   0 |  0/60043F0 |   0 |
| ubuntu12 | 192.168.152.122:9000 | Leader       | running   |  5 |             |     |            |     |
| ubuntu13 | 192.168.152.123:9000 | Sync Standby | streaming |  5 |   0/60043F0 |   0 |  0/60043F0 |   0 |
+----------+----------------------+--------------+-----------+----+-------------+-----+------------+-----+
root@ubuntu12:/usr/local/patroni_install#
root@ubuntu12:/usr/local/patroni_install#

Ubuntu11上的日志，自动执行pg_rewind，然后以从节点的身份加入集群

2026-05-22 15:02:40,971 ERROR: Failed to get list of machines from http://192.168.152.122:2379/v3: MaxRetryError("HTTPConnectionPool(host='192.168.152.122', port=2379): Max retries exceeded with url: /v3/cluster/member/list (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f2027e5c4c0>, 'Connection to 192.168.152.122 timed out. (connect timeout=1.6666666666666667)'))")
2026-05-22 15:02:42,626 ERROR: Request to server http://192.168.152.121:2379 failed: ReadTimeoutError("HTTPConnectionPool(host='192.168.152.121', port=2379): Read timed out. (read timeout=1.3500901345867078)")
2026-05-22 15:02:42,626 INFO: Reconnection allowed, looking for another server.
2026-05-22 15:02:44,295 ERROR: Failed to get list of machines from http://192.168.152.122:2379/v3: MaxRetryError("HTTPConnectionPool(host='192.168.152.122', port=2379): Max retries exceeded with url: /v3/cluster/member/list (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f2027e5cb80>, 'Connection to 192.168.152.122 timed out. (connect timeout=1.6666666666666667)'))")
2026-05-22 15:02:44,296 ERROR: Error communicating with DCS
2026-05-22 15:02:44,297 ERROR: watchprefix failed: ProtocolError('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
2026-05-22 15:02:44,297 INFO: DCS is not accessible
2026-05-22 15:02:44,298 WARNING: Loop time exceeded, rescheduling immediately.
2026-05-22 15:02:45,967 ERROR: Failed to get list of machines from http://192.168.152.122:2379/v3: MaxRetryError("HTTPConnectionPool(host='192.168.152.122', port=2379): Max retries exceeded with url: /v3/cluster/member/list (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f205c04b3a0>, 'Connection to 192.168.152.122 timed out. (connect timeout=1.6666666666666667)'))")
2026-05-22 15:02:45,975 INFO: Lock owner: ubuntu11; I am ubuntu11
2026-05-22 15:02:47,985 ERROR: failed to update leader lock
2026-05-22 15:02:47,994 INFO: not promoting because failed to update leader lock in DCS
2026-05-22 15:02:47,994 INFO: Lock owner: ubuntu12; I am ubuntu11
2026-05-22 15:02:48,001 INFO: Local timeline=4 lsn=0/70000A0
2026-05-22 15:02:48,027 INFO: primary_timeline=5
2026-05-22 15:02:48,030 INFO: primary: history=1	0/504F580	no recovery target specified
2	0/6003D20	no recovery target specified
3	0/6003EC0	no recovery target specified
4	0/6004148	no recovery target specified
2026-05-22 15:02:48,049 INFO: running pg_rewind from ubuntu12
2026-05-22 15:02:49,312 INFO: running pg_rewind from dbname=postgres user=rewind_user host=192.168.152.122 port=9000 target_session_attrs=read-write
2026-05-22 15:02:50,305 INFO: pg_rewind exit code=0
2026-05-22 15:02:50,305 INFO:  stdout=
2026-05-22 15:02:50,305 INFO:  stderr=pg_rewind: servers diverged at WAL location 0/6004148 on timeline 4
pg_rewind: rewinding from last common checkpoint at 0/6004038 on timeline 4
pg_rewind: Done!

2026-05-22 15:02:50,307 WARNING: Postgresql is not running.
2026-05-22 15:02:50,308 INFO: Lock owner: ubuntu12; I am ubuntu11
2026-05-22 15:02:50,319 INFO: pg_controldata:
  pg_control version number: 1700
  Catalog version number: 202406281
  Database system identifier: 7642589780522937440
  Database cluster state: in archive recovery
  pg_control last modified: Fri May 22 15:02:50 2026
  Latest checkpoint location: 0/6004340
  Latest checkpoint's REDO location: 0/60042E8
  Latest checkpoint's REDO WAL file: 000000050000000000000006
  Latest checkpoint's TimeLineID: 5
  Latest checkpoint's PrevTimeLineID: 5
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:762
  Latest checkpoint's NextOID: 24576
  Latest checkpoint's NextMultiXactId: 1
  Latest checkpoint's NextMultiOffset: 0
  Latest checkpoint's oldestXID: 731
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 762
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Fri May 22 14:53:31 2026
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/60043F0
  Min recovery ending loc's timeline: 5
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: replica
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 10
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 3587dd0ff212f7ed05a16aa24aa1d6a6f187f55d5d6a2e158ce45327a7e55005

2026-05-22 15:02:50,320 INFO: Lock owner: ubuntu12; I am ubuntu11
2026-05-22 15:02:50,367 INFO: starting as a secondary
2026-05-22 15:02:50,368 INFO: closed patroni connections to postgres
2026-05-22 15:02:50,738 INFO: postmaster pid=3952
2026-05-22 15:02:51,774 INFO: Lock owner: ubuntu12; I am ubuntu11
2026-05-22 15:02:51,774 INFO: establishing a new patroni heartbeat connection to postgres
2026-05-22 15:02:51,795 INFO: Local timeline=5 lsn=0/60043F0
2026-05-22 15:02:51,803 INFO: primary_timeline=5
2026-05-22 15:02:51,812 INFO: no action. I am (ubuntu11), a secondary, and following a leader (ubuntu12)
2026-05-22 15:02:52,281 INFO: no action. I am (ubuntu11), a secondary, and following a leader (ubuntu12)
2026-05-22 15:03:02,819 INFO: no action. I am (ubuntu11), a secondary, and following a leader (ubuntu12)
2026-05-22 15:03:12,778 INFO: no action. I am (ubuntu11), a secondary, and following a leader (ubuntu12)
2026-05-22 15:03:22,777 INFO: no action. I am (ubuntu11), a secondary, and following a leader (ubuntu12)

4，自动故障转移场景4：暴力删除主节点PostgreSQL数据文件

这种情况实际情况中几乎不会发生，除非刻意为之，那么直接暴力删除运行中的主节点的PostgreSQL的数据文件会发生什么？

以下是实测，可以发现，暴力删除数据文件之后：1，patroni集群会自动故障转移（因为主节点无法对外提供服务了），2，原主节点会自动从集群中克隆一份数据作为从节点运行

######当前集群正常状态
root@ubuntu11:~# patronictl -c /usr/local/pgsql17/patroni/patroni.yml list
+ Cluster: pg_cluster_wy_prod (7642589780522937440) ---------+----+-------------+-----+------------+-----+
| Member   | Host                 | Role         | State     | TL | Receive LSN | Lag | Replay LSN | Lag |
+----------+----------------------+--------------+-----------+----+-------------+-----+------------+-----+
| ubuntu11 | 192.168.152.121:9000 | Sync Standby | streaming |  6 |   0/70002D0 |   0 |  0/70002D0 |   0 |
| ubuntu12 | 192.168.152.122:9000 | Replica      | streaming |  6 |   0/70002D0 |   0 |  0/70002D0 |   0 |
| ubuntu13 | 192.168.152.123:9000 | Leader       | running   |  6 |             |     |            |     |
+----------+----------------------+--------------+-----------+----+-------------+-----+------------+-----+
root@ubuntu11:~#
root@ubuntu11:~#
root@ubuntu11:~#
root@ubuntu11:~#
######暴力删除原主节点数据文件
root@ubuntu11:~# patronictl -c /usr/local/pgsql17/patroni/patroni.yml list
+ Cluster: pg_cluster_wy_prod (7642589780522937440) ----------------+----+-------------+-----+------------+-----+
| Member   | Host                 | Role         | State            | TL | Receive LSN | Lag | Replay LSN | Lag |
+----------+----------------------+--------------+------------------+----+-------------+-----+------------+-----+
| ubuntu11 | 192.168.152.121:9000 | Leader       | running          |  7 |             |     |            |     |
| ubuntu12 | 192.168.152.122:9000 | Sync Standby | streaming        |  7 |   0/7000410 |   0 |  0/7000410 |   0 |
| ubuntu13 | 192.168.152.123:9000 | Replica      | creating replica |    |     unknown |     |    unknown |     |
+----------+----------------------+--------------+------------------+----+-------------+-----+------------+-----+
root@ubuntu11:~#
######集群恢复正常
root@ubuntu11:~# patronictl -c /usr/local/pgsql17/patroni/patroni.yml list
+ Cluster: pg_cluster_wy_prod (7642589780522937440) ---------+----+-------------+-----+------------+-----+
| Member   | Host                 | Role         | State     | TL | Receive LSN | Lag | Replay LSN | Lag |
+----------+----------------------+--------------+-----------+----+-------------+-----+------------+-----+
| ubuntu11 | 192.168.152.121:9000 | Leader       | running   |  7 |             |     |            |     |
| ubuntu12 | 192.168.152.122:9000 | Sync Standby | streaming |  7 |   0/7000410 |   0 |  0/7000410 |   0 |
| ubuntu13 | 192.168.152.123:9000 | Replica      | streaming |  7 |   0/9000000 |   0 |  0/9000000 |   0 |
+----------+----------------------+--------------+-----------+----+-------------+-----+------------+-----+

原主节点日志，可以发现：replica has been created using basebackup，也就是说当前损坏的节点，会自动基于basebackup备份一份数据，自动恢复并加入集群，就像打不死的小强一样坚强。当然如果库比较大的情况就另说。

026-05-22 16:39:25,903 INFO: no action. I am (ubuntu13), the leader with the lock
2026-05-22 16:39:35,943 INFO: no action. I am (ubuntu13), the leader with the lock
2026-05-22 16:39:45,901 INFO: no action. I am (ubuntu13), the leader with the lock
2026-05-22 16:39:55,902 INFO: no action. I am (ubuntu13), the leader with the lock
2026-05-22 16:40:05,944 INFO: no action. I am (ubuntu13), the leader with the lock
2026-05-22 16:40:15,902 INFO: no action. I am (ubuntu13), the leader with the lock
2026-05-22 16:40:25,901 INFO: no action. I am (ubuntu13), the leader with the lock
2026-05-22 16:40:35,947 INFO: no action. I am (ubuntu13), the leader with the lock
2026-05-22 16:40:45,904 INFO: no action. I am (ubuntu13), the leader with the lock
2026-05-22 16:40:55,899 INFO: Lock owner: ubuntu13; I am ubuntu13
2026-05-22 16:40:57,570 ERROR: Failed to get list of machines from http://192.168.152.122:2379/v3: MaxRetryError("HTTPConnectionPool(host='192.168.152.122', port=2379): Max retries exceeded with url: /v3/cluster/member/list (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7fd2642bc7c0>, 'Connection to 192.168.152.122 timed out. (connect timeout=1.6666666666666667)'))")
2026-05-22 16:40:57,573 INFO: no action. I am (ubuntu13), the leader with the lock
2026-05-22 16:41:05,902 INFO: no action. I am (ubuntu13), the leader with the lock
2026-05-22 16:41:15,901 INFO: no action. I am (ubuntu13), the leader with the lock
2026-05-22 16:41:25,905 INFO: no action. I am (ubuntu13), the leader with the lock
2026-05-22 16:41:35,944 INFO: no action. I am (ubuntu13), the leader with the lock
2026-05-22 16:41:45,903 INFO: no action. I am (ubuntu13), the leader with the lock
2026-05-22 16:41:55,908 INFO: no action. I am (ubuntu13), the leader with the lock
2026-05-22 16:42:05,945 INFO: no action. I am (ubuntu13), the leader with the lock
2026-05-22 16:42:15,905 INFO: no action. I am (ubuntu13), the leader with the lock
2026-05-22 16:42:25,902 INFO: no action. I am (ubuntu13), the leader with the lock
2026-05-22 16:42:35,943 INFO: no action. I am (ubuntu13), the leader with the lock
2026-05-22 16:42:45,901 INFO: no action. I am (ubuntu13), the leader with the lock
2026-05-22 16:42:55,901 INFO: no action. I am (ubuntu13), the leader with the lock
2026-05-22 16:43:05,944 INFO: no action. I am (ubuntu13), the leader with the lock
2026-05-22 16:43:15,901 INFO: no action. I am (ubuntu13), the leader with the lock
2026-05-22 16:43:25,904 INFO: no action. I am (ubuntu13), the leader with the lock
2026-05-22 16:43:35,906 INFO: Lock owner: ubuntu13; I am ubuntu13
2026-05-22 16:43:35,958 INFO: Leader key released
2026-05-22 16:43:35,961 INFO: released leader key voluntarily as data dir empty and currently leader
2026-05-22 16:43:35,961 INFO: Lock owner: None; I am ubuntu13
2026-05-22 16:43:36,003 INFO: waiting for leader to bootstrap
2026-05-22 16:43:36,016 INFO: Lock owner: ubuntu11; I am ubuntu13
2026-05-22 16:43:36,018 INFO: trying to bootstrap from leader 'ubuntu11'
2026-05-22 16:43:36,134 ERROR: Error when fetching backup: pg_basebackup exited with code=1
2026-05-22 16:43:36,135 WARNING: Trying again in 5 seconds
2026-05-22 16:43:37,069 INFO: Lock owner: ubuntu11; I am ubuntu13
2026-05-22 16:43:37,116 INFO: bootstrap from leader 'ubuntu11' in progress
2026-05-22 16:43:42,134 INFO: replica has been created using basebackup
2026-05-22 16:43:42,135 INFO: bootstrapped from leader 'ubuntu11'
2026-05-22 16:43:42,531 INFO: postmaster pid=25110
2026-05-22 16:43:43,580 INFO: Lock owner: ubuntu11; I am ubuntu13
2026-05-22 16:43:43,580 INFO: establishing a new patroni heartbeat connection to postgres
2026-05-22 16:43:43,602 INFO: Local timeline=7 lsn=0/9000000
2026-05-22 16:43:43,628 INFO: primary_timeline=7
2026-05-22 16:43:43,665 INFO: no action. I am (ubuntu13), a secondary, and following a leader (ubuntu11)
2026-05-22 16:43:47,112 INFO: no action. I am (ubuntu13), a secondary, and following a leader (ubuntu11)
2026-05-22 16:43:57,127 INFO: no action. I am (ubuntu13), a secondary, and following a leader (ubuntu11)
2026-05-22 16:44:07,584 INFO: no action. I am (ubuntu13), a secondary, and following a leader (ubuntu11)
2026-05-22 16:44:17,625 INFO: no action. I am (ubuntu13), a secondary, and following a leader (ubuntu11)
2026-05-22 16:44:27,579 INFO: no action. I am (ubuntu13), a secondary, and following a leader (ubuntu11)
2026-05-22 16:44:37,584 INFO: no action. I am (ubuntu13), a secondary, and following a leader (ubuntu11)
2026-05-22 16:44:47,627 INFO: no action. I am (ubuntu13), a secondary, and following a leader (ubuntu11)
2026-05-22 16:44:57,579 INFO: no action. I am (ubuntu13), a secondary, and following a leader (ubuntu11)
2026-05-22 16:45:07,587 INFO: no action. I am (ubuntu13), a secondary, and following a leader (ubuntu11)
2026-05-22 16:45:17,619 INFO: no action. I am (ubuntu13), a secondary, and following a leader (ubuntu11)

5，总结

本文通过三种实际的故障，严苛测试了patroni故障转移集群的高可用性，可以发现patroni可以完美处理各种故障，实现集群的高可用性，同时对于故障转移集群的ttl参数，以及loop_wait参数，在故障转移中的作用，做了实操性的验证，笔者本身也对这两个参数有了更加深刻的认识。

此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。原文来自 — 版权归原作者所有。

推荐订阅源

博客园_首页

1，自动故障转移场景1：主节点OS正常，patroni服务异常故障

2，自动故障转移场景2：主节点服务器断电

3，自动故障转移场景3：主节点网络分区

4，自动故障转移场景4：暴力删除主节点PostgreSQL数据文件

5，总结