惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

SecWiki News
SecWiki News
I
InfoQ
The Cloudflare Blog
人人都是产品经理
人人都是产品经理
博客园 - Franky
T
Tailwind CSS Blog
让小产品的独立变现更简单 - ezindie.com
让小产品的独立变现更简单 - ezindie.com
量子位
博客园_首页
罗磊的独立博客
V
V2EX
李成银的技术随笔
大猫的无限游戏
大猫的无限游戏
钛媒体:引领未来商业与生活新知
钛媒体:引领未来商业与生活新知
T
True Tiger Recordings
Vercel News
Vercel News
Cyberwarzone
Cyberwarzone
Cisco Talos Blog
Cisco Talos Blog
F
Fox-IT International blog
D
Darknet – Hacking Tools, Hacker News & Cyber Security
M
Microsoft Research Blog - Microsoft Research
Know Your Adversary
Know Your Adversary
爱范儿
爱范儿
The Register - Security
The Register - Security
G
Google Developers Blog
The Hacker News
The Hacker News
Malwarebytes
Malwarebytes
S
Securelist
博客园 - 三生石上(FineUI控件)
Jina AI
Jina AI
T
Threat Research - Cisco Blogs
T
The Exploit Database - CXSecurity.com
S
SegmentFault 最新的问题
博客园 - 叶小钗
F
Fortinet All Blogs
Apple Machine Learning Research
Apple Machine Learning Research
宝玉的分享
宝玉的分享
博客园 - 聂微东
T
Threatpost
博客园 - 【当耐特】
D
Docker
P
Privacy & Cybersecurity Law Blog
www.infosecurity-magazine.com
www.infosecurity-magazine.com
G
GRAHAM CLULEY
V
Visual Studio Blog
C
Cisco Blogs
IT之家
IT之家
S
Security Archives - TechRepublic
Latest news
Latest news
阮一峰的网络日志
阮一峰的网络日志

运维开发绿皮书

HTTPS 双向认证与 USB 加密锁配置实战 颜色转换 字数统计 UUID 生成 URL 编码 时间戳转换 房贷计算 JWT 解码 JSON 格式化 Hash 生成器 BMI 计算 二维码批量生成器 TOTP验证码生成器 CKS Simulator Kubernetes 1.25 direct-ssh-passthrough-nat 脚本使用说明 汽车的分类和特点 Python软件授权 为 Containerd 配置 Harbor 无证书镜像站 必应搜索屏蔽垃圾网站 VMware的ubuntu完整安装vm-tools支持粘贴板 Docker Desktop 安装到其他位置 tc常用命令总结 稳定币监管与投资指南 VMware ESXi 密码恢复指南 VMware 加密 state.tgz 文件解密教程 六百万数据 MySQL count(*) 优化 使用iptables禁止特定子网访问指定端口 Linux 一键测试脚本 Gitlab配置详解 Gitlab批量创建用户 Dell EMC PowerEdge R740服务器内存插槽使用说明及正确安装方法 Ubuntu安装VNC教程 Windows11跳过TPM2.0 vue配置Nginx伪静态 Linux下Ollama开放端口 解决 iptables DNAT 无法转发到 127.0.0.1 的问题及安全风险分析 模型详解配置 修改Git提交历史中的作者信息 在PowerShell中启动Git Bash的方法 一键安装OpenVPN 大语言模型 训练相关资料 模型架构-Transformer模型 GPT&DeepSeek模型 大模型技术基础 Git Bash 中一键安装 pacman 牛马时钟 Gitlab-Runner的一些问题 Docker运行kwaivgi-liveportrait FRP 在x86-64上构建和运行多种架构的Docker容器 Firefox密码提取 flutter配置镜像站 一键安装Conda Docker老版本runc报错无法启动 清理Rancher节点 aapanel 7.x 中文语言配置指南 Ubuntu 22 安装 Kubernetes 1.29 集群指南 Ubuntu修复CNVD-2024-4920726 GRUB锁定密码 Cron在线表达式生成器 VMware最新下载地址(纪念碑) 文本字符串倒序 Windows禁用任务管理器 Windows11恢复Windows10右键菜单 Ubuntu一键设置镜像源 Base64在线编码解码 代码差异对比 SSL 证书工具 为Docker分配物理网卡 UOS(1070a)服务器版本部署Kubernetes1.28 解密CFSSL生成的CRL数据 已经在谷底了 怎么走都是向上 GitLab一键设置镜像源 Ubuntu备份为LiveOS Windows11的24H2出现扩展错误 CentOS 7 一键设置Vault镜像 Fossy平台离线部署分析 在WSL中移除Windows环境变量 Grub2手动引导Linux Windows11跳过微软账号登录 Windows绕过MicroSoft Store直接下载应用 重装MicroSoft Store Docker一键部署Meta和MetacubexD面板 pnpm在DevContainer中存储出错解决方法 从SVN迁移仓库到Git Ubuntu中移除Snap软件包 GitLab统计提交代码行数Python代码 使用Netcat检测UDP端口连通性 Sony Xperia 10 IV(pdx225)AOSP14编译和刷机方法 Linux命令行百度网盘 rke2集群命令行调试方法 Ubuntu隐私优化-关闭公共门户连接检查 Cisco路由器配置ipv4和ipv6的VTY账户密码 使用Powershell卸载windows默认程序 Python实现九九乘法表 Cisco路由器OSPF配置 Windows10中用多网卡链路聚合来解决网卡网速瓶颈 CentOS7 安装 OpenResty
OpenStack排错
2024-08-02 · via 运维开发绿皮书

OpenStack排错

问题:创建云主机失败 - "Exceeded maximum number of retries"

问题描述

创建云主机时报错:Exceeded maximum number of retries. Exhausted all hosts available for retrying build,导致无法正常创建云主机实例。

问题分析

控制节点(Controller)日志分析

1. Neutron服务日志

查看neutron服务器日志中的错误信息:

grep ERROR /var/log/neutron/server.log

关键错误信息:

2020-11-22 18:17:30.537 27477 ERROR neutron.plugins.ml2.managers [req-c117507f-a7fd-452d-a0a3-fcf928c7d115 66c4a34b461444afac62a6c7416c109b df1082a78def4615a858b64a6c12c198 - 9a37718adbcf441aa294c93d40bb91bc 9a37718adbcf441aa294c93d40bb91bc] Failed to bind port 6c119e38-8772-49f4-ada1-a4d8ae3073fd on host compute for vnic_type normal using segments [{'network_id': '328a483e-a346-448c-9c45-c26fd0162754', 'segmentation_id': 3, 'physical_network': None, 'id': 'e11308a4-d612-4604-851a-8aa90754885d', 'network_type': u'vxlan'}]
2020-11-22 18:17:30.564 27477 ERROR neutron.plugins.ml2.managers [req-c117507f-a7fd-452d-a0a3-fcf928c7d115 66c4a34b461444afac62a6c7416c109b df1082a78def4615a858b64a6c12c198 - 9a37718adbcf441aa294c93d40bb91bc 9a37718adbcf441aa294c93d40bb91bc] Failed to bind port 6c119e38-8772-49f4-ada1-a4d8ae3073fd on host compute for vnic_type normal using segments [{'network_id': '328a483e-a346-448c-9c45-c26fd0162754', 'segmentation_id': 3, 'physical_network': None, 'id': 'e11308a4-d612-4604-851a-8aa90754885d', 'network_type': u'vxlan'}]
2020-11-22 18:17:30.588 27477 ERROR neutron.plugins.ml2.managers [req-c117507f-a7fd-452d-a0a3-fcf928c7d115 66c4a34b461444afac62a6c7416c109b df1082a78def4615a858b64a6c12c198 - 9a37718adbcf441aa294c93d40bb91bc 9a37718adbcf441aa294c93d40bb91bc] Failed to bind port 6c119e38-8772-49f4-ada1-a4d8ae3073fd on host compute for vnic_type normal using segments [{'network_id': '328a483e-a346-448c-9c45-c26fd0162754', 'segmentation_id': 3, 'physical_network': None, 'id': 'e11308a4-d612-4604-851a-8aa90754885d', 'network_type': u'vxlan'}]
2020-11-22 18:17:30.612 27477 ERROR neutron.plugins.ml2.managers [req-c117507f-a7fd-452d-a0a3-fcf928c7d115 66c4a34b461444afac62a6c7416c109b df1082a78def4615a858b64a6c12c198 - 9a37718adbcf441aa294c93d40bb91bc 9a37718adbcf441aa294c93d40bb91bc] Failed to bind port 6c119e38-8772-49f4-ada1-a4d8ae3073fd on host compute for vnic_type normal using segments [{'network_id': '328a483e-a346-448c-9c45-c26fd0162754', 'segmentation_id': 3, 'physical_network': None, 'id': 'e11308a4-d612-4604-851a-8aa90754885d', 'network_type': u'vxlan'}]
2020-11-22 18:17:30.635 27477 ERROR neutron.plugins.ml2.managers [req-c117507f-a7fd-452d-a0a3-fcf928c7d115 66c4a34b461444afac62a6c7416c109b df1082a78def4615a858b64a6c12c198 - 9a37718adbcf441aa294c93d40bb91bc 9a37718adbcf441aa294c93d40bb91bc] Failed to bind port 6c119e38-8772-49f4-ada1-a4d8ae3073fd on host compute for vnic_type normal using segments [{'network_id': '328a483e-a346-448c-9c45-c26fd0162754', 'segmentation_id': 3, 'physical_network': None, 'id': 'e11308a4-d612-4604-851a-8aa90754885d', 'network_type': u'vxlan'}]

其他neutron组件日志检查:

[root@controller neutron]# grep ERROR linuxbridge-agent.log 
[root@controller neutron]# grep ERROR l3-agent.log 
[root@controller neutron]# grep ERROR dhcp-agent.log 
[root@controller neutron]# grep ERROR metadata-agent.log
2. Nova服务日志

查看nova-conductor.log中的关键错误:

2020-11-22 18:09:49.570 22881 WARNING oslo_config.cfg [req-33860c7c-9b1f-4b2d-a0f1-401ac21d43d0 83b70ce8339e4673ae33edf613291108 b91453e898d2402495f54b8533bc
002b - 9a37718adbcf441aa294c93d40bb91bc 9a37718adbcf441aa294c93d40bb91bc] Option "url" from group "neutron" is deprecated for removal (Endpoint lookup uses t
he service catalog via common keystoneauth1 Adapter configuration options. In the current release, "url" will override this behavior, but will be ignored and
/or removed in a future release. To achieve the same result, use the endpoint_override option instead.).  Its value may be silently ignored in the future.
2020-11-22 18:11:31.636 22880 ERROR nova.scheduler.utils [req-fe2b1525-0dc9-44ef-8ac1-013fc083323f 83b70ce8339e4673ae33edf613291108 b91453e898d2402495f54b853
3bc002b - 9a37718adbcf441aa294c93d40bb91bc 9a37718adbcf441aa294c93d40bb91bc] [instance: a9f3b569-708e-4a56-8f57-51e17924a1bc] Error from last host: compute (
node compute): [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1862, in _do_build_and_run
_instance\n    filter_properties, request_spec)\n', u'  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2142, in _build_and_run_instanc
e\n    instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance a9f3b569-708e-4a56-8f57-51e17924a1bc was re-schedu
led: Binding failed for port b924b7e9-0dfc-41a5-860a-4c0002fc7c68, please check neutron logs for more information.\n']
2020-11-22 18:11:31.638 22880 WARNING nova.scheduler.utils [req-fe2b1525-0dc9-44ef-8ac1-013fc083323f 83b70ce8339e4673ae33edf613291108 b91453e898d2402495f54b8
533bc002b - 9a37718adbcf441aa294c93d40bb91bc 9a37718adbcf441aa294c93d40bb91bc] Failed to compute_task_build_instances: Exceeded maximum number of retries. Ex
hausted all hosts available for retrying build failures for instance a9f3b569-708e-4a56-8f57-51e17924a1bc.: MaxRetriesExceeded: Exceeded maximum number of re
tries. Exhausted all hosts available for retrying build failures for instance a9f3b569-708e-4a56-8f57-51e17924a1bc.
2020-11-22 18:11:31.638 22880 WARNING nova.scheduler.utils [req-fe2b1525-0dc9-44ef-8ac1-013fc083323f 83b70ce8339e4673ae33edf613291108 b91453e898d2402495f54b8
533bc002b - 9a37718adbcf441aa294c93d40bb91bc 9a37718adbcf441aa294c93d40bb91bc] [instance: a9f3b569-708e-4a56-8f57-51e17924a1bc] Setting instance to ERROR sta
te.: MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance a9f3b569-708e-4a56-8f57-5
1e17924a1bc.
2020-11-22 18:17:33.047 22878 ERROR nova.scheduler.utils [req-47aa5b56-67c2-4dd3-9491-077d0bc12a11 83b70ce8339e4673ae33edf613291108 b91453e898d2402495f54b853
3bc002b - 9a37718adbcf441aa294c93d40bb91bc 9a37718adbcf441aa294c93d40bb91bc] [instance: 3c141ca8-0545-4627-af9b-30c84ec3454b] Error from last host: compute (
node compute): [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1862, in _do_build_and_run
_instance\n    filter_properties, request_spec)\n', u'  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2142, in _build_and_run_instanc
e\n    instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 3c141ca8-0545-4627-af9b-30c84ec3454b was re-schedu
led: Binding failed for port 6c119e38-8772-49f4-ada1-a4d8ae3073fd, please check neutron logs for more information.\n']
2020-11-22 18:17:33.048 22878 WARNING nova.scheduler.utils [req-47aa5b56-67c2-4dd3-9491-077d0bc12a11 83b70ce8339e4673ae33edf613291108 b91453e898d2402495f54b8
533bc002b - 9a37718adbcf441aa294c93d40bb91bc 9a37718adbcf441aa294c93d40bb91bc] Failed to compute_task_build_instances: Exceeded maximum number of retries. Ex
hausted all hosts available for retrying build failures for instance 3c141ca8-0545-4627-af9b-30c84ec3454b.: MaxRetriesExceeded: Exceeded maximum number of re
tries. Exhausted all hosts available for retrying build failures for instance 3c141ca8-0545-4627-af9b-30c84ec3454b.
2020-11-22 18:17:33.049 22878 WARNING nova.scheduler.utils [req-47aa5b56-67c2-4dd3-9491-077d0bc12a11 83b70ce8339e4673ae33edf613291108 b91453e898d2402495f54b8
533bc002b - 9a37718adbcf441aa294c93d40bb91bc 9a37718adbcf441aa294c93d40bb91bc] [instance: 3c141ca8-0545-4627-af9b-30c84ec3454b] Setting instance to ERROR sta
te.: MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 3c141ca8-0545-4627-af9b-3
0c84ec3454b.
2020-11-22 18:17:33.151 22878 WARNING oslo_config.cfg [req-47aa5b56-67c2-4dd3-9491-077d0bc12a11 83b70ce8339e4673ae33edf613291108 b91453e898d2402495f54b8533bc
002b - 9a37718adbcf441aa294c93d40bb91bc 9a37718adbcf441aa294c93d40bb91bc] Option "url" from group "neutron" is deprecated for removal (Endpoint lookup uses t
he service catalog via common keystoneauth1 Adapter configuration options. In the current release, "url" will override this behavior, but will be ignored and
/or removed in a future release. To achieve the same result, use the endpoint_override option instead.).  Its value may be silently ignored in the future.

均无法看出错误

计算节点(Compute)日志分析

LinuxBridge代理日志

查看计算节点的linuxbridge-agent.log:

tail -f /var/log/neutron/linuxbridge-agent.log
_ip bound to an interface on the host. Please configure local_ip 172.17.1.10 on the host interface to be used for tunneling and restart the agent.
2020-11-22 18:18:37.411 12909 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Tunneling cannot be enabled without the local_ip bound to an interface on the host. Please configure local_ip 172.17.1.10 on the host interface to be used for tunneling and restart the agent.
2020-11-22 18:18:40.376 12933 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Tunneling cannot be enabled without the local_ip bound to an interface on the host. Please configure local_ip 172.17.1.10 on the host interface to be used for tunneling and restart the agent.
2020-11-22 18:18:43.384 12957 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Tunneling cannot be enabled without the local_ip bound to an interface on the host. Please configure local_ip 172.17.1.10 on the host interface to be used for tunneling and restart the agent.
2020-11-22 18:18:46.179 12981 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Tunneling cannot be enabled without the local_ip bound to an interface on the host. Please configure local_ip 172.17.1.10 on the host interface to be used for tunneling and restart the agent.
2020-11-22 18:18:49.139 13005 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Tunneling cannot be enabled without the local_ip bound to an interface on the host. Please configure local_ip 172.17.1.10 on the host interface to be used for tunneling and restart the agent.
2020-11-22 18:18:52.156 13029 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Tunneling cannot be enabled without the local_ip bound to an interface on the host. Please configure local_ip 172.17.1.10 on the host interface to be used for tunneling and restart the agent.
2020-11-22 18:18:54.938 13056 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Tunneling cannot be enabled without the local_ip bound to an interface on the host. Please configure local_ip 172.17.1.10 on the host interface to be used for tunneling and restart the agent.
2020-11-22 18:18:57.886 13086 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Tunneling cannot be enabled without the local_ip bound to an interface on the host. Please configure local_ip 172.17.1.10 on the host interface to be used for tunneling and restart the agent.
2020-11-22 18:19:00.898 13114 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Tunneling cannot be enabled without the local_ip bound to an interface on the host. Please configure local_ip 172.17.1.10 on the host interface to be used for tunneling and restart the agent.
2020-11-22 18:19:03.881 13146 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Tunneling cannot be enabled without the local_ip bound to an interface on the host. Please configure local_ip 172.17.1.10 on the host interface to be used for tunneling and restart the agent.
2020-11-22 18:19:06.870 13171 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Tunneling cannot be enabled without the local_ip bound to an interface on the host. Please configure local_ip 172.17.1.10 on the host interface to be used for tunneling and restart the agent.
2020-11-22 18:19:09.873 13195 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Tunneling cannot be enabled without the local_ip bound to an interface on the host. Please configure local_ip 172.17.1.10 on the host interface to be used for tunneling and restart the agent.
2020-11-22 18:19:12.878 13219 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Tunneling cannot be enabled without the local_ip bound to an interface on the host. Please configure local_ip 172.17.1.10 on the host interface to be used for tunneling and restart the agent.

问题找到了,compute找不到Tunnel Network Interface. example:x.x.x.x

INTERFACE_IP=172.17.1.20

手误,更改 TUNNELIP后创建云主机正常

对于Q版快速更改方法

根本原因

计算节点无法找到正确的隧道网络接口(Tunnel Network Interface),配置的local_ip地址与实际网络接口不匹配。

解决方案

方法一:手动修改配置文件

  1. 确认计算节点的正确IP地址:
ip addr show
  1. 修改linuxbridge_agent配置文件:
vi /etc/neutron/plugins/ml2/linuxbridge_agent.ini

[vxlan]段中修改:

[vxlan]
local_ip = 172.17.1.20  # 替换为正确的IP地址

方法二:使用crudini工具快速修改(推荐)

对于OpenStack Queens版本,可以使用以下命令快速修改配置:

# 修改物理接口映射
crudini --set /etc/neutron/plugins/ml2/linuxbridge_agent.ini linux_bridge physical_interface_mappings provider:172.17.1.20

# 修改VXLAN本地IP
crudini --set /etc/neutron/plugins/ml2/linuxbridge_agent.ini vxlan local_ip 172.17.1.20

重启服务

配置修改完成后,重启neutron-linuxbridge-agent服务:

systemctl restart neutron-linuxbridge-agent

验证解决方案

检查服务状态:

systemctl status neutron-linuxbridge-agent

查看日志确认无错误:

tail -f /var/log/neutron/linuxbridge-agent.log

尝试重新创建云主机实例

预防措施

  • 在部署OpenStack时,确保计算节点的网络配置正确
  • 定期检查neutron代理的状态和日志
  • 在修改网络配置后及时更新OpenStack相关配置文件