1、故障概述

客户的一台Exalogic，ZFS存储的一个机头出现了主板故障。为了保险起见，客户选择了停止业务，停止Exalogic上所有的虚拟机，以及虚拟环境控制台，最后再进行主板更换。 当天晚上，硬件工程师更换完主板，ZFS存储恢复正常，但在启动Exalogic虚拟环境时，问题出现了，有两台虚拟机的网络无法正常启动。

2、故障分析及处理

2.1 查看Exalogic中有问题的虚拟机的启动过程，如图所示：

 从启动日志可以看出，eth646_1.252 和 eth1286_2.252这两个网口设备有问题，提示有不同的MAC地址。

2.2  想了下，Linux主机的/etc/udev/rules.d/目录下，一些规则文件，例如：70-persistent-net.rules，有可能会记录网络设备的MAC地址，于是，让现场人员查看该目录下的文件。 一会后，现场人员反馈该目录下的文件中，没有MAC地址信息。

2.3 此时，又想到另外一个配置文件，可能会记录MAC地址，那就是虚拟机的配置文件，例如：vm.cfg。 于是，让现场人员核对虚拟机配置文件中的MAC地址与虚拟机网络的MAC地址是否一致？ 一会后，现场人员反馈虚拟机配置文件中的MAC地址与虚拟机网口配置文件中的MAC地址相同。

2.4 没办法，只能让现场人员把/etc/sysconfig/network-scripts整个目录打包，同时把 ifconfig -a输出全部发过来。分析发现，ifconfig -a输出中就不存在eth646_1.252 和 eth1286_2.252这两个网口设备。

2.5 该Exalogic虚拟环境，有十几台虚拟机，只有两台虚拟机的网络出现问题，其他的虚拟机的网络都正常。对比发现：

正常的虚拟机：
# mlx4_vnic_info -l
eth1286_2.252
eth646_1.252

故障的虚拟机：
# mlx4_vnic_infc -l
Couldn't find any vNic interfaces

从这可以看出，报错信息“Device has different MAC address than expected” 其实是表面现象，而故障的真实原因是故障虚拟机没有获取到虚拟的网口。

2.6 为什么这两台虚拟机无法获取虚拟的网口？尝试重新建立一台虚拟机，看能否获取到虚拟的网口。结果新建虚拟机也报错，日志如下：

03/07/2026 01:06:57 AM GMT INFO Invoking the prepare method of this target. (15018)
03/07/2026 01:06:57 AM GMT INFO Task is posted to the Proxy Controller (ecu-pc-IPoIB-vserver-shared-storage-primary-6d589e9f), from the Enterprise Controller. (15011)
03/07/2026 01:07:00 AM GMT INFO Task is received by the Proxy Controller. (15012)
03/07/2026 01:07:00 AM GMT INFO Exposing internal fabric ports ...
03/07/2026 01:07:00 AM GMT INFO Exposing internal fabric ports with mac address: ["00:14:4F:F9:DC:35"]
03/07/2026 01:07:00 AM GMT INFO Exposing internal fabric ports with temporary mac address: null
03/07/2026 01:07:00 AM GMT INFO Exposing internal fabric ports with temporary macaddress: ["00:14:4F:F9:DC:35"]
03/07/2026 01:07:00 AM GMT INFO InfiniBand switch to PortGUID mapping: {"com.sun.hss.domain:name=00-E0-4B-47-AC-F1,type=Switch":"[\"00:14:4F:F9:DC:35\"]"}
03/07/2026 01:07:00 AM GMT INFO virtual PortGUID to mac address mapping string : null
03/07/2026 01:07:00 AM GMT INFO virtual PortGUID to mac address map: null
03/07/2026 01:07:00 AM GMT INFO virtual PortGUIDs: [0x0f4a65783dc5a97c]
03/07/2026 01:07:00 AM GMT INFO Exposing internal fabric ports: [0x0f4a65783dc5a97c], using MAC addresses: [00:14:4F:F9:DC:35], using pKey: 0008, using vLan: 252
03/07/2026 01:07:00 AM GMT INFO Normalized port: 0x0f4a65783dc5a97c, to: 0f4a65783dc5a97c
03/07/2026 01:07:00 AM GMT INFO Starting eoib vnic creation ...
03/07/2026 01:07:00 AM GMT ERROR Unable to expose internal fabric ports on switch: com.sun.hss.domain:name=00-E0-4B-47-AC-F1,type=Switch. Reason: None of the InfiniBand Gateway Switch ports are UP
03/07/2026 01:07:00 AM GMT ERROR Could not perform exposeInternalFabricPorts task for com.sun.hss.domain:name=00-E0-4B-47-AC-F1,type=Switch. Reason: None of the InfiniBand Gateway Switch ports are UP
03/07/2026 01:07:01 AM GMT INFO Posting this task for rollback, as per failure policy
03/07/2026 01:07:01 AM GMT INFO Task is posted to the Proxy Controller (ecu-pc-IPoIB-vserver-shared-storage-primary-6d589e9f), from the Enterprise Controller. (15011)
03/07/2026 01:07:05 AM GMT INFO Task is received by the Proxy Controller. (15012)
03/07/2026 01:07:05 AM GMT INFO Performing rollback ...
03/07/2026 01:07:05 AM GMT INFO Un-exposing internal fabric ports ...
03/07/2026 01:07:05 AM GMT INFO Un-exposing 1 internal fabric ports.
03/07/2026 01:07:05 AM GMT ERROR Unable to un-expose internal fabric ports on switch: com.sun.hss.domain:name=00-E0-4B-47-AC-F1,type=Switch. Reason: Unable to get vnicId and connector for EoIB VNIC 00:14:4F:F9:DC:35
03/07/2026 01:07:05 AM GMT INFO Failed to un-expose internal fabric ports.
03/07/2026 01:07:05 AM GMT ERROR could not perform UnExposeInternalFabricPorts task for com.sun.hss.domain:name=00-E0-4B-47-AC-F1,type=Switch; reason: Unable to get vnicId and connector for EoIB VNIC 00:14:4F:F9:DC:35

从报错日志可以看出，新建虚拟机也提示无法获取EoIB VNIC，而这个EoIB VNIC是从IB交换机上虚拟出来的，问题可能在IB交换机层面。

2.7 检查IB交换机配置。

# showvnics
ID STATE FLG IOA_GUID NODE IID MAC VLN PKEY GW
--- -------- --- ----------------------- -------------------------------- ---- ----------------- --- ------ --------
0 UP N 13C71C5D8166C209 exl1isuzucn01 EL-C 10.50.250.10 0000 00:21:F6:79:22:E0 251 0x8007 0A-ETH-1
14 UP N D9A78493C82B3224 exl1isuzucn02 EL-C 10.50.250.10 0000 00:14:4F:F8:67:FF 252 0x8008 0A-ETH-1
23 UP N EB3C73A36EAE123C exl1isuzucn02 EL-C 10.50.250.10 0000 00:14:4F:FA:02:AE 252 0x8008 0A-ETH-1
19 UP N 0F46851055327464 exl1isuzucn02 EL-C 10.50.250.10 0000 00:14:4F:F8:21:96 252 0x8008 0A-ETH-1
15 UP N 9549A67BA886686F exl1isuzucn01 EL-C 10.50.250.10 0000 00:14:4F:F8:E6:42 252 0x8008 0A-ETH-1
24 UP N 7D416D6A3AFC088C exl1isuzucn02 EL-C 10.50.250.10 0000 00:14:4F:FB:ED:5E 252 0x8008 0A-ETH-1
16 UP N 4F08B13AFB3C0A98 exl1isuzucn02 EL-C 10.50.250.10 0000 00:14:4F:F8:82:FB 252 0x8008 0A-ETH-1
25 UP N 8C9EB0354A679099 exl1isuzucn03 EL-C 10.50.250.10 0000 00:14:4F:F9:B9:6C 252 0x8008 0A-ETH-1
20 UP N F22C28169B13D9A6 exl1isuzucn02 EL-C 10.50.250.10 0000 00:14:4F:FA:16:98 244 0x8009 0A-ETH-1
17 UP N 6EC69EFC90285FC1 exl1isuzucn04 EL-C 10.50.250.10 0000 00:14:4F:FB:33:84 252 0x8008 0A-ETH-1
18 UP N 92377C6CE4AA6BCA exl1isuzucn01 EL-C 10.50.250.10 0000 00:14:4F:FB:02:E2 252 0x8008 0A-ETH-1

# showgwports

INTERNAL PORTS:
---------------

Device Port Portname PeerPort PortGUID LID IBState GWState
---------------------------------------------------------------------------
Bridge-0 1 Bridge-0-1 4 0x0010e038b02cc001 0x0001 Active Up
Bridge-0 2 Bridge-0-2 3 0x0010e038b02cc002 0x0002 Active Up
Bridge-1 1 Bridge-1-1 2 0x0010e038b02cc041 0x0003 Active Up
Bridge-1 2 Bridge-1-2 1 0x0010e038b02cc042 0x0004 Active Up

CONNECTOR 0A-ETH:
-----------------

Port Bridge Adminstate Link State MTU TxPause RxPause
-------------------------------------------------------------------------
0A-ETH-1 Bridge-0-2 Enabled Down Reset 9600 Global Global
0A-ETH-2 Bridge-0-2 Enabled Down Reset 9600 Global Global
0A-ETH-3 Bridge-0-1 Enabled Down Reset 9600 Global Global
0A-ETH-4 Bridge-0-1 Enabled Down Reset 9600 Global Global

CONNECTOR 1A-ETH:
-----------------

Port Bridge Adminstate Link State MTU TxPause RxPause
-------------------------------------------------------------------------
1A-ETH-1 Bridge-1-2 Enabled Down Reset 9600 Global Global
1A-ETH-2 Bridge-1-2 Enabled Down Reset 9600 Global Global
1A-ETH-3 Bridge-1-1 Enabled Down Reset 9600 Global Global
1A-ETH-4 Bridge-1-1 Enabled Down Reset 9600 Global Global

发现没，这台IB交换机的gwport全为Down状态，而正常的那台IB交换机，0A-ETH-1 这个端口是UP状态。会不会是因为晚上更换主板时，挪动机器时，不小心把IB交换机上的那根网线给弄掉了？现场人员赶紧去机房检查网线情况，果然发现那个网口的灯是灭的，那根网线没有掉，但完全松动了。把网线重新插入后，网口的LED灯恢复正常。

2.8 重新启动Exalogic虚拟机，故障消失，虚拟机的网络恢复正常。

3、故障感悟

一个小小的动作，导致了网线松动，最终花大量时间来排查故障，还好客户申请了停机窗口，否则真是酿成大故障了。所以说，再小的操作，也要谨慎，尽量申请停机窗口。