1、故障概述
刚刚搭建的一套19.25 RAC环境，GI集群和数据库软件的安装过程都非常顺利，没有任何异常。但DBCA后，数据库实例只能正常启动其中的一个实例，另外一个数据库实例无法启动，提示被当前已经存活的实例给驱逐掉了。本文主要记录相关的处理过程。
2、故障分析及处理
2.1 dbca后的集群资源状态。
[root@19crac1 ~]# crsctl status resource -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
ONLINE ONLINE 19crac1 STABLE
ONLINE ONLINE 19crac2 STABLE
ora.chad
ONLINE ONLINE 19crac1 STABLE
ONLINE ONLINE 19crac2 STABLE
ora.net1.network
ONLINE ONLINE 19crac1 STABLE
ONLINE ONLINE 19crac2 STABLE
ora.ons
ONLINE ONLINE 19crac1 STABLE
ONLINE ONLINE 19crac2 STABLE
ora.proxy_advm
OFFLINE OFFLINE 19crac1 STABLE
OFFLINE OFFLINE 19crac2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.19crac1.vip
1 ONLINE ONLINE 19crac1 STABLE
ora.19crac2.vip
1 ONLINE ONLINE 19crac2 STABLE
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
1 ONLINE ONLINE 19crac1 STABLE
2 ONLINE ONLINE 19crac2 STABLE
ora.DATA.dg(ora.asmgroup)
1 ONLINE ONLINE 19crac1 STABLE
2 ONLINE ONLINE 19crac2 STABLE
ora.GRID.dg(ora.asmgroup)
1 ONLINE ONLINE 19crac1 STABLE
2 ONLINE ONLINE 19crac2 STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE 19crac2 STABLE
ora.asm(ora.asmgroup)
1 ONLINE ONLINE 19crac1 Started,STABLE
2 ONLINE ONLINE 19crac2 Started,STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
1 ONLINE ONLINE 19crac1 STABLE
2 ONLINE ONLINE 19crac2 STABLE
ora.cdb.db
1 ONLINE ONLINE 19crac1 Open,HOME=/u01/app/o
racle/product/19.0.0
/dbhome_1,STABLE
2 ONLINE INTERMEDIATE 19crac2 Instance Starting,ST
ABLE
ora.cvu
1 ONLINE ONLINE 19crac2 STABLE
ora.qosmserver
1 ONLINE ONLINE 19crac2 STABLE
ora.scan1.vip
1 ONLINE ONLINE 19crac2 STABLE
--------------------------------------------------------------------------------
[root@19crac1 ~]#
在dbca的过程中，就感觉到已经不太正常。正常情况下，dbca应该很快就能完成，但是，这次dbca的过程，差不多花了一个多小时。从集群资源的状态也可以看出，该实例一直处于Starting状态，最终 会启动失败。
尝试手动关闭实例1，则实例2能够正常启动，但此时，实例1就无法正常启动。也就是：同一时刻，只能启动一个数据库实例。
2.2 查看dbca的过程中，两个节点的alert日志。
2026-06-24T08:07:40.603950+08:00
My CSS node number is 2
My CSS hostname is 19crac2
lmon registered with NM - instance number 2 (internal mem no 1)
2026-06-24T08:07:42.620645+08:00
Using default pga_aggregate_limit of 2500 MB
2026-06-24T08:08:35.472274+08:00
PMON (ospid: 15609): terminating the instance due to ORA error 481
2026-06-24T08:08:35.472384+08:00
Cause - 'Instance is being terminated due to fatal process death (pid: 22, ospid: 15708, LMON)'
2026-06-24T08:08:35.484665+08:00
Process termination requested for pid 15758 [source = unknown], [info = 233284845] [request issued by pid: 15758, uid: 502]
2026-06-24T08:08:35.540502+08:00
System state dump requested by (instance=2, osid=15609 (PMON)), summary=[abnormal instance termination]. error - 'Instance is terminating.
'
System State dumped to trace file /u01/app/oracle/diag/rdbms/cdb/cdb2/trace/cdb2_diag_15682.trc
2026-06-24T08:08:35.861448+08:00
Dumping diagnostic data in directory=[cdmp_20260624080835], requested by (instance=2, osid=15609 (PMON)), summary=[abnormal instance termination].
2026-06-24T08:08:37.046583+08:00
Instance terminated by PMON, pid = 15609
2026-06-24T08:08:48.163308+08:00
2026-06-24T08:14:55.187683+08:00
IPC Send timeout to 1.1 inc 6 for msg type 65521 from opid 29
2026-06-24T08:14:55.188138+08:00
Communications reconfiguration: instance_number 1 from RS00 (ospid: 17287_17310)
2026-06-24T08:15:51.031691+08:00
LMON received an instance eviction notification from instance 1
The instance eviction reason is 0x2
The instance eviction map is 2
2026-06-24T08:15:57.031690+08:00
PMON (ospid: 17107): terminating the instance due to ORA error 481
2026-06-24T08:15:57.031928+08:00
Cause - 'Instance is being terminated due to fatal process death (pid: 22, ospid: 17281, LMON)'
2026-06-24T08:15:58.130024+08:00
System state dump requested by (instance=2, osid=17107 (PMON)), summary=[abnormal instance termination]. error - 'Instance is terminating.
'
System State dumped to trace file /u01/app/oracle/diag/rdbms/cdb/cdb2/trace/cdb2_diag_17240.trc
2026-06-24T08:15:58.475497+08:00
Dumping diagnostic data in directory=[cdmp_20260624081557], requested by (instance=2, osid=17107 (PMON)), summary=[abnormal instance termination].
2026-06-24T08:15:59.720450+08:00
Instance terminated by PMON, pid = 17107
从数据库的alert日志可以看出，节点间的通信超时，然后节点1将节点2的实例进行了驱逐。 以上的日志大量重复出现，这也就是为什么dbca花费的时间明显高于正常情况的原因。
2.3 节点间的通信超时，无非就是心跳网络异常、主机资源使用过载、BUG之类的原因。检查了心跳网络、主机资源使用率，没有发现异常，只能去MOS搜搜，看有没有类似的BUG，结果搜索到了《RAC DB creation fails with error - LMON (ospid XXXX) terminating the instance due to ORA error 481(KB103507)》这篇文章，文章中的描述与当前遇到的情况比较相似。文章建议将心跳网卡的rp_filter 设置成2。(net.ipv4.conf.<PRIVATE_INETRFACE1>.rp_filter = 2)
2.4 检查当前环境的内核参数设置。
修改前：
[root@19crac1 ~]# oifcfg getif
enp0s3 192.168.56.0 global public
enp0s8 10.0.0.0 global cluster_interconnect,asm
enp0s9 10.0.0.0 global cluster_interconnect,asm
[root@19crac1 ~]#
[root@19crac1 ~]# grep filter /etc/sysctl.conf
net.ipv4.conf.enp0s9.rp_filter = 0
net.ipv4.conf.enp0s8.rp_filter = 0
net.ipv4.conf.enp0s3.rp_filter = 1
[root@19crac1 ~]#
此时，仍然不相信是这个参数导致的故障。因为官方的很多文档，都是推荐将心跳网卡的该参数值设置成0或者2。
Interconnect packets may potentially be blocked/discarded. This can cause various failures, such as: unpublished Bug 10245620, "LNX64-112020-GIBONE-CMP: CAN'T START MORE THAN ONE INST DUE TO IMR RECONF HUNG"
Due to the above Linux bug fix, such “32+/multi-interconnect” systems must set "rp_filter" to the value of 0 or 2 on the private interconnect NICs. This will either disable (0) or relax (2) the filtering and allow RAC to function correctly. It is not considered unsafe to disable or relax this filtering since the private interconnect is supposed to be private/isolated network.
The rp_filter parameter can be set globally on all NICs, or specific on each NIC. The maximum (between the "all" and the specific interface) value is taken. The values are defined as follows:
rp_filter - INTEGER
• 0 - No source validation.
• 1 - Strict mode as defined in RFC3704 Strict Reverse Path Each incoming packet is tested against the FIB and if the interface is not the best reverse path the packet check will fail. By default failed packets are discarded.
• 2 - Loose mode as defined in RFC3704 Loose Reverse Path Each incoming packet's source address is also tested against the FIB and if the source address is not reachable via any interface the packet check will fail.
从官方文档可以看出，0表示不进行来源验证；2表示验证宽松。
但MOS上的一些案例实践来看，都是将心跳网卡的该参数值设置成2。
2.5 继续搜索MOS，没有找到更加匹配的案例。抱着试试的态度，将rp_filter的值从0修改为2。
修改生效后：
[root@19crac1 ~]# grep filter /etc/sysctl.conf
net.ipv4.conf.enp0s9.rp_filter = 2
net.ipv4.conf.enp0s8.rp_filter = 2
net.ipv4.conf.enp0s3.rp_filter = 1
[root@19crac1 ~]#
没想到，修改后，所有的实例正常启动。原来就是因为这个参数导致心跳网络异常。
[root@19crac1 ~]# crsctl status resource -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
ONLINE ONLINE 19crac1 STABLE
ONLINE ONLINE 19crac2 STABLE
ora.chad
ONLINE ONLINE 19crac1 STABLE
ONLINE ONLINE 19crac2 STABLE
ora.net1.network
ONLINE ONLINE 19crac1 STABLE
ONLINE ONLINE 19crac2 STABLE
ora.ons
ONLINE ONLINE 19crac1 STABLE
ONLINE ONLINE 19crac2 STABLE
ora.proxy_advm
OFFLINE OFFLINE 19crac1 STABLE
OFFLINE OFFLINE 19crac2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.19crac1.vip
1 ONLINE ONLINE 19crac1 STABLE
ora.19crac2.vip
1 ONLINE ONLINE 19crac2 STABLE
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
1 ONLINE ONLINE 19crac1 STABLE
2 ONLINE ONLINE 19crac2 STABLE
ora.DATA.dg(ora.asmgroup)
1 ONLINE ONLINE 19crac1 STABLE
2 ONLINE ONLINE 19crac2 STABLE
ora.GRID.dg(ora.asmgroup)
1 ONLINE ONLINE 19crac1 STABLE
2 ONLINE ONLINE 19crac2 STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE 19crac2 STABLE
ora.asm(ora.asmgroup)
1 ONLINE ONLINE 19crac1 Started,STABLE
2 ONLINE ONLINE 19crac2 Started,STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
1 ONLINE ONLINE 19crac1 STABLE
2 ONLINE ONLINE 19crac2 STABLE
ora.cdb.db
1 ONLINE ONLINE 19crac1 Open,HOME=/u01/app/o
racle/product/19.0.0
/dbhome_1,STABLE
2 ONLINE ONLINE 19crac2 Open,HOME=/u01/app/o
racle/product/19.0.0
/dbhome_1,STABLE
ora.cvu
1 ONLINE ONLINE 19crac2 STABLE
ora.qosmserver
1 ONLINE ONLINE 19crac2 STABLE
ora.scan1.vip
1 ONLINE ONLINE 19crac2 STABLE
--------------------------------------------------------------------------------
[root@19crac1 ~]#
3、总结
多心跳网卡的场景，在内核参数中，需要手动将心跳网卡的rp_filter的值设置为2。
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。