1、案例概述

在公有云环境，一套新上线不久的11gR2 RAC， 公网使用的eth0， 心跳网使用的是eth1， 客户突然需要更换心跳网的网段，并且是新增了一个eth2的网口，然后在eth2上配置了新的心跳网IP。在GI集群层面更新了心跳信息。命令如下所示：

oifcfg setif -global eth2/*.*.*.*:cluster_interconnect
oifcfg delif -global eth1/*.*.*.*:cluster_interconnect
然后删除了以前的心跳网卡。

重启GI集群时，发现集群无法启动。进一步检查发现，是因为GPNP中的心跳网卡是eth2，而公有云环境，删除了以前的eth1网卡后，后来新增的eth2网卡变成了eth1。

当时的解决办法是：用udev将eth1重新绑定为eth2，GI集群正常启动后，再次oifcfg配置心跳指向eth1，删除udev中绑定的eth2网卡信息，让系统重新识别为eth1。至此，问题解决。

除了以上的解决办法，还可以手动修改GPnP-Profile信息，将心跳网卡从eth2手动修改为eth1。

下面，将在测试环境重现这一故障，并使用gpnptool手动修改GPnP-Profile信息，解决GI集群因心跳配置异常而无法启动的问题。

2、案例重现

2.1 当前的GI集群状态，一切正常，心跳网卡使用enp0s8。

[root@11grac1 ~]# crsctl check crs

CRS-4638: Oracle High Availability Services is online

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

[root@11grac1 ~]#

[root@11grac1 ~]# oifcfg getif

enp0s3  192.168.56.0  global  public

enp0s8  10.0.0.0  global  cluster_interconnect

[root@11grac1 ~]#

[root@11grac1 ~]# ifconfig -a

enp0s3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet 192.168.56.120  netmask 255.255.255.0  broadcast 192.168.56.255

        inet6 fe80::a00:27ff:feee:b8ba  prefixlen 64  scopeid 0x20<link>

        ether 08:00:27:ee:b8:ba  txqueuelen 1000  (Ethernet)

        RX packets 1043  bytes 123291 (120.4 KiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 979  bytes 127462 (124.4 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp0s3:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet 192.168.56.121  netmask 255.255.255.0  broadcast 192.168.56.255

        ether 08:00:27:ee:b8:ba  txqueuelen 1000  (Ethernet)

enp0s3:2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet 192.168.56.124  netmask 255.255.255.0  broadcast 192.168.56.255

        ether 08:00:27:ee:b8:ba  txqueuelen 1000  (Ethernet)

enp0s8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet 10.0.0.5  netmask 255.255.255.0  broadcast 10.0.0.255

        inet6 fe80::a00:27ff:fe47:30c0  prefixlen 64  scopeid 0x20<link>

        ether 08:00:27:47:30:c0  txqueuelen 1000  (Ethernet)

        RX packets 32907  bytes 20792850 (19.8 MiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 30720  bytes 16709036 (15.9 MiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp0s8:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet 169.254.218.133  netmask 255.255.0.0  broadcast 169.254.255.255

        ether 08:00:27:47:30:c0  txqueuelen 1000  (Ethernet)

2.2 模拟故障重现，故障将心跳网卡从enp0s8修改为enp0s9。并重启GI集群。

[root@11grac1 ~]#

[root@11grac1 ~]# oifcfg setif -global enp0s9/10.0.0.0:cluster_interconnect

[root@11grac1 ~]#

[root@11grac1 ~]# oifcfg getif

enp0s3  192.168.56.0  global  public

enp0s8  10.0.0.0  global  cluster_interconnect

enp0s9  10.0.0.0  global  cluster_interconnect

[root@11grac1 ~]#

[root@11grac1 ~]# oifcfg delif -global enp0s8/10.0.0.0

[root@11grac1 ~]#

[root@11grac1 ~]# oifcfg getif

enp0s3  192.168.56.0  global  public

enp0s9  10.0.0.0  global  cluster_interconnect

[root@11grac1 ~]#

2.3 集群无法启动,CRS服务无法启动。查看crsd.log日志，可以看到Error: PROC-44: Error in network address and interface operations Network address and interface operations error [7]之类的错误日志。

[root@11grac1 ~]# crsctl check crs

CRS-4638: Oracle High Availability Services is online

CRS-4535: Cannot communicate with Cluster Ready Services

CRS-4529: Cluster Synchronization Services is online

CRS-4534: Cannot communicate with Event Manager

[root@11grac1 ~]# crsctl status resource -t -init

--------------------------------------------------------------------------------

NAME           TARGET  STATE        SERVER                   STATE_DETAILS

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.asm

      1        ONLINE  OFFLINE                               Instance Shutdown

ora.cluster_interconnect.haip

      1        ONLINE  OFFLINE

ora.crf

      1        ONLINE  ONLINE       11grac1

ora.crsd

      1        ONLINE  OFFLINE

ora.cssd

      1        ONLINE  ONLINE       11grac1

ora.cssdmonitor

      1        ONLINE  ONLINE       11grac1

ora.ctssd

      1        ONLINE  ONLINE       11grac1                  ACTIVE:0

ora.diskmon

      1        OFFLINE OFFLINE

ora.evmd

      1        ONLINE  INTERMEDIATE 11grac1

ora.gipcd

      1        ONLINE  ONLINE       11grac1

ora.gpnpd

      1        ONLINE  ONLINE       11grac1

ora.mdnsd

      1        ONLINE  ONLINE       11grac1

[root@11grac1 ~]#

 2.4 使用如下步骤手动修改GPnP-Profile信息。

2.4.1 检查gpnptool能否正常工作

[root@11grac1 11grac1]# gpnptool get

Warning: some command line parameters were defaulted. Resulting command line:

         /u01/app/11.2.0.4/grid/bin/gpnptool.bin get -o-

<?xml version="1.0" encoding="UTF-8"?><gpnp:GPnP-Profile Version="1.0" xmlns="http://www.grid-pnp.org/2005/11/gpnp-profile" xmlns:gpnp="http://www.grid-pnp.org/2005/11/gpnp-profile" xmlns:orcl="http://www.oracle.com/gpnp/2005/11/gpnp-profile" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.grid-pnp.org/2005/11/gpnp-profile gpnp-profile.xsd" ProfileSequence="16" ClusterUId="e99aa56dbdef4fcbbfd4bfe0a5962ebd" ClusterName="sdsw-cluster" PALocation=""><gpnp:Network-Profile><gpnp:HostNetwork id="gen" HostName="*"><gpnp:Network id="net1" IP="192.168.56.0" Adapter="enp0s3" Use="public"/><gpnp:Network id="net3" Adapter="enp0s9" IP="10.0.0.0" Use="cluster_interconnect"/></gpnp:HostNetwork></gpnp:Network-Profile><orcl:CSS-Profile id="css" DiscoveryString="+asm" LeaseDuration="400"/><orcl:ASM-Profile id="asm" DiscoveryString="/dev/oracleasm/disks/*" SPFile="+dg_data/spfileasm.ora"/><ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#"><ds:SignedInfo><ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/><ds:SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1"/><ds:Reference URI=""><ds:Transforms><ds:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/><ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"> <InclusiveNamespaces xmlns="http://www.w3.org/2001/10/xml-exc-c14n#" PrefixList="gpnp orcl xsi"/></ds:Transform></ds:Transforms><ds:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/><ds:DigestValue>xq+lWf4SOzpesNFRTnlvE54kYr4=</ds:DigestValue></ds:Reference></ds:SignedInfo><ds:SignatureValue>ASHBCxPloO+E/qYZCA2ykmDGWlvEr7uFDZW9qv0NTBd24g/boLzmi1cZy5iW+jd5gjFPOtbJRJ9X102j8mVj9QDXE4FNm2EJYE5IP8ZHDwitpMeLOnq37j+ISTcUSTQf+J22N8u0S+RWaaPAS+ssA0X4MhXeKeZTtNokuT7FHfE=</ds:SignatureValue></ds:Signature></gpnp:GPnP-Profile>

Success.

[root@11grac1 11grac1]#

2.4.2 备份当前的配置

mkdir /root/gpnp

export GPNPDIR=/root/gpnp

gpnptool get -o=$GPNPDIR/profile.org

cp $GPNPDIR/profile.org $GPNPDIR/p.xml

2.4.3 获取当前的序号和网络信息

[root@11grac1 gpnp]# gpnptool getpval -p=$GPNPDIR/p.xml -prf_sq -o-

16

[root@11grac1 gpnp]# gpnptool getpval -p=$GPNPDIR/p.xml -net -o-

net1 net3

[root@11grac1 gpnp]#

2.4.4 修改，将GPnP-Profile中的enp0s9修改为enp0s8

[root@11grac1 gpnp]# gpnptool edit -p=$GPNPDIR/p.xml -o=$GPNPDIR/p.xml -ovr -prf_sq=17 -net3:net_ada=enp0s8

Resulting profile written to "/root/gpnp/p.xml".

Success.

[root@11grac1 gpnp]#

2.4.5 签名

[root@11grac1 gpnp]# gpnptool sign -p=$GPNPDIR/p.xml -o=$GPNPDIR/p.xml -ovr -w=cw-fs:peer

Resulting profile written to "/root/gpnp/p.xml".

Success.

[root@11grac1 gpnp]#

2.4.6 将新的profile写回gpnpd

[root@11grac1 gpnp]# gpnptool put -p=$GPNPDIR/p.xml

Success.

[root@11grac1 gpnp]#

2.4.7 检查修改是否成功

[root@11grac1 gpnp]# gpnptool get

Warning: some command line parameters were defaulted. Resulting command line:

         /u01/app/11.2.0.4/grid/bin/gpnptool.bin get -o-

<?xml version="1.0" encoding="UTF-8"?><gpnp:GPnP-Profile Version="1.0" xmlns="http://www.grid-pnp.org/2005/11/gpnp-profile" xmlns:gpnp="http://www.grid-pnp.org/2005/11/gpnp-profile" xmlns:orcl="http://www.oracle.com/gpnp/2005/11/gpnp-profile" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.grid-pnp.org/2005/11/gpnp-profile gpnp-profile.xsd" ProfileSequence="17" ClusterUId="e99aa56dbdef4fcbbfd4bfe0a5962ebd" ClusterName="sdsw-cluster" PALocation=""><gpnp:Network-Profile><gpnp:HostNetwork id="gen" HostName="*"><gpnp:Network id="net1" IP="192.168.56.0" Adapter="enp0s3" Use="public"/><gpnp:Network id="net3" Adapter="enp0s8" IP="10.0.0.0" Use="cluster_interconnect"/></gpnp:HostNetwork></gpnp:Network-Profile><orcl:CSS-Profile id="css" DiscoveryString="+asm" LeaseDuration="400"/><orcl:ASM-Profile id="asm" DiscoveryString="/dev/oracleasm/disks/*" SPFile="+dg_data/spfileasm.ora"/><ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#"><ds:SignedInfo><ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/><ds:SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1"/><ds:Reference URI=""><ds:Transforms><ds:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/><ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"> <InclusiveNamespaces xmlns="http://www.w3.org/2001/10/xml-exc-c14n#" PrefixList="gpnp orcl xsi"/></ds:Transform></ds:Transforms><ds:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/><ds:DigestValue>TtAj6I/zw7yIK/SVF3NNSJL9FSI=</ds:DigestValue></ds:Reference></ds:SignedInfo><ds:SignatureValue>Y7sgy76Y7e+C6+RX55T2a0uKmsaDVF9PNwGM+Orm/8HDpXK4BqIERdF34YujBPeLJP3QH+2ZgiVqvaAOjaXpZ5Vvd8g/wcQpU20t4NWm+MOXR/3qlKrwu0pKtIgi1QVRyJnk7PA68luV1jfA5Q54sw2vE4GONQHD72jTJ7FmL8g=</ds:SignatureValue></ds:Signature></gpnp:GPnP-Profile>

Success.

[root@11grac1 gpnp]#

2.5 启动crsd

[root@11grac1 gpnp]#

[root@11grac1 gpnp]# crsctl start resource ora.crsd -init

CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on '11grac1'

CRS-2676: Start of 'ora.cluster_interconnect.haip' on '11grac1' succeeded

CRS-2672: Attempting to start 'ora.asm' on '11grac1'

CRS-2676: Start of 'ora.asm' on '11grac1' succeeded

CRS-2672: Attempting to start 'ora.crsd' on '11grac1'

CRS-2676: Start of 'ora.crsd' on '11grac1' succeeded

[root@11grac1 gpnp]#

[root@11grac1 gpnp]# crsctl check crs

CRS-4638: Oracle High Availability Services is online

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

[root@11grac1 gpnp]#

2.6 重新修改oifcfg

[root@11grac1 gpnp]#

[root@11grac1 gpnp]# oifcfg getif

enp0s3  192.168.56.0  global  public

enp0s8  10.0.0.0  global  cluster_interconnect

Only in OCR: enp0s9  10.0.0.0  global  cluster_interconnect

PRIF-30: Network information in OCR and GPnP profile differs

[root@11grac1 gpnp]#

[root@11grac1 gpnp]# oifcfg setif -global enp0s8/10.0.0.0:cluster_interconnect

[root@11grac1 gpnp]#

[root@11grac1 gpnp]# oifcfg delif -global enp0s9/10.0.0.0:cluster_interconnect

[root@11grac1 gpnp]#

[root@11grac1 gpnp]# oifcfg getif

enp0s3  192.168.56.0  global  public

enp0s8  10.0.0.0  global  cluster_interconnect

[root@11grac1 gpnp]#

 2.7 至此，已经成功修改了GPnP-Profile中的心跳网卡信息。