Impact of improper change of VIP or VIP hostname of an Oracle Cluster

In my last post (Need to change VIP or VIP hostname of an Oracle Cluster), I discussed how to change the VIP or VIP hostname of an Oracle cluster. I recently came across a situation in which the system administrators had changed the VIP hostname and updated the respective entries in listener.ora and tnsnames.ora. But, the next time the server was rebooted, we had all sorts of problems with the database. Listed below are some of the issues that one may come across:

1) DBCA fails at the stage where it checks for the VIPs of the nodes throwing the following error:
Virtual IP addresses not configured for cluster nodes.

2) If you try to run VIPCA and check or even re-configure the VIPs, you might get the following generic error:
PRKR-1062 : Failed to find configuration for node myrac1.mydomain.com.

Take an OCR dump to check the OCR configuration:
    $ $ORA_CRS_HOME/bin/ocrdump

Search for VIP configuration in the resultant dump file:

.
.
.
[DATABASE.NODEAPPS.myrac1.VIP]
ORATEXT : myrac1_virtual_old.mydomain.com
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_ALL_ACCESS, OTHER_PERMISSION : PROCR_READ, USER_NAME : root, GROUP_NAME : root}
.
.
.

Aha!! That's the problem there. The OCR is still configured with the old VIP hostname.

A logical conclusion would be to modify the nodeapps to point to the new VIP hostname:
    $ srvctl modify nodeaps -n myrac1 -o $ORA_CRS_HOME -A myrac1_virtual_new/255.255.255.0/eth0

PS: Take an OCR export before modifying anything. Better safe than sorry!!
    $ $ORA_CRS_HOME/bin/ocrconfig -export ocrexport.dmp

If this doesn't work as well, you might have to drop the nodeapps and re-add them:
1) First, make sure that you have the latest OCR dump

2) Stop all the cluster databases, ASM instances and nodeapps for all the nodes:

    $ srvctl stop database -d mydb

    $ srvctl stop asm -n myrac1

    $ srvctl stop asm -n myrac2

    $ srvctl stop nodeapps -n myrac1

    $ srvctl stop nodeapps -n myrac2

3) Verify that all the components are offline:

    $ crs_stat -t

4) Using NETCA, delete the listeners registered with all the nodes

5) Remove nodeapps from all the nodes:

    $ srvctl remove nodeapps -n myrac1

    $ srvctl remove nodeapps -n myrac2

PS: Use -f option to forcibly remove the nodeapps

6) Verify that the nodeapps are no longer listed by CRS:

    $ crs_stat -t

7) Verify the CRS parameter file points to the correct VIP hostname:

    $ $ORA_CRS_HOME/install/params.crs

8) Restart CRS for parameter changes to take effect.

    $ crsctl stop crs

    $ crsctl start crs

9) Execute VIPCA to create new nodeapps for all the nodes

10) Verify that the nodeapps are listed through crs_stat and that all the services are ONLINE:
  
    $ crs_stat -t

Need to change the VIP or VIP hostname of the cluster??

First of all, lets list the various network components that are involved in an Oracle Cluster. Each of the node members of the cluster should have all of the following:
1) One Public IP,
2) One Private IP (to be used by the Private Interconnect), and
3) One Virtual IP (to be used by the applications)

The Private Interconnect should not be accessible by any other server that is not part of the cluster.

When Oracle cluster is installed and configured for the first time, the network component details are stored in OCR (Oracle Cluster Registry). The OCR parameter file contents can be viewed at $ORA_CRS_HOME/install/paramfile.crs. A typical paramfile.crs file would look like:

############### OCR Parameter File #################
CRS_ORACLE_OWNER=oracle
CRS_DBA_GROUP=oinstall
CRS_VNDR_CLUSTER=false
CRS_OCR_LOCATIONS=/ocrmount/ocr.crs
CRS_CLUSTER_NAME=myrac
CRS_HOST_NAME_LIST=myrac1.mydomain.com,1,myrac2.mydomain.com,2
CRS_NODE_NAME_LIST=myrac1.mydomain.com,1,myrac2.mydomain.com,2
CRS_PRIVATE_NAME_LIST=myrac1_private.mydomain.com,1,myrac2_private.mydomain.com,2
CRS_LANGUAGE_ID='AMERICAN_AMERICA.WE8ISO8859P1'
CRS_VOTING_DISKS=/votingmount/vote.crs
CRS_NODELIST=myrac1,myrac2
CRS_NODEVIPS='myrac1/myrac1_virtual/255.255.255.0/eth0|eth1,myrac2/myrac2_virtual/255.255.255.0/eth0|eth1'
############### OCR Parameter File #################

The cluster resources like, database instances and ASM instances are dependent on the VIP. So these would have to be restarted for the change to take effect. Also, the nodeapps will have to be modified after changing the VIP. The nodeapps includes, VIP address, GSD, Listener and ONS.

To view the current VIP configuration in the cluster, execute:
    $ srvctl config nodeapps -n myrac1 -a
    VIP exists.: /myrac1_virtual/12.34.56.78/255.255.255.0/Public

Now, to change the VIP configuration, the following steps have to be followed:
1)    Stop all the databases, ASM instances and nodeapps on both the nodes:

    $ srvctl stop database -d mydb

    $ srvctl stop asm -n myrac1

    $ srvctl stop asm -n myrac2

    $ srvctl stop nodeapps -n myrac1

    $ srvctl stop nodeapps -n myrac2

2)    Verify that all the components are offline:

    $ crs_stat -t
  
    Additionally, you can also check that the VIP is no longer running by executing 'ifconfig -a'.
  
3)    Update the /etc/hosts file and tnsnames and listener.ora files to reflect the new VIP.
4)    Modify the nodeapps to reflect the new VIP (to modify nodeapps, log into the root account):

    $ srvctl modify nodeaps -n myrac1 -o $ORA_CRS_HOME -A 98.76.54.32/255.255.255.0/eth0
  
    or
  
    $ srvctl modify nodeaps -n myrac1 -o $ORA_CRS_HOME -A myrac1_virtual_new/255.255.255.0/eth0
  
5)    Verify that the configuration change is properly reflecting in OCR:

    $ srvctl config nodeapps -n myrac1 -a
    VIP exists.: /myrac1_virtual_new/98.76.54.32/255.255.255.0/Public
  
6) Start all the databases, ASM instances and nodeapps on both the nodes (in reverse order):

    $ srvctl start nodeapps -n myrac1

    $ srvctl start nodeapps -n myrac2
  
    $ srvctl start asm -n myrac1

    $ srvctl start asm -n myrac2
  
    $ srvctl start database -d mydb
  
But, what if you have changed the VIP hostname in the hosts file but have not modified the nodeapps?? You might even risk the database not starting up automatically after server reboot. The fix for such a situation could be tiring. We would look at such a situation in my next post.