Modifying the VIP in 10g Oracle Cluster Node

Hello,
I got one request to change public, private & vip address in my two nodes oracle cluster. I was thinking this may possible or it will create any other issues. 


In test instance just tried to change VIP IP address in one nodes it's working fine. Now i am looking for change public & private IP address change using Oracle Interface Configuration Tool (oifcfg) & I will post this later when i got free time. 


I would like to share the below changes with you; If you have any query or suggestion you can reach me any time.  Thanks ...


How to change VIP - IP address in Oracle Cluster Node


There are few steps to change VIP IP Address. 


1. Check current configuration
E:\oracle\product\10.2.0\crs\BIN>srvctl config nodeapps -h

Usage: srvctl config nodeapps -n [-a] [-g] [-o] [-s] [-l]

    -n           Node name

    -a                  Display VIP configuration

    -g                  Display GSD configuration

    -s                  Display ONS daemon configuration

    -l                  Display listener configuration

    -h                  Print usage

E:\oracle\product\10.2.0\crs\BIN>srvctl config nodeapps -n babu-node1 -a

VIP exists.: /babu-node1-vip/192.168.200.34/255.255.255.0/public

2. Stop your asm, db instance and other services on node 1

E:\oracle\product\10.2.0\crs\BIN>srvctl stop asm -n babu-node1


E:\oracle\product\10.2.0\crs\BIN>srvctl stop instance -d devdb -i devdb1


E:\oracle\product\10.2.0\crs\BIN>srvctl stop nodeapps -n babu-node1


E:\oracle\product\10.2.0\crs\BIN>crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora....SM1.asm application    OFFLINE   OFFLINE
ora....E1.lsnr application    OFFLINE   OFFLINE
ora....de1.gsd application    OFFLINE   OFFLINE
ora....de1.ons application    OFFLINE   OFFLINE
ora....de1.vip application    OFFLINE   OFFLINE
ora....SM2.asm application    ONLINE    ONLINE    babu-node2
ora....E2.lsnr application    ONLINE    ONLINE    babu-node2
ora....de2.gsd application    ONLINE    ONLINE    babu-node2
ora....de2.ons application    ONLINE    ONLINE    babu-node2
ora....de2.vip application    ONLINE    ONLINE    babu-node2
ora.devdb.db   application    ONLINE    ONLINE    babu-node2
ora....b1.inst application    OFFLINE   OFFLINE
ora....b2.inst application    ONLINE    ONLINE    babu-node2


3. Modify your new VIP - IP Address or VIP Host Name in /etc/hosts or C:\WINDOWS\system32\drivers\etc\hosts file ( you should update your new ip on both nodes )


4. Now modify new ip address on node1

E:\oracle\product\10.2.0\crs\BIN>srvctl config nodeapps -n babu-node1 -a

VIP exists.: /babu-node1-vip/192.168.200.34/255.255.255.0/public

E:\oracle\product\10.2.0\crs\BIN>srvctl modify nodeapps -n babu-node1 -A 192.168.200.36/255.255.255.0/"public"


E:\oracle\product\10.2.0\crs\BIN>srvctl config nodeapps -n babu-node1 -a
VIP exists.: /babu-node1-vip/192.168.200.36/255.255.255.0/public


5. Double check VIP modified or not using the above command. If everything fine then start all services & check your new VIP - IP address. 


E:\oracle\product\10.2.0\crs\BIN>srvctl start nodeapps -n babu-node1


E:\oracle\product\10.2.0\crs\BIN>srvctl start asm -n babu-node1


E:\oracle\product\10.2.0\crs\BIN>srvctl start instance -d devdb -i devdb1


E:\oracle\product\10.2.0\crs\BIN>ipconfig


Windows IP Configuration




Ethernet adapter public:


   Connection-specific DNS Suffix  . :
   IP Address. . . . . . . . . . . . : 192.168.200.36
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   IP Address. . . . . . . . . . . . : 192.168.200.33
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.200.1


Ethernet adapter private:


   Connection-specific DNS Suffix  . :
   IP Address. . . . . . . . . . . . : 10.1.1.1
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . :


E:\oracle\product\10.2.0\crs\BIN>crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora....SM1.asm application    ONLINE    ONLINE    babu-node1
ora....E1.lsnr application    ONLINE    ONLINE    babu-node1
ora....de1.gsd application    ONLINE    ONLINE    babu-node1
ora....de1.ons application    ONLINE    ONLINE    babu-node1
ora....de1.vip application    ONLINE    ONLINE    babu-node1
ora....SM2.asm application    ONLINE    ONLINE    babu-node2
ora....E2.lsnr application    ONLINE    ONLINE    babu-node2
ora....de2.gsd application    ONLINE    ONLINE    babu-node2
ora....de2.ons application    ONLINE    ONLINE    babu-node2
ora....de2.vip application    ONLINE    ONLINE    babu-node2
ora.devdb.db   application    ONLINE    ONLINE    babu-node2
ora....b1.inst application    ONLINE    ONLINE    babu-node1
ora....b2.inst application    ONLINE    ONLINE    babu-node2

PROC-26: Error while accessing the physical storage Operating System error

Hello; Sorry for delay response. Oracle 10g RAC Installation already done in Windows 2003 but not able to post.

Oracle Version: 10.2.0.1 

Os : Windows 2003 SP2

Shared Storage: RAW 

Using the above configuration I was done RAC Installation but during installation  i was facing some issues; I would share you my installation & troubleshot experience.

The below error & warning message got from Installation inventory logs and CRS,CSSD from both nodes. According to oracle document there is some firewall issues as well as need to disable UAC (User Account Control)

From my case I don't have UAC & firewall also disabled but still not able to install RAC on Windows 2003.

When I try to analyze the problem have found two things. 

1. OCR & VOT Disk not able to access from both nodes. 

2. Oracle 10.2.0.1; This is BUG we would have to install 10.2.0.4 Patch in RAC. 

How to resolve  this issue:

1. Configure your shared storage as per document & install 10.2.0.1 Cluster on both nodes.

2. When you complete installation you may face some error for post installation. ( Just ignore it )

3. Disable CRS on both nodes.

4. Change startup type to manual for oracle related services. ( Using Regedit - Try to change startup type other wise you may face other side effects ) 

[ Regedit -> HKEY_LOCAL_MACHINE -> SYSTEM -> CurrentControlSet -> Services -> OracleCSService -> Select (Start) -> Edit -> Specify 3 in Value Date ]  Repeat same step to for all oracle related services. 

5. Reboot your both nodes & install 10.2.0.4 continue post installation activity. Please let me know if you have any trouble and question :-) 

Thanks ..

Oracle Installation Inventory Log:

Command = C:\WINDOWS\system32\cmd /c call E:\oracle\product\10.2.0\crs/install/crssetup.config.bat

Successfully accumulated necessary OCR keys.

Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.

node :

node 1: babu-node1 babu-node1-priv babu-node1

node 2: babu-node2 babu-node2-priv babu-node2

Creating OCR keys for user 'oracle', privgrp ''..

Operation successful.

Now formatting voting device: \\.\votedsk1

Format of 1 voting devices complete.

Step 1:  checking status of CRS cluster

Step 2:  creating directories (E:\oracle\product\10.2.0\crs)

Step 3:  configuring OCR repository

Step 4:  configuring safe mode for CRS stack

Step 5:  Starting up CRS stack on all nodes

  babu-node1            service OracleCSService in improper PENDING state, err(0)

  babu-node2            service OracleCSService in improper PENDING state, err(997)

Configuration assistant "Oracle Clusterware Configuration Assistant" failed

CRS , CSSD and EVMD logs:

Oracle Database 10g CRS Release 10.2.0.1.0 Production Copyright 1996, 2005 Oracle.  All rights reserved.

2010-05-10 04:57:00.531: [  OCROSD][3864]utgdv:11:could not read reg value ocrmirrorconfig_loc os error= The system could not find the environment option that was entered.

2010-05-10 04:57:00.531: [  OCROSD][3864]utgdv:11:could not read reg value ocrmirrorconfig_loc os error= The system could not find the environment option that was entered.

2010-05-10 04:57:00.640: [  OCROSD][3864]utgdv:11:could not read reg value ocrmirrorconfig_loc os error= The system could not find the environment option that was entered.

2010-05-10 04:57:00.640: [  OCROSD][3864]utgdv:11:could not read reg value ocrmirrorconfig_loc os error= The system could not find the environment option that was entered.

2010-05-10 04:57:02.000: [  OCROSD][3880]utgdv:11:could not read reg value ocrmirrorconfig_loc os error= The system could not find the environment option that was entered.

Oracle Database 10g CRS Release 10.2.0.1.0 Production Copyright 1996, 2005 Oracle.  All rights reserved.

2010-05-10 06:38:06.343: [  OCROSD][2036]utgdv:11:could not read reg value ocrmirrorconfig_loc os error= The system could not find the environment option that was entered.

2010-05-10 06:38:06.343: [  OCROSD][2036]utopen:9:failed to open OCR file/disk \\.\ocrcfg,  oserror=2

2010-05-10 06:38:06.343: [  OCROSD][2036]utopen:Could not open any of the devices

2010-05-10 06:38:06.343: [  OCRRAW][2036]proprinit: Could not open raw device

2010-05-10 06:38:06.343: [ default][2036]a_init:7!: Backend init unsuccessful : [26]

2010-05-10 06:38:06.343: [ CSSCLNT][2036]clsssinit: Unable to access OCR device in OCR init.

05/11/10 13:29:43  ssmain_run_css:  boot check returned 8, looping

05/11/10 13:29:44  ssmain_run_css:  launching boot check 240 with e:\oracle\product\10.2.0\crs\bin\crsctl.exe check boot

OCR initialization failed accessing OCR device: PROC-26: Error while accessing the physical storage Operating System error [The system cannot find the file specified.

] [2]

CRS diagnostic collection tool Mandatory argument 'crshome' is missing.

diagcollection.pl fails with mandatory argument 'crshome' is missing

# Usage:

# diagcollection --collect :: collects diagnostic information

#                --clean   :: cleans up diagnostic information archive

#                --coreanalyze   :: extracts diag info from core files

Run the diagcollection.pl script as the root user to collect diagnostic information from an Oracle Clusterware installation.

The additional information provides to oracle support services to resolve internal issues. When I try to collect crs logs using diagcollection.pl got the below error message.

diagcollection.pl fails with mandatory argument 'crshome' is missing

[root@linux1 bin]# ./diagcollection.pl collect

Production Copyright 2004, 2005, Oracle.  All rights reserved

Cluster Ready Services (CRS) diagnostic collection tool

diagcollection

    --collect

             [--crs] For collecting crs diag information

             [--oh]  For collecting oracle home diag information

             [--all] Default.For collecting all diag information.

             [--core] UNIX only. Does not package core files.

             [--afterdate] UNIX only. Collects archives from the specified date.

             [--crshome] Argument that specifies the CRS Home location

             NOTE:

             1. You can also do the following

                ./diagcollection.pl --collect --crs --oh

             2. ORACLE_HOME env variable needs to be set.

     --clean        cleans up the diagnosability

                    information gathered by this script

     --coreanalyze  UNIX only. Extracts information from core files

                    and stores it in a text file

[root@linux1 bin]# pwd

/u01/app/oracle/product/10.2.0/crs/bin

 [root@linux1 bin]# ./diagcollection.pl --collect

Production Copyright 2004, 2005, Oracle.  All rights reserved

Cluster Ready Services (CRS) diagnostic collection tool

Mandatory argument 'crshome' is missing.

As per oracle document this bug from 10.2.0.4 to 11.1.0.6. Tried the below workaround it was working fine.

 [root@linux1 bin]# pwd

/u01/app/oracle/product/10.2.0/crs/bin

 [root@linux1 bin]# export ORA_CRS_HOME=/u01/app/oracle/product/10.2.0/crs

 [root@linux1 bin]# ./diagcollection.pl --collect --crs --crshome $ORA_CRS_HOME

Production Copyright 2004, 2005, Oracle.  All rights reserved

Cluster Ready Services (CRS) diagnostic collection tool

The following CRS diagnostic archives will be created in the local directory.

crsData_linux1.tar.gz -> logs,traces and cores from CRS home. Note: core files will be packaged only with the -core option.

ocrData_linux1.tar.gz -> ocrdump, ocrcheck etc

coreData_linux1.tar.gz -> contents of CRS core files in text format

Collecting crs data

/bin/tar: Removing leading `/' from member names

Collecting OCR data

Collecting information from core files

[root@linux1 bin]# ./diagcollection.pl --clean --crs --crshome $ORA_CRS_HOME

Production Copyright 2004, 2005, Oracle.  All rights reserved

Cluster Ready Services (CRS) diagnostic collection tool

Cleaning up tar and gzip files

Done

Have a great day.. :)

RMAN Incomplete Recovery

Take complete RMAN FULL backup

rman > backup database;

SQL> create table test (a int);

Table created.

SQL> begin

  2  for i in 1..50 loop

  3  insert into test values (i);

  4  end loop;

  5  commit;

  6  end;

  7  /

PL/SQL procedure successfully completed.

SQL> commit;

Commit complete.

SQL> select to_char(sysdate,'dd-mm-yy hh24:mi:ss') from dual;

TO_CHAR(SYSDATE,'

-----------------

01-05-10 16:25:00

SQL> delete from test;

50 rows deleted.

SQL> commit;

Commit complete.

SQL> select count(*) from test;

  COUNT(*)

----------

         0

RMAN> run

2> {

3> shutdown immediate

4> startup mount

5> allocate channel c1 type disk;

6> allocate channel c2 type disk;

7> set until time "to_date('01-05-2010 16:25:00','DD-MM-YYYY HH24:MI:SS')";

8> restore database;

9> recover database;

10> }

database dismounted

Oracle instance shut down

connected to target database (not started)

Oracle instance started

database mounted

Total System Global Area     209235968 bytes

Fixed Size                     1332188 bytes

Variable Size                138415140 bytes

Database Buffers              62914560 bytes

Redo Buffers                   6574080 bytes

allocated channel: c1

channel c1: SID=154 device type=DISK

allocated channel: c2

channel c2: SID=153 device type=DISK

executing command: SET until clause

Starting restore at 01-MAY-10

channel c1: starting datafile backup set restore

channel c1: specifying datafile(s) to restore from backup set

channel c1: restoring datafile 00003 to F:\ORACLE\PRODUCT\11.0.1\ORADATA\ORCL\DATA\UNDOTBS01.DBF

channel c1: restoring datafile 00004 to F:\ORACLE\PRODUCT\11.0.1\ORADATA\ORCL\DATA\USERS01.DBF

channel c1: reading from backup piece I:\BACKUP\ORCL\ORCL_20100501_0ILCJKUR_S18_P1

channel c2: starting datafile backup set restore

channel c2: specifying datafile(s) to restore from backup set

channel c2: restoring datafile 00001 to F:\ORACLE\PRODUCT\11.0.1\ORADATA\ORCL\DATA\SYSTEM01.DBF

channel c2: restoring datafile 00002 to F:\ORACLE\PRODUCT\11.0.1\ORADATA\ORCL\DATA\SYSAUX01.DBF

channel c2: restoring datafile 00005 to F:\ORACLE\PRODUCT\11.0.1\ORADATA\ORCL\DATA\TEST010_2.DBF

channel c2: reading from backup piece I:\BACKUP\ORCL\ORCL_20100501_0HLCJKUQ_S17_P1

channel c1: piece handle=I:\BACKUP\ORCL\ORCL_20100501_0ILCJKUR_S18_P1 tag=TAG20100501T162042

channel c1: restored backup piece 1

channel c1: restore complete, elapsed time: 00:00:45

channel c2: piece handle=I:\BACKUP\ORCL\ORCL_20100501_0HLCJKUQ_S17_P1 tag=TAG20100501T162042

channel c2: restored backup piece 1

channel c2: restore complete, elapsed time: 00:01:05

Finished restore at 01-MAY-10

Starting recover at 01-MAY-10

starting media recovery

media recovery complete, elapsed time: 00:00:02

Finished recover at 01-MAY-10

released channel: c1

released channel: c2

RMAN> alter database open resetlogs;

database opened

RMAN>

SQL> select count(*) from test;

  COUNT(*)

----------

        50

For recover single database objects not require go with RMAN. According to oracle document we should go with oracle utilities (export/import) or oracle flashback option.

For my scenario I don’t have logical backup as well as flashback option also not enabled.

Using RMAN incomplete recovery tried to recover single objects J

RMAN-06555: datafile 1 must be restored from backup created before 01-MAY-10

RMAN> recover database until time="TO_DATE('2010-05-01 1300:00','YYYY-MM-DD HH24:MI:SS')";


Starting recover at 01-MAY-10
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=150 device type=DISK
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 05/01/2010 15:19:10
RMAN-06555: datafile 1 must be restored from backup created before 01-MAY-10


The  above error got during database recover stage. Because of this i tried with different point in time


RMAN> shutdown immediate


database dismounted
Oracle instance shut down




RMAN>  run
2> {
3> startup mount
4> allocate channel c1 type disk;
5> allocate channel c2 type disk;
6> set until time="TO_DATE('2010-05-01 14:40:00','YYYY-MM-DD HH24:MI:SS')";
7> restore database;
8> recover database;
9> }


database is already started


allocated channel: c1
channel c1: SID=154 device type=DISK


allocated channel: c2
channel c2: SID=153 device type=DISK


executing command: SET until clause


Starting restore at 01-MAY-10


channel c1: starting datafile backup set restore
channel c1: specifying datafile(s) to restore from backup set
channel c1: restoring datafile 00003 to F:\ORACLE\PRODUCT\11.0.1\ORADATA\ORCL\DATA\UNDOTBS01.DBF
channel c1: restoring datafile 00004 to F:\ORACLE\PRODUCT\11.0.1\ORADATA\ORCL\DATA\USERS01.DBF
channel c1: reading from backup piece I:\BACKUP\ORCL\ORCL_20100501_04LCJA23_S4_P1
channel c2: starting datafile backup set restore
channel c2: specifying datafile(s) to restore from backup set
channel c2: restoring datafile 00001 to F:\ORACLE\PRODUCT\11.0.1\ORADATA\ORCL\DATA\SYSTEM01.DBF
channel c2: restoring datafile 00002 to F:\ORACLE\PRODUCT\11.0.1\ORADATA\ORCL\DATA\SYSAUX01.DBF
channel c2: restoring datafile 00005 to F:\ORACLE\PRODUCT\11.0.1\ORADATA\ORCL\DATA\TEST010_2.DBF
channel c2: reading from backup piece I:\BACKUP\ORCL\ORCL_20100501_03LCJA23_S3_P1
channel c1: piece handle=I:\BACKUP\ORCL\ORCL_20100501_04LCJA23_S4_P1 tag=TAG20100501T131443
channel c1: restored backup piece 1
channel c1: restore complete, elapsed time: 00:00:45
channel c2: piece handle=I:\BACKUP\ORCL\ORCL_20100501_03LCJA23_S3_P1 tag=TAG20100501T131443
channel c2: restored backup piece 1
channel c2: restore complete, elapsed time: 00:01:05
Finished restore at 01-MAY-10


Starting recover at 01-MAY-10


starting media recovery


channel c1: starting archived log restore to default destination
channel c1: restoring archived log
archived log thread=1 sequence=72
channel c1: reading from backup piece I:\BACKUP\ORCL\ORCL_ARCH_20100501_07LCJA81_S7_P1
channel c1: piece handle=I:\BACKUP\ORCL\ORCL_ARCH_20100501_07LCJA81_S7_P1 tag=TAG20100501T131550
channel c1: restored backup piece 1
channel c1: restore complete, elapsed time: 00:00:01
archived log file name=D:\ARCHIVELOG\ORCL\ORCL_ARC00072_0715935770.001 thread=1 sequence=72
unable to find archived log
archived log thread=1 sequence=73
released channel: c1
released channel: c2
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 05/01/2010 15:28:25
RMAN-06054: media recovery requesting unknown archived log for thread 1 with sequence 73 and starting SCN of 1178194

Even still i got different error message. The sequence 73 archive log file not in my archive log destination but still it's asking archive log file. 

I guess there is no way for medio recovery with this archive log. I just try to open the database with resetlog it's working fine. :)..

Share me with your feedback/suggestions. Thanks 

Oracle Clusterware daemons Enable/Disable Status

crsctl Using this command we can enable or disable oracle clusterware damemons. Run the following command to enable startup of all oracle clusterware daemons

crsctl enable crs

Run the following command to disable startup for all oracle clusterware daemons

crsctl diable crs

Usually we're enabling/disableing oracle clusterware daemons status using crsctl but no one checking wheather cluster already disabled/enabled or not ?

I just found YES we can check oracle clusterware daemons startup status in scls_scr directory from /etc/oracle

Example:

Enable:


[root@linux1 ~]# cd /u01/app/oracle/product/10.2.0/crs/bin


[root@linux1 bin]# ./crsctl enable crs
[root@linux1 bin]# cat /etc/oracle/scls_scr/linux1/root/crsstart
enable

After executing crsctl enable crs command clusterware daemons status updated in crsstart file "enable"

Disable:


[root@linux1 bin]# ./crsctl disable crs


[root@linux1 bin]# cat /etc/oracle/scls_scr/linux1/root/crsstart


disable

After executing crsctl disable crs command clusterware daemons status updated in crsstart file "disable". Have a great day to found oracle clusterware daemons status :-) ..