Oracle CRS failure. Rebooting for cluster integrityOracle CRS failure. Rebooting for cluster integrity.

Today I faced some issue in my cluster "Oracle CRS failure. Rebooting for cluster integrity" due to this error my CRS not able to start.

DB & CRS Version: 10.2.0.4
OS Version: Red Hat Linux 4 - 64 bit

Symptoms related to this issue as they were reported to Oracle Support have been identified as (but are not necessarily limited to):

- Cluster member reboots
- CLSOMON failing with status 13
- high cpu usage of ocssd.bin

Due to this nodes got rebooted & CRS failed

When I troubleshoot this issue found some logs from OS & crs.

Operating System Log:

Dec 7 10:57:22 babuhost4 logger: Oracle clsomon failed with fatal status 137.
Dec 7 10:57:23 babuhost4 logger: Oracle CRS failure. Rebooting for cluster integrity.
Dec 7 11:02:19 babuhost4 syslogd 1.4.1: restart.
Dec 7 11:02:19 babuhost4 syslog: syslogd startup succeeded

Cluster Log:

[ CSSD]2009-12-08 11:22:35.200 [1262557536] >TRACE: clssnmRcfgMgrThread: Local Join
[ CSSD]2009-12-08 11:22:35.200 [1262557536] >WARNING: clssnmLocalJoinEvent: takeover aborted due to ALIVE node on Disk
[ CSSD]2009-12-08 11:22:35.885 [1136679264] >TRACE: clssnmReadDskHeartbeat: node(1) is down. rcfg(3) wrtcnt(5213) LATS(86809174) Disk lastSeqNo(5213)

Operating System:

Linux babuhost4 2.6.9-67.ELsmp #1 SMP Wed Nov 7 13:56:44 EST 2007 x86_64 x86_64 x86_64 GNU/Linux

CRS Health Check Failed

Checking CRS health...

Check: Health of CRS
Node Name CRS OK?
------------------------------------ ------------------------
babuhost5 yes
babuhost4 unknown
babuhost3 yes
Result: CRS health check failed.


Refer: Document ID " 731599.1". As per document this issue occur from Oracle Server - Enterprise Edition - Version: 10.2.0.4 to 11.1.0.7.

Looks glibc package version lower need to upgrade higher version.

Oracle Enterprise Linux (OEL) / RHEL 4

* Problem exists with glibc-2.3.4-2.39
* Fixed in glibc-2.3.4-2.40 and above (only version -2.41 was actually released)

Current Version:

[root@babuhost4 log]# rpm -q glibc
glibc-2.3.4-2.39
glibc-2.3.4-2.39
[root@babuhost4 log]

Feel free write your comments here...

Thanks

CRS installation Aborts With "The Operation Has Failed Unexpectedly"

CRS installation Aborts With "The Operation Has Failed Unexpectedly"

Oracle Server Enterprise Edition: 10.2.0.1
OS Verison : Windows 2003 SP2

When i try to install oracle cluster in windows 2003 i got the bellow error message.

CRS installation Aborts With "The Operation Has Failed Unexpectedly"

nodeNames = baburac1-priv, baburac2
The operation has failed unexpectedly

SEVERE: The specified nodes are not clusterable
The operation has failed unexpectedly

Cause:

Private IP listed in NIC first.

Solution:

Simply changing the NIC order and setting the public NIC first.

Step for modifying this are:

1. Open explorer
2. Right click on my network places
3. Select properties
4. From Advanced menu, select Advanced settings
5. With Adapters and bindings open, move public NIC to be first on list

After changing NIC order it’s working fine & successfully completed cluster installation in windows 2003.

Fell free to write your comments here… Thanks

Restore backup from NON-DEFAULT location

I would like to share. How to restore database backup from non-default location. Using target database control file instead of recovery catalog database trying to create database.

Using CATALOG BACKUPPIECE trying to restore backup from non-default location.

I have one test database name called "TempDb" & database version 10.2.0.4 & linux 32-bit. Now taking full database backup in TempDb under "/dbbackup/backup/TEMPDB"

[oracle@Babu TEMPDB]$ cat tempdb_backup_log

Recovery Manager: Release 10.2.0.4.0 - Production on Sat Oct 24 11:08:49 2009

Copyright (c) 1982, 2007, Oracle. All rights reserved.

connected to target database: TEMPDB (DBID=3762582214)

RMAN> run
2> {
3> allocate channel c1 type disk format '/dbbackup/backup/TEMPDB/%s_%p_%t';
4> backup database;
5> backup archivelog all;
6> release channel c1;
7> }
8>
9>
using target database control file instead of recovery catalog
allocated channel: c1
channel c1: sid=142 devtype=DISK

Starting backup at 24-OCT-09
channel c1: starting full datafile backupset
channel c1: specifying datafile(s) in backupset
input datafile fno=00001 name=/dbbackup/oradata/TempDb/system01.dbf
input datafile fno=00003 name=/dbbackup/oradata/TempDb/sysaux01.dbf
input datafile fno=00002 name=/dbbackup/oradata/TempDb/undotbs01.dbf
input datafile fno=00004 name=/dbbackup/oradata/TempDb/users01.dbf
channel c1: starting piece 1 at 24-OCT-09
channel c1: finished piece 1 at 24-OCT-09
piece handle=/dbbackup/backup/TEMPDB/10_1_701089743 tag=TAG20091024T110902 comment=NONE
channel c1: backup set complete, elapsed time: 00:01:26
Finished backup at 24-OCT-09

Starting backup at 24-OCT-09
current log archived
channel c1: starting archive log backupset
channel c1: specifying archive log(s) in backup set
input archive log thread=1 sequence=28 recid=28 stamp=701089834
channel c1: starting piece 1 at 24-OCT-09
channel c1: finished piece 1 at 24-OCT-09
piece handle=/dbbackup/backup/TEMPDB/11_1_701089839 tag=TAG20091024T111036 comment=NONE
channel c1: backup set complete, elapsed time: 00:00:02
Finished backup at 24-OCT-09

Starting Control File and SPFILE Autobackup at 24-OCT-09
piece handle=/dbbackup/backup/TEMPDB/_c-3762582214-20091024-00 comment=NONE
Finished Control File and SPFILE Autobackup at 24-OCT-09

released channel: c1

Recovery Manager complete.
[oracle@Babu TEMPDB]$ ls -lrt
total 560360
-rw-r--r-- 1 oracle oinstall 142 Oct 24 11:08 tempdb_backup.rcv
-rw-r----- 1 oracle oinstall 543621120 Oct 24 11:10 10_1_701089743
-rw-r----- 1 oracle oinstall 22456832 Oct 24 11:10 11_1_701089839
-rw-r----- 1 oracle oinstall 7143424 Oct 24 11:10 _c-3762582214-20091024-00
-rw-r--r-- 1 oracle oinstall 1844 Oct 24 11:10 tempdb_backup_log
[oracle@Babu TEMPDB]$

After taking backup under /dbbackup/backup/TEMPDB/*.* moving to different location. (New location /dbbackup/backup/CATDB/)


[oracle@Babu TEMPDB]$ mv 10_1_701089743 11_1_701089839 _c-3762582214-20091024-00
/dbbackup/backup/CATDB/

[oracle@Babu TEMPDB]$ ls -lrt
total 8
-rw-r--r-- 1 oracle oinstall 142 Oct 24 11:08 tempdb_backup.rcv
-rw-r--r-- 1 oracle oinstall 1844 Oct 24 11:10 tempdb_backup_log

[oracle@Babu TEMPDB]$ cd ../CATDB/

[oracle@Babu CATDB]$ ls -lrt
total 560352
-rw-r----- 1 oracle oinstall 543621120 Oct 24 11:10 10_1_701089743
-rw-r----- 1 oracle oinstall 22456832 Oct 24 11:10 11_1_701089839
-rw-r----- 1 oracle oinstall 7143424 Oct 24 11:10 _c-3762582214-20091024-00
[oracle@Babu CATDB]$

Trying to create "CATDB" database from different backup location using RMAN DU

[oracle@Babu dbs]$ export ORACLE_SID=CATDB

[oracle@Babu dbs]$ pwd
/u01/app/oracle/product/10.2.0/db_1/dbs

[oracle@Babu dbs]$ orapwd file=/u01/app/oracle/product/10.2.0/db_1/dbs/orapwCATDB

password=sys entries=8

[oracle@Babu dbs]$ sqlplus '/as sysdba'

SQL*Plus: Release 10.2.0.4.0 - Production on Sat Oct 24 12:37:06 2009

Copyright (c) 1982, 2007, Oracle. All Rights Reserved.

Connected to an idle instance.

SQL> exit
Disconnected


[oracle@Babu dbs]$ cd /dbbackup/oradata/
[oracle@Babu oradata]$ ls
DEVDB JAVA TempDb

[oracle@Babu oradata]$ mkdir CATDB

[oracle@Babu oradata]$ cd /u01/app/oracle/product/10.2.0/db_1/admin/

[oracle@Babu admin]$ ls
TempDb TEMPDB

[oracle@Babu admin]$ cp -R T
TempDb/ TEMPDB/

[oracle@Babu admin]$ cp -R TEMPDB CATDB

[oracle@Babu admin]$

[oracle@Babu admin]$ ls
CATDB TempDb TEMPDB

[oracle@Babu admin]$ cd CATDB/

[oracle@Babu CATDB]$ ls
adump bdump cdump dpdump pfile udump

[oracle@Babu CATDB]$ cd bdump/
[oracle@Babu bdump]$ ls
alert_TEMPDB.log

[oracle@Babu bdump]$ rm alert_TEMPDB.log

[oracle@Babu bdump]$ export ORACLE_SID=CATDB
[oracle@Babu bdump]$
[oracle@Babu bdump]$ sqlplus '/as sysdba'

SQL*Plus: Release 10.2.0.4.0 - Production on Sat Oct 24 12:38:07 2009

Copyright (c) 1982, 2007, Oracle. All Rights Reserved.

Connected to an idle instance.

SQL> startup nomount
ORACLE instance started.

Total System Global Area 209715200 bytes
Fixed Size 1266608 bytes
Variable Size 117443664 bytes
Database Buffers 83886080 bytes
Redo Buffers 7118848 bytes
SQL>
SQL> exit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
[oracle@Babu bdump]$


[oracle@Babu CATDB]$ rman target=sys/sys@tempdb auxiliary / cmdfile=duplicate_catdb.sql

log=duplicate_catdb_log
RMAN> 2> 3> 4> 5> 6> 7> 8> 9> 10> [oracle@Babu CATDB]$

[oracle@Babu CATDB]$ cat duplicate_catdb_log

Recovery Manager: Release 10.2.0.4.0 - Production on Sat Oct 24 12:49:59 2009

Copyright (c) 1982, 2007, Oracle. All rights reserved.

connected to target database: TEMPDB (DBID=3762582214)
connected to auxiliary database: CATDB (not mounted)

RMAN> run
2> {
3> allocate auxiliary channel c1 type disk;
4> allocate auxiliary channel c2 type disk;
5> CATALOG BACKUPPIECE '/dbbackup/backup/CATDB/_c-3762582214-20091024-00';
6> CATALOG BACKUPPIECE '/dbbackup/backup/CATDB/11_1_701089839';
7> CATALOG BACKUPPIECE '/dbbackup/backup/CATDB/10_1_701089743';
8> duplicate target database to CATDB;
9> }
10>
using target database control file instead of recovery catalog
allocated channel: c1
channel c1: sid=156 devtype=DISK

allocated channel: c2
channel c2: sid=155 devtype=DISK

cataloged backuppiece
backup piece handle=/dbbackup/backup/CATDB/_c-3762582214-20091024-00 recid=12

stamp=701095838

cataloged backuppiece
backup piece handle=/dbbackup/backup/CATDB/11_1_701089839 recid=13 stamp=701095839

cataloged backuppiece
backup piece handle=/dbbackup/backup/CATDB/10_1_701089743 recid=14 stamp=701095840

Starting Duplicate Db at 24-OCT-09

contents of Memory Script:
{
set until scn 616663;
set newname for datafile 1 to
"/dbbackup/oradata/CATDB/system01.dbf";
set newname for datafile 2 to
"/dbbackup/oradata/CATDB/undotbs01.dbf";
set newname for datafile 3 to
"/dbbackup/oradata/CATDB/sysaux01.dbf";
set newname for datafile 4 to
"/dbbackup/oradata/CATDB/users01.dbf";
restore
check readonly
clone database
;
}
executing Memory Script

executing command: SET until clause

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

Starting restore at 24-OCT-09

channel c1: starting datafile backupset restore
channel c1: specifying datafile(s) to restore from backup set
restoring datafile 00001 to /dbbackup/oradata/CATDB/system01.dbf
restoring datafile 00002 to /dbbackup/oradata/CATDB/undotbs01.dbf
restoring datafile 00003 to /dbbackup/oradata/CATDB/sysaux01.dbf
restoring datafile 00004 to /dbbackup/oradata/CATDB/users01.dbf
channel c1: reading from backup piece /dbbackup/backup/TEMPDB/10_1_701089743
channel c1: restored backup piece 1
failover to piece handle=/dbbackup/backup/CATDB/10_1_701089743 tag=TAG20091024T110902
channel c1: restore complete, elapsed time: 00:00:47
Finished restore at 24-OCT-09
sql statement: CREATE CONTROLFILE REUSE SET DATABASE "CATDB" RESETLOGS ARCHIVELOG
MAXLOGFILES 16
MAXLOGMEMBERS 3
MAXDATAFILES 100
MAXINSTANCES 8
MAXLOGHISTORY 292
LOGFILE
GROUP 1 ( '/dbbackup/oradata/CATDB/redo01.log' ) SIZE 50 M REUSE,
GROUP 2 ( '/dbbackup/oradata/CATDB/redo02.log' ) SIZE 50 M REUSE,
GROUP 3 ( '/dbbackup/oradata/CATDB/redo03.log' ) SIZE 50 M REUSE
DATAFILE
'/dbbackup/oradata/CATDB/system01.dbf'
CHARACTER SET WE8ISO8859P1


contents of Memory Script:
{
switch clone datafile all;
}
executing Memory Script

datafile 2 switched to datafile copy
input datafile copy recid=1 stamp=701095901 filename=/dbbackup/oradata/CATDB/undotbs01.dbf
datafile 3 switched to datafile copy
input datafile copy recid=2 stamp=701095901 filename=/dbbackup/oradata/CATDB/sysaux01.dbf
datafile 4 switched to datafile copy
input datafile copy recid=3 stamp=701095901 filename=/dbbackup/oradata/CATDB/users01.dbf

contents of Memory Script:
{
set until scn 616663;
recover
clone database
delete archivelog
;
}
executing Memory Script

executing command: SET until clause

Starting recover at 24-OCT-09

starting media recovery

archive log thread 1 sequence 28 is already on disk as file

/dbbackup/archivelog/TEMPDB/1_28_699301197.dbf
archive log filename=/dbbackup/archivelog/TEMPDB/1_28_699301197.dbf thread=1 sequence=28
media recovery complete, elapsed time: 00:00:03
Finished recover at 24-OCT-09

contents of Memory Script:
{
shutdown clone;
startup clone nomount ;
}
executing Memory Script

database dismounted
Oracle instance shut down

connected to auxiliary database (not started)
Oracle instance started

Total System Global Area 209715200 bytes

Fixed Size 1266608 bytes
Variable Size 117443664 bytes
Database Buffers 83886080 bytes
Redo Buffers 7118848 bytes
sql statement: CREATE CONTROLFILE REUSE SET DATABASE "CATDB" RESETLOGS ARCHIVELOG
MAXLOGFILES 16
MAXLOGMEMBERS 3
MAXDATAFILES 100
MAXINSTANCES 8
MAXLOGHISTORY 292
LOGFILE
GROUP 1 ( '/dbbackup/oradata/CATDB/redo01.log' ) SIZE 50 M REUSE,
GROUP 2 ( '/dbbackup/oradata/CATDB/redo02.log' ) SIZE 50 M REUSE,
GROUP 3 ( '/dbbackup/oradata/CATDB/redo03.log' ) SIZE 50 M REUSE
DATAFILE
'/dbbackup/oradata/CATDB/system01.dbf'
CHARACTER SET WE8ISO8859P1


contents of Memory Script:
{
set newname for tempfile 1 to
"/dbbackup/oradata/CATDB/temp01.dbf";
switch clone tempfile all;
catalog clone datafilecopy "/dbbackup/oradata/CATDB/undotbs01.dbf";
catalog clone datafilecopy "/dbbackup/oradata/CATDB/sysaux01.dbf";
catalog clone datafilecopy "/dbbackup/oradata/CATDB/users01.dbf";
switch clone datafile all;
}
executing Memory Script

executing command: SET NEWNAME

renamed temporary file 1 to /dbbackup/oradata/CATDB/temp01.dbf in control file

cataloged datafile copy
datafile copy filename=/dbbackup/oradata/CATDB/undotbs01.dbf recid=1 stamp=701095955

cataloged datafile copy
datafile copy filename=/dbbackup/oradata/CATDB/sysaux01.dbf recid=2 stamp=701095955

cataloged datafile copy
datafile copy filename=/dbbackup/oradata/CATDB/users01.dbf recid=3 stamp=701095956

datafile 2 switched to datafile copy
input datafile copy recid=1 stamp=701095955 filename=/dbbackup/oradata/CATDB/undotbs01.dbf
datafile 3 switched to datafile copy
input datafile copy recid=2 stamp=701095955 filename=/dbbackup/oradata/CATDB/sysaux01.dbf
datafile 4 switched to datafile copy
input datafile copy recid=3 stamp=701095956 filename=/dbbackup/oradata/CATDB/users01.dbf

contents of Memory Script:
{
Alter clone database open resetlogs;
}
executing Memory Script

database opened
Finished Duplicate Db at 24-OCT-09

Recovery Manager complete.
[oracle@Babu CATDB]$

[oracle@Babu dbbackup]$ . oraenv
ORACLE_SID = [CATDB] ?
[oracle@Babu dbbackup]$
[oracle@Babu dbbackup]$ sqlplus '/as sysdba'

SQL*Plus: Release 10.2.0.4.0 - Production on Sat Oct 24 14:04:39 2009

Copyright (c) 1982, 2007, Oracle. All Rights Reserved.


Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> select name from v$database;

NAME
---------
CATDB

SQL> select file_name from dba_data_files;

FILE_NAME
--------------------------------------------------------------------------------
/dbbackup/oradata/CATDB/system01.dbf
/dbbackup/oradata/CATDB/undotbs01.dbf
/dbbackup/oradata/CATDB/sysaux01.dbf
/dbbackup/oradata/CATDB/users01.dbf

Successfully created/restored database from different backup location. Feel free write your comments here...

An unexpected error has been detected by HotSpot Virtual Machine

When i Try to open "netca" in my guest vmware operating system i got the below error message.

netca

Oracle Net Services Configuration:
#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# SIGSEGV (0xb) at pc=0x00a1c303, pid=31162, tid=4160608480
#
# Java VM: Java HotSpot(TM) Client VM (1.4.2_14-b05 mixed mode)
# Problematic frame:
# C [libc.so.6+0x69303] index+0x63
#
# An error report file with more information is saved as hs_err_pid31162.log
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp

Work Around:

Basically this error comes hostname/ip does't exist in /etc/hosts files.

[oracle@Babu admin]$ cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
192.168.200.49 Babu ------------> Newly added this entry.

Thanks.

Fell free write your comments here.....

PRKN-1011 : Failed to retrieve value for "local_only" under registry key

During oracle rac database installation. I got the below error details from oracle inventory log

Applies to

Windows 2000 & 2003

envp[0]:path=C:\DOCUME~1\oracle\LOCALS~1\Temp\OraInstall2009-08-31_12-55-30AM\oui\lib\win32
Caught Cluster ExceptionPRKN-1011 : Failed to retrieve value for "local_only" under registry key "HKEY_LOCAL_MACHINE\Software\Oracle\Ocr" on node "BabuRAC1-priv", The system cannot read from the specified device.

[PRKN-1011 : Failed to retrieve value for "local_only" under registry key "HKEY_LOCAL_MACHINE\Software\Oracle\Ocr" on node "BabuRAC1-priv", The system cannot read from the specified device.
]

Action Plan:

This is looks like oracle bug for more details refrer 737961.1

Feel free wirte your comments here...

Thanks

PART II: Oracle Database Upgrade & ASM Upgration

PART II: Oracle Database Upgrade & ASM Upgration

I don’t have separate ASM home. I’m keeping ASM in ORACLE_HOME itself; So Now going to continue my database & asm upgrade.

1. From oracle user you need to exeucte “runInstaller” then you will get below screen shot.
2. Please make sure your ORACLE_HOME in your path.
3. Please check any listener running from all nodes; (Else it’s will create problem)
4. Please check any database/asm instance running (If yes; pls shutdown those instance)

For example:

[oracle@linux1 bin]$ ps -ef | grep pmon

oracle 11993 22392 0 13:16 pts/2 00:00:00 grep pmon

[oracle@linux1 bin]$ ps -ef | grep lsn

oracle 5155 1 0 11:06 ? 00:00:08

/u01/app/oracle/product/10.2.0/db_1/bin/tnslsnr LISTENER_LINUX1 -inherit

oracle 12000 22392 0 13:16 pts/2 00:00:00 grep lsn

[oracle@linux1 bin]$

[oracle@linux1 bin]$ lsnrctl stop LISTENER_LINUX1

LSNRCTL for Linux: Version 10.2.0.1.0 - Production on 19-JUL-2009 13:16:42

Copyright (c) 1991, 2005, Oracle. All rights reserved.

Connecting to

(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=linux1-vip.gen.com)(PORT=1521)(IP=FIRST)))

The command completed successfully

[oracle@linux2 bin]$ ps -ef | grep LISTENER_LINUX2

oracle 12726 1 0 13:29 ? 00:00:01

/u01/app/oracle/product/10.2.0/db_1/bin/tnslsnr LISTENER_LINUX2 -inherit

oracle 14894 8929 0 13:36 pts/1 00:00:00 grep LISTENER_LINUX2

[oracle@linux2 bin]$ lsnrctl stop LISTENER_LINUX2

LSNRCTL for Linux: Version 10.2.0.1.0 - Production on 19-JUL-2009 13:36:35

Copyright (c) 1991, 2005, Oracle. All rights reserved.

Connecting to

(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=linux2-vip.gen.com)(PORT=1521)(IP=FIRST)))

The command completed successfully

[oracle@linux2 bin]$

[oracle@linux2 bin]$ ps -ef | grep pmon

oracle 14993 8929 9 13:36 pts/1 00:00:00 grep pmon

[oracle@linux2 bin]$ ps -ef | grep lsn

oracle 15002 8929 0 13:36 pts/1 00:00:00 grep lsn

[oracle@linux2 bin]$

Before upgrade cluster status & Listener status

[root@linux1 bin]# ./crs_stat -t

Name Type Target State Host

------------------------------------------------------------

ora.devdb.db application OFFLINE OFFLINE

ora....b1.inst application OFFLINE OFFLINE

ora....b2.inst application OFFLINE OFFLINE

ora...._taf.cs application OFFLINE OFFLINE

ora....db1.srv application OFFLINE OFFLINE

ora....db2.srv application OFFLINE OFFLINE

ora....SM1.asm application ONLINE ONLINE linux1

ora....X1.lsnr application ONLINE ONLINE linux1

ora.linux1.gsd application ONLINE ONLINE linux1

ora.linux1.ons application ONLINE ONLINE linux1

ora.linux1.vip application ONLINE ONLINE linux1

ora....SM2.asm application ONLINE ONLINE linux2

ora....X2.lsnr application ONLINE ONLINE linux2

ora.linux2.gsd application ONLINE ONLINE linux2

ora.linux2.ons application ONLINE ONLINE linux2

ora.linux2.vip application ONLINE ONLINE linux2

[oracle@linux1 bin]$ ps -ef | grep lsn

oracle 8117 1 0 03:17 ? 00:00:02

/u01/app/oracle/product/10.2.0/db_1/bin/tnslsnr LISTENER_LINUX1 -inherit

oracle 12039 10732 0 03:28 pts/3 00:00:00 grep lsn

[oracle@linux1 bin]$

[oracle@linux1 bin]$ lsnrctl stop LISTENER_LINUX1

LSNRCTL for Linux: Version 10.2.0.4.0 - Production on 25-JUL-2009 03:28:18

Copyright (c) 1991, 2007, Oracle. All rights reserved.

Connecting to

(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=linux1-vip.gen.com)(PORT=1521)(IP=FIRST)))

The command completed successfully

[oracle@linux1 bin]$

[oracle@linux1 bin]$ ps -ef | grep lsn

oracle 12085 10732 0 03:28 pts/3 00:00:00 grep lsn

[oracle@linux1 bin]$

[oracle@linux1 bin]$ ps -ef | grep pmon

oracle 8024 1 0 03:17 ? 00:00:03 asm_pmon_+ASM1

oracle 12162 10732 0 03:28 pts/3 00:00:00 grep pmon

[oracle@linux1 bin]$

[oracle@linux2 ~]$ cd /u01/app/oracle/product/10.2.0/crs/bin/

[oracle@linux2 bin]$ ps -ef | grep lsn

oracle 7134 1 0 03:19 ? 00:00:01

/u01/app/oracle/product/10.2.0/db_1/bin/tnslsnr LISTENER_LINUX2 -inherit

oracle 10678 10552 0 03:29 pts/2 00:00:00 grep lsn

[oracle@linux2 bin]$

[oracle@linux2 bin]$ lsnrctl stop LISTENER_LINUX2

LSNRCTL for Linux: Version 10.2.0.4.0 - Production on 25-JUL-2009 03:29:45

Copyright (c) 1991, 2007, Oracle. All rights reserved.

Connecting to

(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=linux2-vip.gen.com)(PORT=1521)(IP=FIRST)))

The command completed successfully

[oracle@linux2 bin]$

[oracle@linux2 bin]$ ps -ef | grep pmon

oracle 10784 1 21 03:29 ? 00:00:00 asm_pmon_+ASM2

oracle 10806 10552 0 03:30 pts/2 00:00:00 grep pmon

Step2:

This is TWO Nodes cluster; that’s why you will get your cluster nodes names list.

Step3

If you want register you CSI Nr then you can otherwise ignore this slide

Step4

This is pre-request checking steps; in case any thing wrong here you will come to know all the details.

Step5

Once successfully completed your pre-request then continue the patch installation on cluster.

Step6:

Now you need to execute below script for all nodes (Using root user)

/u01/app/oracle/product/10.2.0/db_1/root.sh

[root@linux1 bin]# /u01/app/oracle/product/10.2.0/db_1/root.sh

Running Oracle10 root.sh script...

The following environment variables are set as:

ORACLE_OWNER= oracle

ORACLE_HOME= /u01/app/oracle/product/10.2.0/db_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:

The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n)

[n]: y

Copying dbhome to /usr/local/bin ...

The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)

[n]: y

Copying oraenv to /usr/local/bin ...

The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)

[n]: y

Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root.sh script.

Now product-specific root actions will be performed.

[root@linux2 bin]# /u01/app/oracle/product/10.2.0/db_1/root.sh

Running Oracle10 root.sh script...

The following environment variables are set as:

ORACLE_OWNER= oracle

ORACLE_HOME= /u01/app/oracle/product/10.2.0/db_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:

The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n)

[n]: y

Copying dbhome to /usr/local/bin ...

The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)

[n]: y

Copying oraenv to /usr/local/bin ...

The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)

[n]: y

Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root.sh script.

Now product-specific root actions will be performed.

[oracle@linux1 bin]$ . oraenv

ORACLE_SID = [devdb1] ? devdb

[oracle@linux1 bin]$ export ORACLE_SID=devdb1

[oracle@linux1 bin]$ sqlplus '/as sysdba'

SQL*Plus: Release 10.2.0.4.0 - Production on Sat Jul 25 03:38:20 2009

Copyright (c) 1982, 2007, Oracle. All Rights Reserved.

Connected to an idle instance.

SQL> startup nomount

ORACLE instance started.

Total System Global Area 285212672 bytes

Fixed Size 1267044 bytes

Variable Size 100666012 bytes

Database Buffers 176160768 bytes

Redo Buffers 7118848 bytes

SQL>

SQL> show parameter clus

NAME TYPE VALUE

------------------------------------ ----------- ------------------------------

cluster_database boolean TRUE

cluster_database_instances integer 2

cluster_interconnects string

SQL> alter system set cluster_database=FALSE scope=spfile;

System altered.

SQL> shutdown

ORA-01507: database not mounted

ORACLE instance shut down.

SQL>

SQL> exit

Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Production

With the Partitioning, Real Application Clusters, OLAP, Data Mining

and Real Application Testing options

[oracle@linux1 bin]$ sqlplus '/as sysdba'

SQL*Plus: Release 10.2.0.4.0 - Production on Sat Jul 25 03:40:20 2009

Copyright (c) 1982, 2007, Oracle. All Rights Reserved.

Connected to an idle instance.

SQL> startup upgrade

ORACLE instance started.

Total System Global Area 285212672 bytes

Fixed Size 1267044 bytes

Variable Size 100666012 bytes

Database Buffers 176160768 bytes

Redo Buffers 7118848 bytes

Database mounted.

Database opened.

SQL>

SQL> show parameter cluster

NAME TYPE VALUE

------------------------------------ ----------- ------------------------------

cluster_database boolean FALSE

cluster_database_instances integer 1

cluster_interconnects string

SQL>

SQL>

SQL> spool /u01/patch.log

SQL>

SQL> @$ORACLE_HOME/rdbms/admin/catupgrd.sql

...

..

......

No errors.

No errors.

TIMESTAMP

--------------------------------------------------------------------------------

COMP_TIMESTAMP RUL 2009-07-25 11:54:47

DBUA_TIMESTAMP RUL VALID 2009-07-25 11:54:47

TIMESTAMP

--------------------------------------------------------------------------------

COMP_TIMESTAMP UPGRD_END 2009-07-25 11:54:50

.

Oracle Database 10.2 Upgrade Status Utility 07-25-2009 11:54:50

.

Component Status Version HH:MM:SS

Oracle Database Server VALID 10.2.0.4.0 01:23:50

JServer JAVA Virtual Machine VALID 10.2.0.4.0 00:15:05

Oracle XDK VALID 10.2.0.4.0 00:02:21

Oracle Database Java Packages VALID 10.2.0.4.0 00:02:39

Oracle Text VALID 10.2.0.4.0 00:01:48

Oracle XML Database VALID 10.2.0.4.0 00:11:57

Oracle Real Application Clusters VALID 10.2.0.4.0 00:00:22

Oracle Workspace Manager VALID 10.2.0.4.3 00:04:23

Oracle Data Mining VALID 10.2.0.4.0 00:03:23

OLAP Analytic Workspace VALID 10.2.0.4.0 00:03:25

OLAP Catalog VALID 10.2.0.4.0 00:08:07

Oracle OLAP API VALID 10.2.0.4.0 00:06:20

Oracle interMedia VALID 10.2.0.4.0 00:31:09

Spatial VALID 10.2.0.4.0 00:13:33

Oracle Expression Filter VALID 10.2.0.4.0 00:02:04

Oracle Enterprise Manager VALID 10.2.0.4.0 00:12:57

Oracle Rule Manager VALID 10.2.0.4.0 00:01:11

.

Total Upgrade Time: 03:24:44

DOC>#######################################################################

DOC>#######################################################################

DOC>

DOC> The above PL/SQL lists the SERVER components in the upgraded

DOC> database, along with their current version and status.

DOC>

DOC> Please review the status and version columns and look for

DOC> any errors in the spool log file. If there are errors in the spool

DOC> file, or any components are not VALID or not the current version,

DOC> consult the Oracle Database Upgrade Guide for troubleshooting

DOC> recommendations.

DOC>

DOC> Next shutdown immediate, restart for normal operation, and then

DOC> run utlrp.sql to recompile any invalid application objects.

DOC>

DOC>#######################################################################

DOC>#######################################################################

DOC>#

SQL> shutdown immediate

Database closed.

Database dismounted.

ORACLE instance shut down.

SQL>

SQL> startup

ORACLE instance started.

Total System Global Area 285212672 bytes

Fixed Size 1267044 bytes

Variable Size 146803356 bytes

Database Buffers 130023424 bytes

Redo Buffers 7118848 bytes

Database mounted.

Database opened.

SQL> @$ORACLE_HOME/rdbms/admin/utlrp.sql

TIMESTAMP

--------------------------------------------------------------------------------

COMP_TIMESTAMP UTLRP_BGN 2009-07-25 12:12:32

DOC> The following PL/SQL block invokes UTL_RECOMP to recompile invalid

.

...

.......

ERRORS DURING RECOMPILATION

---------------------------

0

PL/SQL procedure successfully completed.

SQL> show parameter cluster

NAME TYPE VALUE

------------------------------------ ----------- ------------------------------

cluster_database boolean FALSE

cluster_database_instances integer 1

cluster_interconnects string

SQL>

SQL> alter system set cluster_database=TRUE scope=spfile;

System altered.

SQL> alter system set cluster_database=TRUE scope=spfile;

System altered.

SQL> shutdown

Database closed.

Database dismounted.

ORACLE instance shut down.

SQL>

SQL> startup

ORACLE instance started.

Total System Global Area 285212672 bytes

Fixed Size 1267044 bytes

Variable Size 163580572 bytes

Database Buffers 113246208 bytes

Redo Buffers 7118848 bytes

Database mounted.

Database opened.

SQL>

SQL> show parameter cluster

NAME TYPE VALUE

------------------------------------ ----------- ------------------------------

cluster_database boolean TRUE

cluster_database_instances integer 2

cluster_interconnects string

[oracle@linux1 bin]$ ./srvctl start instance -d devdb -i devdb1

[oracle@linux1 bin]$ ./crs_stat -t

Name Type Target State Host

------------------------------------------------------------

ora.devdb.db application ONLINE ONLINE linux2

ora....b1.inst application ONLINE ONLINE linux1

ora....b2.inst application ONLINE OFFLINE

ora...._taf.cs application OFFLINE OFFLINE

ora....db1.srv application OFFLINE OFFLINE

ora....db2.srv application OFFLINE OFFLINE

ora....SM1.asm application ONLINE ONLINE linux1

ora....X1.lsnr application ONLINE ONLINE linux1

ora.linux1.gsd application ONLINE ONLINE linux1

ora.linux1.ons application ONLINE ONLINE linux1

ora.linux1.vip application ONLINE ONLINE linux1

ora....SM2.asm application ONLINE ONLINE linux2

ora....X2.lsnr application ONLINE ONLINE linux2

ora.linux2.gsd application ONLINE ONLINE linux2

ora.linux2.ons application ONLINE ONLINE linux2

ora.linux2.vip application ONLINE ONLINE linux2

[oracle@linux2 bin]$ ./srvctl start instance -d devdb -i devdb2

[oracle@linux1 bin]$ ./crs_stat -t

Name Type Target State Host

------------------------------------------------------------

ora.devdb.db application ONLINE ONLINE linux2

ora....b1.inst application ONLINE ONLINE linux1

ora....b2.inst application ONLINE ONLINE linux2

ora...._taf.cs application OFFLINE OFFLINE

ora....db1.srv application OFFLINE OFFLINE

ora....db2.srv application OFFLINE OFFLINE

ora....SM1.asm application ONLINE ONLINE linux1

ora....X1.lsnr application ONLINE ONLINE linux1

ora.linux1.gsd application ONLINE ONLINE linux1

ora.linux1.ons application ONLINE ONLINE linux1

ora.linux1.vip application ONLINE ONLINE linux1

ora....SM2.asm application ONLINE ONLINE linux2

ora....X2.lsnr application ONLINE ONLINE linux2

ora.linux2.gsd application ONLINE ONLINE linux2

ora.linux2.ons application ONLINE ONLINE linux2

ora.linux2.vip application ONLINE ONLINE linux2

[oracle@linux2 bin]$ . oraenv

ORACLE_SID = [devdb] ?

[oracle@linux2 bin]$ sqlplus "/as sysdba"

SQL*Plus: Release 10.2.0.4.0 - Production on Sat Jul 25 12:45:02 2009

Copyright (c) 1982, 2007, Oracle. All Rights Reserved.

Connected to an idle instance.

SQL> conn sys@devdb1 as sysdba

Enter password:

Connected.

SQL>

SQL> select * from v$version where rownum=1;

BANNER

----------------------------------------------------------------

Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Prod

SQL> conn sys@devdb2 as sysdba

Enter password:

Connected.

SQL> select * from v$version where rownum=1;

BANNER

----------------------------------------------------------------

Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Prod

Please feel free to post your comments here.

Thanks