Rac 10gr2 Guide

eTEAM

Oracle 10g RAC Installation Guide

Document Information Document Owner:

eTeam DBA

Prepared by:

Karthik

Document Version No.:

V.01.00.00

Preparation Date:

August 11th, 2006

Reviewers Name

Position

Version History Version No. V.01.00.01

Version Date 08/11/06

ADP Confidential

Revised by

Description Initial Document Creation

Filename

Page 1 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Document Information................................................................................................... 1 Reviewers ..................................................................................................................... 1 Version History ............................................................................................................. 1 RAC Architecture Overview .......................................................................................... 4 System Requirements................................................................................................... 5 Storage Foundation........................................................................................................................................... 5 EMC Shared Storage and I/O Fencing ........................................................................................................... 5 Private and Virtual IP’s ................................................................................................................................... 5 ifconfig -a on d1sol5................................................................................................................................... 6 ifconfig -a on d1sol6................................................................................................................................... 6

UNIX Preinstallation Steps .............................................................................................................................. 7 Create Oracle User and Group .................................................................................................................. 7 Configure Secure Shell -SSH..................................................................................................................... 7 Configure rsh ............................................................................................................................................. 8 Set Environment variables ......................................................................................................................... 8 Configure Kernel Resources ...................................................................................................................... 9

Before the Installation .................................................................................................................................... 11 Pre-Install checks for 10gR2 RDBMS (10.2.x) - SUN Solaris Platforms ................................................... 11 Copying SFRAC Libraries to /opt/ORCLcluster/lib ................................................................................... 12 Creating $CRS_HOME on each node...................................................................................................... 12 Create volumes for CRS and VOTE disk on shared disk ......................................................................... 12 Modify the CVM service group (main.cf) to add CRS resources............................................................... 13 Final Checkup before installing CRS and Oracle Binaries....................................................................... 14

Installing CRS for Oracle 10g........................................................................................................................ 14 Installing the Oracle 10g Binaries ................................................................................................................. 35 After Oracle Binary Installation.................................................................................................................... 52 ADP Confidential

Page 2 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Copying and Verifying IPC and VCSMM libraries..................................................................................... 52 Verifying ODM library............................................................................................................................... 52

Adding Oracle 10g Patches 10.2.0.2 .............................................................................................................. 53 Adding patch 10.2.0.2 to CRS.................................................................................................................. 53 Unzip the file ............................................................................................................................................ 53 Set the Environment Variables................................................................................................................. 53 Stop all Oracle processes ........................................................................................................................ 53 Adding patch 10.2.0.2 to Oracle Binaries ................................................................................................. 64 Post Installation ....................................................................................................................................... 76 Mandatory Patch for 10.2.0.2................................................................................................................... 78

main.cf with CRS resources ......................................................................................................................... 118

ADP Confidential

Page 3 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

RAC Architecture Overview Oracle Real Application Clusters (RAC) allows Oracle Database to run any packaged or custom application, unchanged across a set of clustered servers. This provides the highest levels of availability and the most flexible scalability. If a clustered server fails, Oracle continues running on the remaining servers. In addition, when you need more processing power, simply add another server without taking users offline. To keep costs low, even the highest-end systems can be built out of standardized, commodity parts. Oracle Real Application Clusters provides a foundation for Oracle’s Enterprise Grid Computing Architecture. Oracle RAC technology enables a low-cost hardware platform to deliver the highest quality of service that rivals and exceeds the levels of availability achieved by the most expensive, mainframe SMP computers. By dramatically reducing administration costs and providing new levels of administration flexibility, Oracle is enabling the enterprise Grid environment. Real Application Clusters run on top of a hardware cluster. A cluster is a group of independent servers (nodes) that cooperate as a single system. The primary cluster components are processor nodes, a cluster interconnect, and a shared storage subsystem. The nodes share access to the storage subsystem and resources that manage data, but they do not physically share main memory in their respective nodes. Oracle cluster database combines the memory in the individual nodes to provide a single view of the distributed cache memory for the entire database system. A node can be made up of multiple processors. Each node has its own dedicated system memory as well as its own operating system and database instance.

ADP Confidential

Page 4 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

System Requirements Please consult with the infrastructure team on the requirements for Sun servers, EMC Storage, Cluster Interconnects, Public networks, Switch options, Memory, Swap and CPU. Moreover, check the RAC/Veritas on Sun certification matrix for RAC updates on currently supported hardware/software in Metalink. Before the installation of Oracle 10g RAC on Solaris 10, make sure following components has been installed and configured appropriately. Storage Foundation Installed Veritas Storage foundation for Oracle Real Application Cluster (SFRAC 4.1 packages with MP1) and Clustering configured on all Nodes that are part of the Cluster. EMC Shared Storage and I/O Fencing Verify that the shared disk arrays support SCSI-3 persistent reservations and I/O fencing. Three types of shared storage required 1. Data disks to store shared data, created based on the file system layout, 2. CRS disks to store OCR and VOTE volumes and Directories 3. Coordinator disks for I/O fencing, at least three-coordinator disk possibly smallest LUN’s on different disk arrays to resolve potential split-brain conditions and prevent data corruption. Private and Virtual IP’s Apart from host IP, make sure a public virtual IP exist on public NIC and a private IP address on the Private NIC. Refer to page 44 and 47 on Release Notes: VERITAS Storage Foundation 4.1 for Oracle RAC Maintenance Pack 1 Document id 280145 regarding creation of Private and Public IP address. Hostname d1sol5 d1sol6

ADP Confidential

Host IP 192.168.201.203 192.168.201.204

Virtual IP d1sol5-virt-192.168.201.193 d1sol6-virt-192.168.201.194

Private IP d1sol5-priv-10.10.10.1 d1sol6-priv10.10.10.2

Page 5 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

ifconfig -a on d1sol5

ip for Veritas Cluster

Public Virtual ip on Public NIC

Private ip on Private NIC

ifconfig -a on d1sol6

Public Virtual ip on Public NIC

Private ip on Private NIC

ADP Confidential

Page 6 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

UNIX Preinstallation Steps Create Oracle User and Group

Logon as root and Create a local group (dba and oinstall) and local user (oracle) # groupadd -g 201 dba # groupadd -g 202 oinstall # useradd -c "Oracle Software Owner" -g dba -G oinstall -u 201 -m -d /export/home/oracle -s /bin/ksh oracle Create a password for user oracle # passwd oracle

To verify the oracle account, # id oracle

Verifying that the User nobody Exists # id nobody Configure Secure Shell -SSH

From each node, logon as oracle $ mkdir ~/.ssh $ chmod 755 ~/.ssh $ /usr/bin/ssh-keygen -t rsa

Cut and paste the following line separately $ /usr/bin/ssh-keygen -t dsa

From the first node (d1sol5) ONLY, logged in as oracle (copy the local account's keys so that ssh to the local node will work): $ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys $ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

Now copy the keys to the other node (d1sol6) so that we can ssh to the remote node without being prompted for a password. $ ssh oracle@d1sol6 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

(Run each of them separately. SSH will prompt for the oracle password each time and if the commands are pasted at the same time, the other commands will be lost when the first one flushes the input buffer prior to prompting for the password.) $ ssh oracle@d1sol6 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys ADP Confidential

Page 7 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM $ $ $ $ $ $

Oracle 10g RAC Installation Guide

chmod 644 ~/.ssh/authorized_keys cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys ssh oracle@d1sol5 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys ssh oracle@d1sol5 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys chmod 644 ~/.ssh/authorized_keys

As oracle on the node where the Oracle 10g Release 2 software will be installed (d1sol5) $ exec /usr/bin/ssh-agent $SHELL $ /usr/bin/ssh-add

To Test $ ssh d1sol6 date

Should connect to the remote host without password prompt, If not refer to Metalink and Oracle Document for further information on setting up the ssh. Configure rsh

Make sure that the user Oracle can rsh between the nodes without having to use or confirm a password. Veritas Vxfentsthdw command requires “+” in /.rhosts file. If not, in the user oracle account directory /export/home/oracle, create a .rhosts file that includes a plus character(“+”) in the first line. On Node1 (d1sol5) create a sample test.txt file and copy the file to node2 (d1sol6). Logon to node2 (d1sol6) and check if the file exists. This tests both rsh and rcp services. $ $ $ $ $ $

touch test.txt rcp test.txt d1sol6:/tmp rsh d1sol6 ls -al /tmp/test.txt rm /tmp/test.txt rm test.txt

Set Environment variables

Edit .profile file in /export/home/oracle directory and add following lines, save it

ORACLE_BASE=/opt/oracle; export ORACLE_BASE ORACLE_HOME=$ORACLE_BASE/product/10g/db_1; export ORACLE_HOME CRS_HOME=$ORACLE_BASE/product/10g/crs; export CRS_HOME LD_LIBRARY_PATH_64=$ORACLE_HOME/lib:$CRS_HOME/lib; export LD_LIBRARY_PATH_64 LD_LIBRARY_PATH=$ORACLE_HOME/lib32; export LD_LIBRARY_PATH CLASSPATH=$CLASSPATH:$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/j lib:$ORACLE_HOME/network/jlib; export CLASSPATH ORACLE_TERM=xterm; export ORACLE_TERM PATH=$ORACLE_HOME/bin:/usr/ccs/bin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/lo cal/sbin:/sbin:/bin::/usr/lib/vxvm/bin:/opt/VRTSvxfs/sbin:/opt/VRTSvcs/bin:/

ADP Confidential

Page 8 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

opt/VRTS/bin:/opt/VRTSvcs/rac/bin:/opt/VRTSob/bin:/etc:/opt/EMCpower/bin/spa rcv9:/etc/emc/bin:/usr/dt/bin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/ucb:/et c:/usr/openwin/bin:.; export PATH EDITOR=vi; export EDITOR set -o vi stty istrip stty erase ^H umask 022 export umask ulimit -n 8192 >/dev/null 2>&1 ulimit -s 32768 >/dev/null 2>&1 PS1=`hostname`':$PWD[$ORACLE_SID]$ ' ULIMIT=999999999999999 export ULIMIT HISTSIZE=1000 export HISTSIZE HISTFILE=/var/userlog/`who am i|cut -f1 -d' '`.log export HISTFILE alias h=history ORAENV_ASK=NO MAIL=/usr/mail/${LOGNAME:?} Configure Kernel Resources

Prior to Solaris 10, the System V IPC resources, consisting primarily of shared memory, message queues, and semaphores, were set in the /etc/system file and reconfiguration needs a reboot. In Solaris 10 it is no longer requires changes in the /etc/system file. Instead, it uses the resource control facility. For more information, refer to “8.2 Appendix-2 Setting System V IPC kernel parameters” in Metalink Doc ID: 317257.1 Parameter

Value

set shmsys:shminfo_shmmax set shmsys:shminfo_shmmin set shmsys:shminfo_shmmni set semsys:seminfo_semmni set semsys:seminfo_semmns set semsys:seminfo_semmnu set semsys:seminfo_semopm set semsys:seminfo_semvmx set shmsys:shminfo_shmseg

18446744073709551615 1 256 0x1002 0x7631 0x7631 4096 32767 256

Solaris 10 Resource Control project.max-shm-memory N/A project.max-shm-ids project.max-sem-ids N/A Obsolete project.max-sem-ops Obsolete N/A

set semsys:seminfo_semmap set semsys:seminfo_semmsl set semsys:seminfo_semume

0x7631 4096 512

Obsolete project.max-sem-nsems Obsolete

ADP Confidential

Default Value 1/4 of physical memory 128 128 512

512

Page 9 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Before moving further, please refer to Bug No 5237047 regarding INCORRECT SYSTEM REQUIREMENTS FOR SOLARIS 10 in Metalink On each node, logon as root # # # # # #

projadd projmod projmod projmod projmod projmod

-U -s -s -s -s -s

oracle -G dba -c "OracleProject" -p 201 user.oracle -K "project.max-shm-memory=(priv,10g,deny)" user.oracle -K "project.max-shm-ids=(priv,256,deny)" user.oracle -K "project.max-sem-ids=(priv,4096,deny)" user.oracle -K "project.max-sem-ops=(priv,4096,deny)" user.oracle -K "project.max-sem-nsems=(priv,4096,deny)" user.oracle

To set UDP parameters to 65536 bytes, write a startup script in /etc/init.d # cd /etc/init.d # touch udp_rac # chmod 775 udp_rac # vi udp_rac ---------------------------Script Start------------------------------------#!/sbin/sh case "$1" in 'start') ndd -set /dev/udp udp_xmit_hiwat 65536 ndd -set /dev/udp udp_recv_hiwat 65536 ;; . 'state') ndd /dev/udp udp_xmit_hiwat ndd /dev/udp udp_recv_hiwat ;; . *) echo "Usage: $0 { start | state }" exit 1 ;; esac ---------------------------Script End--------------------------------------# cd /etc/rc3.d # ln -s /etc/init.d/udp_rac /etc/rc3.d/S86udp_rac

to view/verify the settings logon as oracle and run $ prctl $$

ADP Confidential

Page 10 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

As the srvctl utility not unable to get the right SHM related settings using prctl/projmod, Add following entries to /etc/system file and reboot the servers. For more information, Please refer to Metalink Doc id 367442.1 forceload: sys/semsys forceload: sys/shmsys set shmsys:shminfo_shmmax=18446744073709551615 set shmsys:shminfo_shmmin=1 set shmsys:shminfo_shmmni=256 set semsys:seminfo_semmni=0x1002 set semsys:seminfo_semmns=0x7631 set semsys:seminfo_semmnu=0x7631 set semsys:seminfo_semopm=4096 set semsys:seminfo_semvmx=32767 set shmsys:shminfo_shmseg=256 set semsys:seminfo_semmap=0x7631 set semsys:seminfo_semmsl=4096 set semsys:seminfo_semume=512 set set set set set set

tune_t_fsflushr=30 autoup=900 rlim_fd_max=65536 rlim_fd_cur=32768 sq_max_size=0 tcp:tcp_conn_hash_size=8192

Reboot the servers ulimit settings, following command will display the current ulimit settings: $ ulimit -a

If needed add following entries to the .profile for oracle to reset stack and nofiles values ulimit -n 4096 >/dev/null 2>&1 ulimit -s 32768 >/dev/null 2>&1

Before the Installation Pre-Install checks for 10gR2 RDBMS (10.2.x) - SUN Solaris Platforms Refer to Doc ID: 334567.1

Logon as Oracle $ cd /ora/scr/preinstall $ tar xvf sun_10gR2_hcve_rdbms_091705.tar

Set the oracle environment variables, ORACLE_HOME

$ perl validate.pl 10gR2_rdbms_solaris_hcve_091705.txt

Or validate 10gR2_rdbms_solaris_hcve_091705.txt

Please review the log files and rectify the components that failed the test. ADP Confidential

Page 11 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Copying SFRAC Libraries to /opt/ORCLcluster/lib

Logon as root and Copy SFRAC libraries to enable RAC recognize the cluster environment # cp /opt/VRTSvcs/rac/lib/libskgxn2_64.so /opt/ORCLcluster/lib/libskgxn2.so Creating $CRS_HOME on each node

CRS and Oracle Binaries will be installed locally on each server. Make sure $CRS_HOME is under $ORACLE_BASE. Make directories for ORACLE_BASE is /opt/oracle CRS_HOME is /opt/oracle/product/10g/crs ORACLE_HOME is /opt/oracle/product/10g/db_1 Create volumes for CRS and VOTE disk on shared disk

Create OCR and VOTE-disk volumes and directories on the cluster file system. Assume that emcpower13s2 is the allocated LUN, execute following commands as root on one of the node (d1sol5) # # # #

vxdg -s init crs_oradg emcpower13s2 vxassist -g crs_oradg make crs_vol 200M vxvol -g crs_oradg startall mkfs -F vxfs /dev/vx/rdsk/crs_oradg/crs_vol

On both nodes (d1sol5 and d1sol6) # # # #

mkdir mount chown chmod

/ora_crs -F vxfs -o cluster /dev/vx/dsk/crs_oradg/crs_vol /ora_crs oracle:dba /ora_crs 755 /ora_crs

On one node (d1sol5), logon as oracle $ cd /ora_crs $ mkdir CRS_OCR $ mkdir VOTE-disk

ADP Confidential

Page 12 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Modify the CVM service group (main.cf) to add CRS resources.

Refer to the Modified main.cf in the appendix Log into one of the nodes in the cluster (d1sol5) as root # # # # #

haconf -dump makero hastop -all -force cd /etc/VRTSvcs/conf/config cp main.cf main.orig vi main.cf

Include the following entries, At the beginning of the file include “PrivNIC.cf”

And in CVM service group CFSMount crs_ora_mnt ( MountPoint = "/ora_crs" BlockDevice = "/dev/vx/dsk/crs_oradg/crs_vol" ) CVMVolDg crs_voldg ( CVMDiskGroup = crs_oradg CVMVolume = { crs_vol } CVMActivation = sw ) PrivNIC ora_priv ( Device = { ipge2 = 0, ipge3 = 1 Address @d1sol5 = "10.10.10.1" Address @d1sol6 = "10.10.10.2" NetMask = "255.255.255.0" )

}

And in dependencies

crs_ora_mnt requires crs_voldg crs_ora_mnt requires vxfsckd crs_voldg requires cvm_clus

Save and Close main.cf Verify the syntax # hacf –verify /etc/VRTSvcs/conf/config

Copy main.cf to the second node (d1sol6) now start VCS in all nodes #hastart

And check the VCS status #hastatus –summ ADP Confidential

Page 13 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

All added resources should be online on both the nodes Final Checkup before installing CRS and Oracle Binaries

Logon as oracle 1. Check ssh config for local node and remote node 2. Use prctl $$ to check the semaphores values 3. Check the Oracle user profile settings 4. Get host ip, public virtual ip and it’s interface, private ip and it’s interface name. 5. Bring down the ip on public virtual interface (as root) # ifconfig ipge0:2 down 6. Create volumes and directories for CRS and VOTE disk 7. CRS resources added to the veritas cluster main.cf and restart SFRAC 8. Copied SFRAC libskgxn2_64.so 9. Check the DISPLAY environmental variable, run xclock and verify 10.

Installing CRS for Oracle 10g Make sure all required software is under /ora/scr directory. Logon to node1 (d1sol5) as Oracle $ cd /ora/scr/crs $ ./runInstaller

Use ./runInstaller –ignoreSysPrereqs to ignore the prerequisites

ADP Confidential

Page 14 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 15 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 16 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Change the Name to CRS_HOME and path to /opt/oracle/product/10g/crs

ADP Confidential

Page 17 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 18 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Click Edit and modify Virtual hostnames (d1sol5_vip to d1sol5_virt and d1sol6_vip t d1sol6_virt) to match the names in /etc/hosts

ADP Confidential

Page 19 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

ipge0 is the public nic, so click Edit and change ipge0 interface type to Public

ADP Confidential

Page 20 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 21 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 22 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 23 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 24 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Don’t click the OK button now. Run the scripts one by one on each node, configure Virtual ip’s using vipca and then comeback and click OK

ADP Confidential

Page 25 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Logon as root on node1 (d1sol5) and run # /opt/oracle/oraInventory/orainstRoot.sh and # /opt/oracle/product/10g/crs/root.sh

ADP Confidential

Page 26 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Logon as root on 2nd node (d1sol6) and run # /opt/oracle/oraInventory/orainstRoot.sh and # /opt/oracle/product/10g/crs/root.sh

ADP Confidential

Page 27 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Make sure CSS is active on both the nodes At the end of the script, we might receive following error messages saying “ipge0 is not public” Running vipca(silent) for configuring nodeapps The given interface(s), "ipge0" is not public. Public interfaces should be used to configure virtual IPs.

This is a known issues with vipca, Refer to page 60 of Release Notes: VERITAS Storage Foundation 4.1 for Oracle RAC Maintenance Pack 1 Document id 280145. please invoke vipca manually root@d1sol6 # cd /oracle/product/10g/crs/bin root@d1sol6 # ./vipca

ADP Confidential

Page 28 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 29 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 30 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Enter Virtual IP alias name, d1sol5-virt.campd.local, press tab remaining column values will be filled automatically

ADP Confidential

Page 31 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 32 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Click OK after running the scripts and vipca

ADP Confidential

Page 33 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 34 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Installing the Oracle 10g Binaries Log on as oracle $ cd /ora/scr/db $ ./runInstaller

ADP Confidential

Page 35 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 36 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Select Custom

ADP Confidential

Page 37 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Change Name to ORACLE_HOME and Path to /opt/oracle/product/10g/db_1

ADP Confidential

Page 38 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 39 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 40 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 41 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Ignore the OUI finding in the area of “noexec_user_stack for the Solaris 10 OS (Refer Metalink 377679.1) Checking for noexec_user_stack=1; found no entry.Failed <<<< Check complete. The overall result of this check is: Failed <<<<

ADP Confidential

Page 42 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 43 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 44 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 45 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 46 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Don’t click OK now

ADP Confidential

Page 47 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Logon as root on node1 (d1sol5) and run # /opt/oracle/product/10g/db_1/root.sh

ADP Confidential

Page 48 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Logon as root on node1 (d1sol6) and run #/opt/oracle/product/10g/db_1/root.sh

ADP Confidential

Page 49 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 50 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 51 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

After Oracle Binary Installation Copying and Verifying IPC and VCSMM libraries

Logon as oracle and copy $ cp /opt/VRTSvcs/rac/lib/libskgxp102_64.so $ORACLE_HOME/lib/libskgxp10.so Verifying ODM library

Login as Oracle user $ cd /opt/oracle/product/10g/db_1/lib $ mv /opt/oracle/product/10g/db_1/lib/libodm10.so /opt/oracle/product/10g/db_1/lib/libodm10.so.old $ ln -s /usr/lib/sparcv9/libodm.so libodm10.so $ ls -al /opt/oracle/product/10g/db_1/lib/libodm10.so

(makesure owner is oracle) On d1sol5

On d1sol6

ADP Confidential

Page 52 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Adding Oracle 10g Patches 10.2.0.2 (Metalink Patch # 4547817) Adding patch 10.2.0.2 to CRS

Should always upgrade Oracle Clusterware before you upgrade Oracle RAC Binaries If we are installing this patch set on an existing Oracle Real Applications Cluster (RAC) 10g release 10.2.0.1 installation, we must run Oracle Universal Installer from the same node(d1sol5) from which the Oracle Database software was installed. Unzip the file $ unzip p4547817_10202_SOLARIS64.zip Set the Environment Variables

Set ORACLE_HOME, CRS_HOME in the .profile of Oracle user. Stop all Oracle processes

Login as oracle user On d1sol5 $ srvctl stop nodeapps -n d1sol5

On d1sol6 $ srvctl stop nodeapps -n d1sol6

Logon as root Shut down the Oracle Clusterware process by entering the following command on all nodes as the root user:

ADP Confidential

Page 53 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

On d1sol5 # /etc/init.d/init.crs stop

On d1sol6 # /etc/init.d/init.crs stop

Make sure there is no Oracle processes running $ ps -ef|grep oracle|grep -v grep

Log in as the Oracle user on node1 (d1sol5) Start Oracle Universal Installer $ cd /ora/scr/patch/10.2.0.2/Disk1 $ ./runInstaller

On the Welcome screen, click Next.

ADP Confidential

Page 54 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 55 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Change the Name to CRS_HOME and Path to /opt/oracle/product/10g/crs

ADP Confidential

Page 56 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 57 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 58 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Don’t click on Exit

ADP Confidential

Page 59 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Don’t click on Exit

ADP Confidential

Page 60 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Login as root on d1sol5 and run # /opt/oracle/product/10g/crs/bin/crsctl stop crs # /opt/oracle/product/10g/crs/install/root102.sh

To Check is crs is stopped # ps -ef|grep d.bin

Refer to Metalink Doc id 358258.1 regarding crsctl stop crs, Failure 15 At Css Context Initialization When Shutting Down Crs

ADP Confidential

Page 61 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Login as root on node 2 (d1sol6) and run # /opt/oracle/product/10g/crs/bin/crsctl stop crs # /opt/oracle/product/10g/crs/install/root102.sh

ADP Confidential

Page 62 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Click Exit now, after running the scripts as root user

ADP Confidential

Page 63 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Adding patch 10.2.0.2 to Oracle Binaries

Login as Oracle User Stop all Oracle Processes On d1sol5 $ srvctl stop nodeapps -n d1sol5

On d1sol6 $ srvctl stop nodeapps -n d1sol6

Login as root User Shut down the Oracle Clusterware process by entering the following command on all nodes as the root user: On d1sol5 # /etc/init.d/init.crs stop

On d1sol6 # /etc/init.d/init.crs stop

Make sure there is no Oracle processes running # ps -ef|grep oracle|grep -v grep ADP Confidential

Page 64 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Log in as the oracle user on node1(d1sol5) Start Oracle Universal Installer $ cd /ora/scr/patch/10.2.0.2/Disk1 $ ./runInstaller

ADP Confidential

Page 65 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 66 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Enter Name as ORACLE_HOME and Path as /opt/oracle/product/10g/db_1

ADP Confidential

Page 67 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 68 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 69 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 70 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Don’t click OK now

ADP Confidential

Page 71 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Login as root on d1sol5 and run # /opt/oracle/product/10g/db_1/root.sh

ADP Confidential

Page 72 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Login as root on d1sol5 and run # /opt/oracle/product/10g/db_1/root.sh

ADP Confidential

Page 73 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 74 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 75 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Post Installation

During patchset installation, all new files and directories are created with restricted access, by default. Users or third party applications with a different group identifier from that of the database, which try to access client-side utilities or libraries in the database home, will see permission errors when trying to access these files or directories. Perform the following steps to change the permissions: Change your directory to: $ cd $ORACLE_HOME/install

Run changePerm.sh and specify the patched server Oracle home location, before accessing client-side utilities or libraries in the database home. For RAC home, we need to run this script on all the nodes. Login as Oracle On d1sol5 $ cd /opt/oracle/product/10g/db_1/install $ ./changePerm.sh

Look for errors in /tmp/changePerm_err.log and resolve it. # chown oracle:dba externaljob.ora # chmod 644 externaljob.ora ADP Confidential

Page 76 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

On d1sol6 $ cd /opt/oracle/product/10g/db_1/install $ ./changePerm.sh

ADP Confidential

Page 77 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Mandatory Patch for 10.2.0.2

After installing 10.2.0.2 apply the mandatory patch for (bug 5117016) for incorrectly located in $ORACLE_HOME/RDBMS/LIB/ For more information, Refer Metalink Doc id 5117016.8 This patch is applicable only on the server side. Ensure all the processes started out of the ORACLE_HOME are being shutdown prior to applying this patch using srvctl , on every node $ srvctl stop nodeapps -n d1sol5 $ srvctl stop nodeapps -n d1sol6

To apply the patch, unzip the PSE container file: $ unzip p5117016_10202_SOLARIS64.zip

Set your current directory to the directory where the patch is located $ cd /ora/scr/patch/5117016

Ensure that the directory containing the opatch script appears in the $PATH, then run the following command on every node $ $ORACLE_HOME/OPatch/opatch apply -local

ADP Confidential

Page 78 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

On d1sol5

ADP Confidential

Page 79 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

On d1sol6

ADP Confidential

Page 80 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Check the inventory $ $ORACLE_HOME/OPatch/opatch lsinventory

ADP Confidential

Page 81 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Oracle® Database Critical Patch Update Release 10.2.0.2 Patch 5225799 Ensure that the Oracle Database installation is the same release for which we are applying this patch. In other words, only apply the Release 10.2.0.2 CPUJul2006 patch to an Oracle Database Release 10.2.0.2. Review Section 4, "Known Issues" before proceeding with this installation. 1. Shut down the instance on one node (d1sol5) 2. Shut down all nodeapps services on the node (d1sol5) in step 1: $ srvctl stop nodeapps -n d1sol5

Apply patch on the node (d1sol5) in step 1 3. Set current directory to the directory where the patch is located and then run OPatch by entering the following command: $ cd /ora/scr/patch/cpu/5225799 $ORACLE_HOME/OPatch/opatch apply OPatch.SKIP_VERIFY=true

(We need to specify OPatch.SKIP_VERIFY=true. For more information please refer to 353150.1 - OPatch Failing Validation Phase When Archiving Really Did Not Fail) 4. If there are errors, refer to Section 4, "Known Issues". Start all nodeapps services on the node in step 1: $ srvctl start nodeapps -n d1sol5

Shut down the instance on the next node. Repeat steps 1-6 on all the nodes in the cluster. Check the opatch inventory Logon as root and start crs on both nodes # /etc/init.d/init.crs start

ADP Confidential

Page 82 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Check the inventory $ $ORACLE_HOME/OPatch/opatch lsinventory

ADP Confidential

Page 83 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Creating the Oracle Database CRS should be running

To check crs daemon running or not $CRS_HOME/bin/crs_stat Create the mount points and directories for the Oracle Database

Login as root and # # # # # # # # # # # # #

mkdir /hapdb cd /hapdb mkdir archive flashdb data1 index1 redo1 redo2 redocopy system temp mount -F vxfs -o cluster /dev/vx/dsk/ora_dg1/ora_archive_vol /hapdb/archive mount -F vxfs -o cluster /dev/vx/dsk/ora_dg1/ora_flashdb_vol /hapdb/flashdb mount -F vxfs -o cluster /dev/vx/dsk/ora_dg1/ora_data1_vol /hapdb/data1 mount -F vxfs -o cluster /dev/vx/dsk/ora_dg1/ora_index1_vol /hapdb/index1 mount -F vxfs -o cluster /dev/vx/dsk/ora_dg1/ora_redo1_vol /hapdb/redo1 mount -F vxfs -o cluster /dev/vx/dsk/ora_dg1/ora_redo2_vol /hapdb/redo2 mount -F vxfs -o cluster /dev/vx/dsk/ora_dg1/ora_redocopy_vol /hapdb/redocopy mount -F vxfs -o cluster /dev/vx/dsk/ora_dg1/ora_system_vol /hapdb/system mount -F vxfs -o cluster /dev/vx/dsk/ora_dg1/ora_temp_vol /hapdb/temp chown -R oracle:dba archive flashdb data1 index1 redo1 redo2 redocopy system temp

Verify all file system exist and owner is Oracle Default template is already created for this database. So Copy hapdb.dbt to /opt/oracle/product/10g/db_1/assistants/dbca/templates. If not select Custom or General Purpose database template Before creating the database using dbca, configure LISTENER using netca. $ netca

ADP Confidential

Page 84 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 85 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 86 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 87 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 88 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 89 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Creating 10g database using dbca

Logon as Oracle $ dbca

ADP Confidential

Page 90 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 91 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 92 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 93 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

If hapdb template is not available select Custom Database option

ADP Confidential

Page 94 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 95 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 96 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 97 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 98 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 99 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 100 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 101 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 102 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 103 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 104 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 105 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Click on All Initialization Parameters and Change the location for all dump (a/b/c/udump) init. parameters

ADP Confidential

Page 106 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 107 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 108 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 109 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 110 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 111 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 112 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 113 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

ADP Confidential

Oracle 10g RAC Installation Guide

Page 114 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

After creating the database, check spfile is in /hapdb/system/admin/pfile/ directory if not $ cd $ORACLE_HOME/dbs

Change the location of spfile in inithapdb1.ora to SPFILE='/hapdb/system/admin/pfile/spfilehapdb.ora' And move the file from /hapdb/data1/hapdb to /hapdb/system/admin/pfile/ (No issue when used a different template, not hapdb_new ) To check the status root@d1sol6 # /opt/oracle/product/10g/crs/bin/crs_stat -t Name Type Target State Host -----------------------------------------------------------ora....L5.lsnr application ONLINE ONLINE d1sol5 ora.d1sol5.gsd application ONLINE ONLINE d1sol5 ora.d1sol5.ons application ONLINE ONLINE d1sol5 ora.d1sol5.vip application ONLINE ONLINE d1sol5 ora....L6.lsnr application ONLINE ONLINE d1sol6 ora.d1sol6.gsd application ONLINE ONLINE d1sol6 ora.d1sol6.ons application ONLINE ONLINE d1sol6 ora.d1sol6.vip application ONLINE ONLINE d1sol6 ADP Confidential

Page 115 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

ora.hapdb.db application ora....b1.inst application ora....b2.inst application

ONLINE ONLINE ONLINE

ONLINE ONLINE ONLINE

d1sol5 d1sol5 d1sol6

To check the service status $ srvctl status service –d hapdb

Configure the CVM and Oracle Service Groups for Oracle 10g Log into one of the nodes in the cluster (d1sol5) as root # # # # #

haconf -dump makero hastop -all -force cd /etc/VRTSvcs/conf/config cp main.cf main.before10gdb vi main.cf

Refer to the Modified and final main.cf in the appendix Save and Close main.cf Verify the syntax # hacf –verify /etc/VRTSvcs/conf/config

Copy main.cf to the second node (d1sol6) now start VCS in all nodes #hastart

And check the VCS status #hastatus –summ

All added resources should be online on both the nodes

If voting is disk unavailable, server will be rebooted every few minutes, to avoid the reboot disable the crs # /etc/init.d/init.crs disable

Correct the issue and reenable it and reboot the server # /etc/init.d/init.crs enable # reboot

If Veritas Agent is used for Detail Monitoring (Secondary monitoring) then create user vcsmon identified by vcsmon default tablespace TOOLS temporary tablespace TEMP quota 500K on TOOLS ADP Confidential

Page 116 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

/ grant create session to vcsmon / create table vcsmon. vcstable_d1sol5 ( tstamp date ) / insert into vcsmon. vcstable_d1sol5 ( tstamp ) values ( SYSDATE ) / create table vcsmon. vcstable_d1sol6 ( tstamp date ) / insert into vcsmon. vcstable_d1sol6 ( tstamp ) values ( SYSDATE ) /

To scp and unzip the files root@D1SOL5 # scp 10gr2_cluster_sol.cpio.gz root@d1sol6:/opt/oracle/software/crs/ 10gr2_cluster_sol.cp 100% |********************************************************************************** *********| 368 MB 00:51 root@D1SOL5 # gunzip 10gr2_cluster_sol.cpio.gz root@D1SOL5 # cpio -idmv <10gr2_cluster_sol.cpio

Stopping CRS Before Stopping VCS in Oracle 10g Clusters In SFRAC clusters with Oracle 10g, stop the CRS daemons before using the hastop -all command to stop VCS. The required sequence of commands is: 1. Stop Oracle on each node: # hagrp -offline oracle_group -sys -system_name

2. Stop the CRS daemons on each node:

# /etc/init.d/init.crs stop

3. Stop VCS. From one node, type: # hastop -all

After 10.2.0.2 patchset is installed, RDBMS crashes with internal errors on remote nodes Doc id 365851.1 1. Ensure all instances are shut down 2. Manually copy $ORACLE_HOME/rdbms/lib/libknlopt.a from the local node to all remote nodes . 3. Relink Oracle on all remote nodes eg: cd $ORACLE_HOME/rdbms/lib make -f ins_rdbms.mk ioracle

ADP Confidential

Page 117 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

main.cf with CRS resources include include include include include

"types.cf" "CFSTypes.cf" "CVMTypes.cf" "OracleTypes.cf" "PrivNIC.cf"

cluster d1sol-c ( UserNames = { admin = IhiAhcHeiDiiGqiChf } ClusterAddress = "192.168.201.192" Administrators = { admin } CredRenewFrequency = 0 UseFence = SCSI3 HacliUserLevel = COMMANDROOT CounterInterval = 5 ) system d1sol5 ( ) ADP Confidential

Page 118 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

system d1sol6 ( ) group ClusterService ( SystemList = { d1sol5 = 0, d1sol6 = 1 } AutoStartList = { d1sol5, d1sol6 } OnlineRetryLimit = 3 OnlineRetryInterval = 120 ) IP webip ( Device = ipge0 Address = "192.168.201.192" NetMask = "255.255.255.0" ) NIC csgnic ( Device = ipge0 ) VRTSWebApp VCSweb ( Critical = 0 AppName = vcs InstallDir = "/opt/VRTSweb/VERITAS" TimeForOnline = 5 RestartLimit = 3 ) VCSweb requires webip webip requires csgnic // resource dependency tree // // group ClusterService // { // VRTSWebApp VCSweb // { // IP webip // { // NIC csgnic // } // } // } group cvm ( SystemList = { d1sol5 = 0, d1sol6 = 1 } AutoFailOver = 0 ADP Confidential

Page 119 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Parallel = 1 AutoStartList = { d1sol5, d1sol6 } ) CFSMount crs_ora_mnt ( MountPoint = "/ora_crs" BlockDevice = "/dev/vx/dsk/crs_oradg/crs_vol" ) CVMVolDg crs_voldg ( CVMDiskGroup = crs_oradg CVMVolume = { crs_vol } CVMActivation = sw ) PrivNIC ora_priv ( Device = { ipge2 = 0, ipge3 = 1 Address @d1sol5 = "10.10.10.1" Address @d1sol6 = "10.10.10.2" NetMask = "255.255.255.0" )

}

CFSQlogckd qlogckd ( Critical = 0 ) CFSfsckd vxfsckd ( ) CVMCluster cvm_clus ( CVMClustName = d1sol-c CVMNodeId = { d1sol5 = 0, d1sol6 = 1 } CVMTransport = gab CVMTimeout = 200 ) CVMVxconfigd cvm_vxconfigd ( Critical = 0 CVMVxconfigdArgs = { syslog } ) crs_ora_mnt requires crs_voldg crs_ora_mnt requires vxfsckd crs_voldg requires cvm_clus cvm_clus requires cvm_vxconfigd qlogckd requires cvm_clus vxfsckd requires qlogckd // resource dependency tree ADP Confidential

Page 120 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

// // // // // // // // // // // // // // // // // // // // // // // // //

ADP Confidential

group cvm { CFSMount crs_ora_mnt { CVMVolDg crs_voldg { CVMCluster cvm_clus { CVMVxconfigd cvm_vxconfigd } } CFSfsckd vxfsckd { CFSQlogckd qlogckd { CVMCluster cvm_clus { CVMVxconfigd cvm_vxconfigd } } } } PrivNIC ora_priv }

Page 121 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

main.cf with CVM and Oracle Service Groups and without the Oracle Agent include include include include include

"types.cf" "CFSTypes.cf" "CVMTypes.cf" "OracleTypes.cf" "PrivNIC.cf"

cluster d1sol-c ( UserNames = { admin = IhiAhcHeiDiiGqiChf, vcsop = GjkIjgJgjRkd } ClusterAddress = "192.168.201.192" Administrators = { admin } Operators = { vcsop } CredRenewFrequency = 0 UseFence = SCSI3 HacliUserLevel = COMMANDROOT CounterInterval = 5 ) system d1sol5 ( ) system d1sol6 ( ) group ClusterService ( SystemList = { d1sol5 = 0, d1sol6 = 1 } AutoStartList = { d1sol5, d1sol6 } OnlineRetryLimit = 3 OnlineRetryInterval = 120 ) IP webip ( Device = ipge0 Address = "192.168.201.192" NetMask = "255.255.255.0" ) NIC csgnic ( Device = ipge0 ) NotifierMngr ntfr ( SmtpServer = "160.0.10.50" SmtpRecipients = { "[email protected]" = Warning } ) VRTSWebApp VCSweb ( ADP Confidential

Page 122 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

Critical = 0 AppName = vcs InstallDir = "/opt/VRTSweb/VERITAS" TimeForOnline = 5 RestartLimit = 3 ) VCSweb requires webip webip requires csgnic // resource dependency tree // // group ClusterService // { // VRTSWebApp VCSweb // { // IP webip // { // NIC csgnic // } // } // NotifierMngr ntfr // } group cvm ( SystemList = { d1sol5 = 0, d1sol6 = 1 } AutoFailOver = 0 Parallel = 1 AutoStartList = { d1sol5, d1sol6 } ) Application cssd ( Critical = 0 StartProgram = "/opt/VRTSvcs/rac/bin/cssd-online" StopProgram = "/opt/VRTSvcs/rac/bin/cssd-offline stopcrs" CleanProgram = "/opt/VRTSvcs/rac/bin/cssd-clean" MonitorProgram = "/opt/VRTSvcs/rac/bin/cssd-monitor" OnlineRetryLimit = 20 ) CFSMount crs_ora_mnt ( MountPoint = "/ora_crs" BlockDevice = "/dev/vx/dsk/crs_oradg/crs_vol" ) CFSMount hapdb_archive_mnt ( Critical = 0 MountPoint = "/hapdb/archive" ADP Confidential

Page 123 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

BlockDevice = "/dev/vx/dsk/ora_dg1/ora_archive_vol" ) CFSMount hapdb_data1_mnt ( Critical = 0 MountPoint = "/hapdb/data1" BlockDevice = "/dev/vx/dsk/ora_dg1/ora_data1_vol" ) CFSMount hapdb_flashdb_mnt ( Critical = 0 MountPoint = "/hapdb/flashdb" BlockDevice = "/dev/vx/dsk/ora_dg1/ora_flashdb_vol" ) CFSMount hapdb_index1_mnt ( Critical = 0 MountPoint = "/hapdb/index1" BlockDevice = "/dev/vx/dsk/ora_dg1/ora_index1_vol" ) CFSMount hapdb_redo1_mnt ( Critical = 0 MountPoint = "/hapdb/redo1" BlockDevice = "/dev/vx/dsk/ora_dg1/ora_redo1_vol" ) CFSMount hapdb_redo2_mnt ( Critical = 0 MountPoint = "/hapdb/redo2" BlockDevice = "/dev/vx/dsk/ora_dg1/ora_redo2_vol" ) CFSMount hapdb_redocopy_mnt ( Critical = 0 MountPoint = "/hapdb/redocopy" BlockDevice = "/dev/vx/dsk/ora_dg1/ora_redocopy_vol" ) CFSMount hapdb_system_mnt ( Critical = 0 MountPoint = "/hapdb/system" BlockDevice = "/dev/vx/dsk/ora_dg1/ora_system_vol" ) CFSMount hapdb_temp_mnt ( Critical = 0 MountPoint = "/hapdb/temp" BlockDevice = "/dev/vx/dsk/ora_dg1/ora_temp_vol" ) ADP Confidential

Page 124 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

CFSQlogckd qlogckd ( Critical = 0 ) CFSfsckd vxfsckd ( ) CVMCluster cvm_clus ( CVMClustName = d1sol-c CVMNodeId = { d1sol5 = 0, d1sol6 = 1 } CVMTransport = gab CVMTimeout = 200 ) CVMVolDg crs_voldg ( CVMDiskGroup = crs_oradg CVMVolume = { crs_vol } CVMActivation = sw ) CVMVolDg hapdb_archive_voldg ( CVMDiskGroup = ora_dg1 CVMVolume = { ora_archive_vol } CVMActivation = sw ) CVMVolDg hapdb_data1_voldg ( CVMDiskGroup = ora_dg1 CVMVolume = { ora_data1_vol } CVMActivation = sw ) CVMVolDg hapdb_flashdb_voldg ( CVMDiskGroup = ora_dg1 CVMVolume = { ora_flashdb_vol } CVMActivation = sw ) CVMVolDg hapdb_index1_voldg ( CVMDiskGroup = ora_dg1 CVMVolume = { ora_index1_vol } CVMActivation = sw ) CVMVolDg hapdb_redo1_voldg ( CVMDiskGroup = ora_dg1 CVMVolume = { ora_redo1_vol } CVMActivation = sw ) ADP Confidential

Page 125 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

CVMVolDg hapdb_redo2_voldg ( CVMDiskGroup = ora_dg1 CVMVolume = { ora_redo2_vol } CVMActivation = sw ) CVMVolDg hapdb_redocopy_voldg ( CVMDiskGroup = ora_dg1 CVMVolume = { ora_redocopy_vol } CVMActivation = sw ) CVMVolDg hapdb_system_voldg ( CVMDiskGroup = ora_dg1 CVMVolume = { ora_system_vol } CVMActivation = sw ) CVMVolDg hapdb_temp_voldg ( CVMDiskGroup = ora_dg1 CVMVolume = { ora_temp_vol } CVMActivation = sw ) CVMVxconfigd cvm_vxconfigd ( Critical = 0 CVMVxconfigdArgs = { syslog } ) PrivNIC ora_priv ( Device = { ipge2 = 0, ipge3 = 1 } Address @d1sol5 = "10.10.10.1" Address @d1sol6 = "10.10.10.2" NetMask = "255.255.255.0" ) crs_ora_mnt requires crs_voldg crs_ora_mnt requires vxfsckd crs_voldg requires cvm_clus cssd requires crs_ora_mnt cssd requires hapdb_archive_mnt cssd requires hapdb_data1_mnt cssd requires hapdb_flashdb_mnt cssd requires hapdb_index1_mnt cssd requires hapdb_redo1_mnt cssd requires hapdb_redo2_mnt cssd requires hapdb_redocopy_mnt cssd requires hapdb_system_mnt cssd requires hapdb_temp_mnt ADP Confidential

Page 126 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

cssd requires ora_priv cvm_clus requires cvm_vxconfigd hapdb_archive_mnt requires hapdb_archive_voldg hapdb_archive_mnt requires vxfsckd hapdb_archive_voldg requires cvm_clus hapdb_data1_mnt requires hapdb_data1_voldg hapdb_data1_mnt requires vxfsckd hapdb_data1_voldg requires cvm_clus hapdb_flashdb_mnt requires hapdb_flashdb_voldg hapdb_flashdb_mnt requires vxfsckd hapdb_flashdb_voldg requires cvm_clus hapdb_index1_mnt requires hapdb_index1_voldg hapdb_index1_mnt requires vxfsckd hapdb_index1_voldg requires cvm_clus hapdb_redo1_mnt requires hapdb_redo1_voldg hapdb_redo1_mnt requires vxfsckd hapdb_redo1_voldg requires cvm_clus hapdb_redo2_mnt requires hapdb_redo2_voldg hapdb_redo2_mnt requires vxfsckd hapdb_redo2_voldg requires cvm_clus hapdb_redocopy_mnt requires hapdb_redocopy_voldg hapdb_redocopy_mnt requires vxfsckd hapdb_redocopy_voldg requires cvm_clus hapdb_system_mnt requires hapdb_system_voldg hapdb_system_mnt requires vxfsckd hapdb_system_voldg requires cvm_clus hapdb_temp_mnt requires hapdb_temp_voldg hapdb_temp_mnt requires vxfsckd hapdb_temp_voldg requires cvm_clus qlogckd requires cvm_clus vxfsckd requires qlogckd // resource dependency tree // // group cvm // { // Application cssd // { // CFSMount crs_ora_mnt // { // CFSfsckd vxfsckd // { // CFSQlogckd qlogckd // { // CVMCluster cvm_clus // { // CVMVxconfigd cvm_vxconfigd // } // } ADP Confidential

Page 127 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

// // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // ADP Confidential

} CVMVolDg crs_voldg { CVMCluster cvm_clus { CVMVxconfigd cvm_vxconfigd } } } CFSMount hapdb_archive_mnt { CFSfsckd vxfsckd { CFSQlogckd qlogckd { CVMCluster cvm_clus { CVMVxconfigd cvm_vxconfigd } } } CVMVolDg hapdb_archive_voldg { CVMCluster cvm_clus { CVMVxconfigd cvm_vxconfigd } } } CFSMount hapdb_data1_mnt { CFSfsckd vxfsckd { CFSQlogckd qlogckd { CVMCluster cvm_clus { CVMVxconfigd cvm_vxconfigd } } } CVMVolDg hapdb_data1_voldg { CVMCluster cvm_clus { CVMVxconfigd cvm_vxconfigd } } } CFSMount hapdb_index1_mnt Page 128 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

// // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // ADP Confidential

{ CFSfsckd vxfsckd { CFSQlogckd qlogckd { CVMCluster cvm_clus { CVMVxconfigd cvm_vxconfigd } } } CVMVolDg hapdb_index1_voldg { CVMCluster cvm_clus { CVMVxconfigd cvm_vxconfigd } } } CFSMount hapdb_redo1_mnt { CFSfsckd vxfsckd { CFSQlogckd qlogckd { CVMCluster cvm_clus { CVMVxconfigd cvm_vxconfigd } } } CVMVolDg hapdb_redo1_voldg { CVMCluster cvm_clus { CVMVxconfigd cvm_vxconfigd } } } CFSMount hapdb_redo2_mnt { CFSfsckd vxfsckd { CFSQlogckd qlogckd { CVMCluster cvm_clus { CVMVxconfigd cvm_vxconfigd } } Page 129 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

// // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // ADP Confidential

} CVMVolDg hapdb_redo2_voldg { CVMCluster cvm_clus { CVMVxconfigd cvm_vxconfigd } } } CFSMount hapdb_redocopy_mnt { CFSfsckd vxfsckd { CFSQlogckd qlogckd { CVMCluster cvm_clus { CVMVxconfigd cvm_vxconfigd } } } CVMVolDg hapdb_redocopy_voldg { CVMCluster cvm_clus { CVMVxconfigd cvm_vxconfigd } } } CFSMount hapdb_system_mnt { CFSfsckd vxfsckd { CFSQlogckd qlogckd { CVMCluster cvm_clus { CVMVxconfigd cvm_vxconfigd } } } CVMVolDg hapdb_system_voldg { CVMCluster cvm_clus { CVMVxconfigd cvm_vxconfigd } } } CFSMount hapdb_temp_mnt Page 130 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

// // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // // //

ADP Confidential

}

{ CFSfsckd vxfsckd { CFSQlogckd qlogckd { CVMCluster cvm_clus { CVMVxconfigd cvm_vxconfigd } } } CVMVolDg hapdb_temp_voldg { CVMCluster cvm_clus { CVMVxconfigd cvm_vxconfigd } } } CFSMount hapdb_flashdb_mnt { CFSfsckd vxfsckd { CFSQlogckd qlogckd { CVMCluster cvm_clus { CVMVxconfigd cvm_vxconfigd } } } CVMVolDg hapdb_flashdb_voldg { CVMCluster cvm_clus { CVMVxconfigd cvm_vxconfigd } } } PrivNIC ora_priv }

Page 131 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

main.cf with CVM and Oracle Service Groups and with the Oracle Agent (make sure table required for detail monitoring is already created) include "types.cf" include "CFSTypes.cf" include "CVMTypes.cf" include "OracleTypes.cf" include "PrivNIC.cf" cluster d1sol-c ( UserNames = { admin = IhiAhcHeiDiiGqiChf } ClusterAddress = "192.168.201.192" Administrators = { admin } CredRenewFrequency = 0 UseFence = SCSI3 HacliUserLevel = COMMANDROOT CounterInterval = 5 ) system d1sol5 ( ) system d1sol6 ( ) group ClusterService ( SystemList = { d1sol5 = 0, d1sol6 = 1 } AutoStartList = { d1sol5, d1sol6 } OnlineRetryLimit = 3 OnlineRetryInterval = 120 ) IP webip ( Device = ipge0 Address = "192.168.201.192" NetMask = "255.255.255.0" ) NIC csgnic ( Device = ipge0 ) ADP Confidential

Page 132 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

NotifierMngr ntfr ( SmtpServer = "160.0.10.50" SmtpRecipients = { "[email protected]" = Warning } ) VRTSWebApp VCSweb ( Critical = 0 AppName = vcs InstallDir = "/opt/VRTSweb/VERITAS" TimeForOnline = 5 RestartLimit = 3 ) VCSweb requires webip webip requires csgnic // resource dependency tree // // group ClusterService // { // VRTSWebApp VCSweb // { // IP webip // { // NIC csgnic // } // } // NotifierMngr ntfr // } group cvm ( SystemList = { d1sol5 = 0, d1sol6 = 1 } AutoFailOver = 0 Parallel = 1 AutoStartList = { d1sol5, d1sol6 } ) Application cssd ( Critical = 0 StartProgram = "/opt/VRTSvcs/rac/bin/cssd-online" StopProgram = "/opt/VRTSvcs/rac/bin/cssd-offline stopcrs" CleanProgram = "/opt/VRTSvcs/rac/bin/cssd-clean" MonitorProgram = "/opt/VRTSvcs/rac/bin/cssd-monitor" OnlineRetryLimit = 20 ) CFSMount crs_ora_mnt ( MountPoint = "/ora_crs" ADP Confidential

Page 133 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

BlockDevice = "/dev/vx/dsk/crs_oradg/crs_vol" ) CFSQlogckd qlogckd ( Critical = 0 ) CFSfsckd vxfsckd ( ) CVMCluster cvm_clus ( CVMClustName = d1sol-c CVMNodeId = { d1sol5 = 0, d1sol6 = 1 } CVMTransport = gab CVMTimeout = 200 ) CVMVolDg crs_voldg ( CVMDiskGroup = crs_oradg CVMVolume = { crs_vol } CVMActivation = sw ) CVMVxconfigd cvm_vxconfigd ( Critical = 0 CVMVxconfigdArgs = { syslog } ) PrivNIC ora_priv ( Device = { ipge2 = 0, ipge3 = 1 } Address @d1sol5 = "10.10.10.1" Address @d1sol6 = "10.10.10.2" NetMask = "255.255.255.0" ) crs_ora_mnt requires crs_voldg crs_ora_mnt requires vxfsckd crs_voldg requires cvm_clus cssd requires crs_ora_mnt cssd requires ora_priv cvm_clus requires cvm_vxconfigd qlogckd requires cvm_clus vxfsckd requires qlogckd // resource dependency tree // // group cvm // { // Application cssd ADP Confidential

Page 134 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

// // // // // // // // // // // // // // // // // // // // // // // //

}

{ CFSMount crs_ora_mnt { CVMVolDg crs_voldg { CVMCluster cvm_clus { CVMVxconfigd cvm_vxconfigd } } CFSfsckd vxfsckd { CFSQlogckd qlogckd { CVMCluster cvm_clus { CVMVxconfigd cvm_vxconfigd } } } } PrivNIC ora_priv }

group hapdb_rac ( SystemList = { d1sol5 = 0, d1sol6 = 1 } AutoFailOver = 0 Parallel = 1 AutoStartList = { d1sol5, d1sol6 } ) CFSMount hapdb_archive_mnt ( Critical = 0 MountPoint = "/hapdb/archive" BlockDevice = "/dev/vx/dsk/ora_dg1/ora_archive_vol" ) CFSMount hapdb_data1_mnt ( Critical = 0 MountPoint = "/hapdb/data1" BlockDevice = "/dev/vx/dsk/ora_dg1/ora_data1_vol" ) CFSMount hapdb_flashdb_mnt ( Critical = 0 MountPoint = "/hapdb/flashdb" BlockDevice = "/dev/vx/dsk/ora_dg1/ora_flashdb_vol" ) ADP Confidential

Page 135 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

CFSMount hapdb_index1_mnt ( Critical = 0 MountPoint = "/hapdb/index1" BlockDevice = "/dev/vx/dsk/ora_dg1/ora_index1_vol" ) CFSMount hapdb_redo1_mnt ( Critical = 0 MountPoint = "/hapdb/redo1" BlockDevice = "/dev/vx/dsk/ora_dg1/ora_redo1_vol" ) CFSMount hapdb_redo2_mnt ( Critical = 0 MountPoint = "/hapdb/redo2" BlockDevice = "/dev/vx/dsk/ora_dg1/ora_redo2_vol" ) CFSMount hapdb_redocopy_mnt ( Critical = 0 MountPoint = "/hapdb/redocopy" BlockDevice = "/dev/vx/dsk/ora_dg1/ora_redocopy_vol" ) CFSMount hapdb_system_mnt ( Critical = 0 MountPoint = "/hapdb/system" BlockDevice = "/dev/vx/dsk/ora_dg1/ora_system_vol" ) CFSMount hapdb_temp_mnt ( Critical = 0 MountPoint = "/hapdb/temp" BlockDevice = "/dev/vx/dsk/ora_dg1/ora_temp_vol" ) CVMVolDg hapdb_archive_voldg ( CVMDiskGroup = ora_dg1 CVMVolume = { ora_archive_vol } CVMActivation = sw ) CVMVolDg hapdb_data1_voldg ( CVMDiskGroup = ora_dg1 CVMVolume = { ora_data1_vol } CVMActivation = sw ) CVMVolDg hapdb_flashdb_voldg ( ADP Confidential

Page 136 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

CVMDiskGroup = ora_dg1 CVMVolume = { ora_flashdb_vol } CVMActivation = sw ) CVMVolDg hapdb_index1_voldg ( CVMDiskGroup = ora_dg1 CVMVolume = { ora_index1_vol } CVMActivation = sw ) CVMVolDg hapdb_redo1_voldg ( CVMDiskGroup = ora_dg1 CVMVolume = { ora_redo1_vol } CVMActivation = sw ) CVMVolDg hapdb_redo2_voldg ( CVMDiskGroup = ora_dg1 CVMVolume = { ora_redo2_vol } CVMActivation = sw ) CVMVolDg hapdb_redocopy_voldg ( CVMDiskGroup = ora_dg1 CVMVolume = { ora_redocopy_vol } CVMActivation = sw ) CVMVolDg hapdb_system_voldg ( CVMDiskGroup = ora_dg1 CVMVolume = { ora_system_vol } CVMActivation = sw ) CVMVolDg hapdb_temp_voldg ( CVMDiskGroup = ora_dg1 CVMVolume = { ora_temp_vol } CVMActivation = sw ) Oracle Ora-hapdb ( Sid @d1sol5 = hapdb1 Sid @d1sol6 = hapdb2 Owner = oracle Home = "/oracle/product/10g/db_1" Pfile @d1sol5 = "/opt/oracle/product/10g/db_1/dbs/inithapdb1.ora" Pfile @d1sol6 = "/opt/oracle/product/10g/db_1/dbs/inithapdb2.ora" EnvFile = "/opt/VRTSvcs/bin/Oracle/envfile" MonScript = "/opt/VRTSvcs/bin/Oracle/SqlTest.pl" ADP Confidential

Page 137 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

DetailMonitor = 1 User = vcsmon Pword = vcsmon Table @d1sol5 = vcstable_d1sol5 Table @d1sol6 = vcstable_d1sol6 ) requires group cvm online local firm Ora-hapdb requires hapdb_archive_mnt Ora-hapdb requires hapdb_data1_mnt Ora-hapdb requires hapdb_index1_mnt Ora-hapdb requires hapdb_redo1_mnt Ora-hapdb requires hapdb_redo2_mnt Ora-hapdb requires hapdb_redocopy_mnt Ora-hapdb requires hapdb_system_mnt Ora-hapdb requires hapdb_temp_mnt Ora-hapdb requires hapdb_flashdb_mnt hapdb_archive_mnt requires hapdb_archive_voldg hapdb_data1_mnt requires hapdb_data1_voldg hapdb_index1_mnt requires hapdb_index1_voldg hapdb_redo1_mnt requires hapdb_redo1_voldg hapdb_redo2_mnt requires hapdb_redo2_voldg hapdb_redocopy_mnt requires hapdb_redocopy_voldg hapdb_system_mnt requires hapdb_system_voldg hapdb_temp_mnt requires hapdb_temp_voldg hapdb_flashdb_mnt requires hapdb_flashdb_voldg // resource dependency tree // // group hapdb_rac // { // Oracle Ora-hapdb // { // CFSMount hapdb_archive_mnt // { // CVMVolDg hapdb_archive_voldg // CVMVolDg hapdb_data1_voldg // CVMVolDg hapdb_index1_voldg // CVMVolDg hapdb_redo1_voldg // CVMVolDg hapdb_redo2_voldg // CVMVolDg hapdb_redocopy_voldg // CVMVolDg hapdb_system_voldg // CVMVolDg hapdb_temp_voldg // } // CFSMount hapdb_data1_mnt // CFSMount hapdb_index1_mnt // CFSMount hapdb_redo1_mnt // CFSMount hapdb_redo2_mnt // CFSMount hapdb_redocopy_mnt ADP Confidential

Page 138 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.

eTEAM

Oracle 10g RAC Installation Guide

// // // // // //

ADP Confidential

CFSMount hapdb_system_mnt CFSMount hapdb_temp_mnt } CFSMount hapdb_flashdb_mnt CVMVolDg hapdb_flashdb_voldg }

Page 139 of 139 Last modified by kappukuttan ©2006 Automatic Data Processing, Inc. All Rights Reserved.