Installation Sun 10gr2 Rac Asm

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Oracle Deutschland GmbH Oracle RAC with ASM

Authors:

Jürgen Dömel Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Created

05/05/2006 14:08:00

Last change:

29/11/2006 14:11:35 A11/P11

Version:

2.0

Page 1 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Table of Contents Table of Contents......................................................................................................................................................2 Document Info...........................................................................................................................................................4 Content..............................................................................................................................................................4 Cluster / host information..................................................................................................................................4 RAC...................................................................................................................................................................4 Hardware Installation................................................................................................................................................5 Architecture.......................................................................................................................................................5 Drawing.............................................................................................................................................................5 Solaris Installation.....................................................................................................................................................6 Solaris 10 u1......................................................................................................................................................6 Setup slvm.........................................................................................................................................................6 EIS-CD..............................................................................................................................................................7 Setup IPMP groups on all nodes for all public network interfaces...................................................................8 Modify NVRAM parameter:.............................................................................................................................8 Setup Trunking for the Clusterinterconnect......................................................................................................9 Last Systemanalyse...........................................................................................................................................9 Prepare Oracle Installation......................................................................................................................................10 Set kernel parameters......................................................................................................................................10 Create Users and Groups.................................................................................................................................10 Share staging area............................................................................................................................................10 Mount staging area..........................................................................................................................................10 Modify /etc/hosts.............................................................................................................................................10 Check SSH daemon.........................................................................................................................................11 Setup SSH for oracle.......................................................................................................................................11 Setup userenv for oracle..................................................................................................................................11 Perform cluster check......................................................................................................................................11 Workaround Bug.............................................................................................................................................12 Prepare storage................................................................................................................................................12 Verify device order..........................................................................................................................................12 Trobleshooting device size..............................................................................................................................13 Label devices...................................................................................................................................................13 Install Clusterware 10.2.0.1.....................................................................................................................................14 Run installer....................................................................................................................................................14 Trobleshooting Assistants...............................................................................................................................23 Check Cluster Status.......................................................................................................................................25 Install crsstat.sh...............................................................................................................................................26 Install ASM Home 10.2.0.1....................................................................................................................................27 Run installer....................................................................................................................................................27 Create ASM Instance..............................................................................................................................................35 Run DBCA......................................................................................................................................................35 Install Database Home 10.2.0.1..............................................................................................................................41 Run installer ...................................................................................................................................................41 Install Management Agent......................................................................................................................................47 Run installer....................................................................................................................................................47 Create Database ZDA..............................................................................................................................................55 Run DBCA......................................................................................................................................................55 Create login script for user oracle...................................................................................................................70 Install Patchset 10.2.0.2 on CRS Home..................................................................................................................71 Stop all oracle processes.................................................................................................................................71 Install binaries.................................................................................................................................................71 Restart Custerware..........................................................................................................................................75 Install Patchset 10.2.0.2 on ASM Home.................................................................................................................76 Stop Nodeapps.................................................................................................................................................76 Install binaries.................................................................................................................................................76 Install Patchset 10.2.0.2 on DB Home....................................................................................................................81 Verify service state..........................................................................................................................................81 Install binaries.................................................................................................................................................81 Install Interim Patch 5117016 on DB Home...........................................................................................................87 Install with opatch...........................................................................................................................................87 Start services required to perform database upgrade......................................................................................89 Page 2 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Upgrade database ZDA...................................................................................................................................89 Restart cluster..................................................................................................................................................90 How to Setup IPMP as Cluster Interconnect Doc ID: Note:368464.1..........................................................90 Add new node to cluster..........................................................................................................................................91 Create links to devices.....................................................................................................................................91 Change permissions.........................................................................................................................................91 Modify /etc/hosts.............................................................................................................................................92 Check SSH daemon.........................................................................................................................................92 Enable user equivalence..................................................................................................................................92 Perform cluster check......................................................................................................................................93 Workaround OUI Bug.....................................................................................................................................93 Install clusterware on new node......................................................................................................................93 Install ASM home on new node......................................................................................................................97 Install DB home on new node.......................................................................................................................101 Create listener in DB Home..........................................................................................................................105 Create ASM instance on new node...............................................................................................................110 Create database instance on new node..........................................................................................................113 End of Installation.........................................................................................................................................119 Remove Node from Cluster...................................................................................................................................120 Remove instance...........................................................................................................................................120 Remove Listener...........................................................................................................................................124 Clean-up the ASM.........................................................................................................................................128 Removing node from the database................................................................................................................128 Removing node from the clusterware:..........................................................................................................128 Cluster Performance Tuning.................................................................................................................................130 Apply Brownout Patches...............................................................................................................................130 Set the CSS misscount to match the service level.........................................................................................130 Run CSS in Real Time..................................................................................................................................130 Set the check_interval for VIP addresses......................................................................................................130 Remove the dependency between the VIP and ASM...................................................................................131 Reduce the ping timeout................................................................................................................................131 Troubleshooting VIP Failover...............................................................................................................................132 Adjusting Listener Configuration..................................................................................................................132 Upgrade Oracle Home of Swingbench .........................................................................................................132 Adjust SQLNet settings.................................................................................................................................132 Set the ping timeout back to default..............................................................................................................132 Addendum.............................................................................................................................................................133 Metalink Note 284752.1................................................................................................................................133 Metalink Note 294430.1................................................................................................................................134 Metalink Note 294430.1................................................................................................................................140

Page 3 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Document Info Content This is an installation summary of the installation and patch process of the ZDA environment. It is based on the official procuct documentation provided by oracle cannot replace the those documents. Cluster / host information Hostnames:

orac-rac-1, ora-rac-2, ora-rac,4

OS:

Solaris 10

RAC Oracle Cluster Name:

crs

Oracle Versions

10.2.0.1

Page 4 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Hardware Installation Architecture The cluster is based on two SF440 with 4 CPU's, 16GB RAM and 4 internal Disks. The internaldisks are mirrored with SLVM. Two disks are used for the OS and the other two are used for Oracle. Oracle will not be installed on shared disks because of the rolling upgrade capability. The shared disks are luns from a SAN and they are mirrored with Oracle ASM. The clusterinterconnect is realized with two trunked gigabitethernet connections. Drawing

IPMP Group ipmp0

IPMP Group ipmp0

2 x Interconnect

Oracle RAC Cluster Node: ora-rac-1

Storage A

MpxIO

MpxIO

Oracle RAC Cluster Node: ora-rac-2

Storage B

Page 5 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Solaris Installation Solaris 10 u1 Solaris 10 u1 SUNWCXall is installed. • For SVM reserve slice 7 with 32MB (EIS recommendation) for metaDB/replicas. • Ensure that the SAN Foundation Suite packages are installed and patched. (SAN is necessary for fibre server/storage) change the home of root #mkdir /root #vi /etc/passwd root:x:0:0:Super-User:/root:/sbin/sh logout,login

Setup slvm #cat /etc/lvm/md.tab mddb1

-c 3

d10

-m

/dev/dsk/c3t0d0s7 d11

d11

1 1

/dev/dsk/c3t0d0s0

d12

1 1

/dev/dsk/c3t1d0s0

d20

-m

d21 d22 d30

d21 1 1

/dev/dsk/c3t0d0s1

1 1

/dev/dsk/c3t1d0s1

-m

d31

d31

1 1

/dev/dsk/c3t0d0s6

d32

1 1

/dev/dsk/c3t1d0s6

Page 6 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

EIS-CD Setup EIS environment cd /cdrom/eis-dvd/sun/install ./setup-standard.sh Install actual Solaris 10 patches cd /cdrom/eis-dvd/patch/10 unpack-patches cd /tmp/10 ./install-patches Analyse basic systemstatus explorer cd /opt/SUNWexplo/output/explorer* /cdrom/eis-dvd/sun/tools/explominer/explominer see the outputfile ExploMiner_EIS_Patchlist_nobody of the explominer and install all neccessary patches.

To enable MpxIO on Solaris 10 start stmsboot -e before OracleCluster will be installed !! For safety reasons clean the devices with: devfsadm -C on all nodes.

Page 7 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Setup IPMP groups on all nodes for all public network interfaces. At least you need one test IP for each network adapter and one production IP for each IPMP group. e.g: IPMP group active-active with interfaces ce0 and ce4 with one physical IP. ora-rac-1 Entry of /etc/hostname.ce0 : ora-rac-1-1 netmask + broadcast + group ipmp0 up \ addif

ora-rac-1 netmask + broadcast + deprecated -failover up

Entry of /etc/hostname.ce4 : ora-rac-1-2 netmask + broadcast + group ipmp0 deprecated -failover up

ora-rac-2 Entry of /etc/hostname.ce0 : ora-rac-2-1 netmask + broadcast + group ipmp0 up \ addif ora-rac-3 netmask + broadcast + deprecated -failover up

Entry of /etc/hostname.ce4 : ora-rac-2-2 netmask + broadcast + group ipmp0 deprecated -failover up

If the defaultrouter is NOT 100% available please read infodocs 70060 and 70062. Notes: • Do not use test IP for normal applications • Test IP for all adapters in the same IPMP group must belong to a single IP subnet • You need an addtional IP for each logical host. • In the /etc/default/mpathd file, the value of TRACK_INTERFACES_ONLY_WITH_GROUPS must be yes. (default)

Modify NVRAM parameter: Set boot-device & diag-device to both sides of the mirror.

Suggested naming convention: • Primary: rootdisk • Secondary: rootmirror

Page 8 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Setup Trunking for the Clusterinterconnect Install Sun Trunking 1.3 for the ce interfaces. pkgadd -d ./SUNWtrku pkgadd -d ./SUNWtrkm #vi /etc/opt/SUNWconn/bin/nettr.sh

Add at the end: nttr -setup 1 device=ce members=1,3 policy=1 #reboot

Check the configuration: #/etc/opt/SUNWconn/bin/nettr -conf Name Head Policy DEV Type Original-Mac-Addr Speed Duplex Link ce1

ce1

ce3

ce1

1

pci

8:0:20:8f:be:24

1000

Full Up

pci

8:0:20:8f:be:24

1000

Full Up

Last Systemanalyse explorer cd /opt/SUNWexplo/output/explorer* /cdrom/eis-dvd/sun/tools/explominer/explominer see the outputfile ExploMiner_EIS_Patchlist_nobody of the explominer and install all neccessary patches.

Page 9 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Prepare Oracle Installation Set kernel parameters *

Oracle settings *

set noexec_user_stack=1 set rlim_fd_cur=8192 * end of Oracle settings *

Create Users and Groups >> ora-rac-1, ora-rac-2 as root /usr/sbin/groupadd -g 61000 oinstall /usr/sbin/groupadd -g 61001 dba /usr/sbin/useradd -u 61000 -g oinstall -G dba oracle -s /usr/bin/bash -d /export/home/oracle mkdir -p /export/home/oracle chown -R oracle:oinstall /export/home/oracle passwd oracle > oracle > oracle

Share staging area All Oracle software is located on Host ora-poc2:/oracle/install. The directory is shared as /install/oracle and can be mounted on every node if needed. Share stage directory: >> ora-poc2 as root chown -R oracle:oinstall /oracle share -rw /oracle/install

Mount staging area In order to install oracle software, the staging area must be mounted on the instaation node of the cluster. >> ora-rac-1 as root mkdir /install mount ora-poc2:/oracle/install /install

Page 10 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Modify /etc/hosts 127.0.0.1 192.168.10.212 192.168.10.234 192.168.10.233 10.0.0.1 10.0.0.2 10.0.0.3 192.168.10.225

localhost ora-rac-1 ora-rac-1-1 ora-rac-1-2 ora-rac-1-priv ora-rac-1-priv-1 ora-rac-1-priv-2 ora-rac-1-vip

192.168.10.214 192.168.10.232 192.168.10.231 10.0.0.4 10.0.0.5 10.0.0.6 192.168.10.229

ora-rac-2 ora-rac-2-1 ora-rac-2-2 ora-rac-2-priv ora-rac-2-priv-1 ora-rac-2-priv-2 ora-rac-2-vip

192.168.10.223

ora-rac-3-vip

192.168.10.239

ora-rac-4-vip

loghost ora-rac-1.

loghost ora-rac-2.

Check SSH daemon ps -ef | grep sshd Setup SSH for oracle Note: This steps are different for any node in the cluster. Do complete the steps on the indicated nodes. >> ora-rac-1, ora-rac-2 : mkdir ~/.ssh chmod 700 ~/.ssh /usr/bin/ssh-keygen -t rsa /usr/bin/ssh-keygen -t dsa

>> ora-rac-1: cat id_dsa.pub > authorized_keys cat id_rsa.pub >> authorized_keys scp authorized_keys ora-rac-2:/export/home/oracle/.ssh/authorized_keys

>> ora-rac-2: cat id_dsa.pub >> authorized_keys cat id_rsa.pub >> authorized_keys

>> ora-rac-1: scp ora-rac-2:/export/home/oracle/.ssh/authorized_keys authorized_keys

>> ora-rac-1, ora-rac-2: ssh ora-rac-1 date ssh ora-rac-2 date exec /usr/bin/ssh-agent $SHELL /usr/bin/ssh-add

Page 11 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Setup userenv for oracle >> ora-rac-1, ora-rac-2 Add line to ~/.profile # Set Environment for oracle . ~/set_oracle_env.sh

Perform cluster check >> ora-rac-1 /install/10gR2/clusterware/runcluvfy.sh stage -pre crsinst -n ora-rac-1,orarac-2 -verbose

Workaround Bug The OUI may fail if it does not find ssh and scp binaries in /usr/local/bin. >> ora-rac-1, ora-rac-2 as root mkdir -p /usr/local/bin ln -s /usr/bin/ssh /usr/local/bin/ssh ln -s /usr/bin/scp /usr/local/bin/scp

Prepare storage Create symbolic links for all disk devices needed for oracle >> ora-rac-1, ora-rac-2 as root mkdir -p /dev/oracle ln -s /dev/rdsk/c7t600015D00022B3000000000000005012d0s1 /dev/oracle/ocr1 ln -s /dev/rdsk/c7t600015D00022B3000000000000005018d0s1 /dev/oracle/ocr2 ln -s /dev/rdsk/c7t600015D00022B300000000000000501Ed0s1 /dev/oracle/vote1 ln -s /dev/rdsk/c7t600015D00022B3000000000000005024d0s1 /dev/oracle/vote2 ln -s /dev/rdsk/c7t600015D00022B300000000000000502Ad0s1 /dev/oracle/vote3 ln -s /dev/rdsk/c7t600015D00022B3000000000000005042d0s1 /dev/oracle/asm01 ln -s /dev/rdsk/c7t600015D00022B3000000000000005048d0s1 /dev/oracle/asm02 ln -s /dev/rdsk/c7t600015D00022B300000000000000504Ed0s1 /dev/oracle/asm03 ln -s /dev/rdsk/c7t600015D00022B3000000000000005054d0s1 /dev/oracle/asm04 ln -s /dev/rdsk/c7t600015D00022B300000000000000505Ad0s1 /dev/oracle/asm05 ln -s /dev/rdsk/c7t600015D00022B3000000000000005060d0s1 /dev/oracle/asm06 ln -s /dev/rdsk/c7t600015D00022B3000000000000005066d0s1 /dev/oracle/asm07 ln -s /dev/rdsk/c7t600015D00022B300000000000000506Cd0s1 /dev/oracle/asm08 ln -s /dev/rdsk/c7t600015D00022B3000000000000005072d0s1 /dev/oracle/asm09 ln -s /dev/rdsk/c7t600015D00022B3000000000000005078d0s1 /dev/oracle/asm10

Change permissions for devices chown oracle:dba /dev/oracle/*

Verify device order Test if devices appear in same order on every node of the cluster. Page 12 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

>> ora-rac-1 Create some textfiles for testing... echo echo echo echo echo

"This "This "This "This "This

is is is is is

orc1" orc2" vote1" vote2" vote3"

> > > > >

/tmp/ocr1 /tmp/ocr2 /tmp/vote1 /tmp/vote2 /tmp/vote3

Copy text to device dd dd dd dd dd

if=/tmp/ocr1 of=/dev/oracle/ocr1 if=/tmp/ocr2 of=/dev/oracle/ocr2 if=/tmp/vote1 of=/dev/oracle/vote1 if=/tmp/vote2 of=/dev/oracle/vote2 if=/tmp/vote3 of=/dev/oracle/vote3

>> ora-rac-2 Copy device contents to file on ervery node dd dd dd dd dd

if=/dev/oracle/ocr1 if=/dev/oracle/ocr2 if=/dev/oracle/vote1 if=/dev/oracle/vote2 if=/dev/oracle/vote3

of=/tmp/ocr1 of=/tmp/ocr2 of=/tmp/vote1 of=/tmp/vote2 of=/tmp/vote3

Check if content is the same as on the other node cat cat cat cat cat

/tmp/ocr1 /tmp/ocr2 /tmp/vote1 /tmp/vote2 /tmp/vote3

Trobleshooting device size Count blocks dd if=/dev/oracle/ocr1

of=/tmp/ocr1

bs=1k count=10

dd if=/dev/oracle/ocr2

of=/tmp/ocr2

bs=1k count=10

dd if=/dev/oracle/vote1 of=/tmp/vote1 bs=1k count=10 dd if=/dev/oracle/vote2 of=/tmp/vote2 bs=1k count=10 dd if=/dev/oracle/vote3 of=/tmp/vote3 bs=1k count=10

Label devices Prepare devices for usage. Template Device: /dev/rdsk/c7t600015D00022B3000000000000005030d0s1

Label devices using template device: prtvtoc /dev/rdsk/c7t600015D00022B3000000000000005030d0s2 | fmthard -s /dev/rdsk/c7t600015D00022B3000000000000005012d0s2 prtvtoc /dev/rdsk/c7t600015D00022B3000000000000005030d0s2 | fmthard -s /dev/rdsk/c7t600015D00022B3000000000000005018d0s2 prtvtoc /dev/rdsk/c7t600015D00022B3000000000000005030d0s2 | fmthard -s /dev/rdsk/c7t600015D00022B300000000000000501Ed0s2 prtvtoc /dev/rdsk/c7t600015D00022B3000000000000005030d0s2 | fmthard -s /dev/rdsk/c7t600015D00022B3000000000000005024d0s2 prtvtoc /dev/rdsk/c7t600015D00022B3000000000000005030d0s2 | fmthard -s /dev/rdsk/c7t600015D00022B300000000000000502Ad0s2

Page 13 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Install Clusterware 10.2.0.1

Run installer >> ora-rac-1 /install/10gR2/clusterware/Disk1/runInstaller

Page 14 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 15 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 16 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 17 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 18 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 19 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 20 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 21 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 22 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Trobleshooting Assistants bash-3.00# ./root.sh WARNING: directory '/oracle/product' is not owned by root WARNING: directory '/oracle' is not owned by root Checking to see if Oracle CRS stack is already configured Setting the permissions on OCR backup directory Setting up NS directories Oracle Cluster Registry configuration upgraded successfully WARNING: directory '/oracle/product' is not owned by root WARNING: directory '/oracle' is not owned by root clscfg: EXISTING configuration version 3 detected. clscfg: version 3 is 10G Release 2. Successfully accumulated necessary OCR keys. Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897. node <nodenumber>: <nodename> <private interconnect name> node 1: ora-rac-1 ora-rac-1-priv ora-rac-1 node 2: ora-rac-2 ora-rac-2-priv ora-rac-2 clscfg: Arguments check out successfully. NO KEYS WERE WRITTEN. Supply -force parameter to override. -force is destructive and will destroy any previous cluster configuration. Oracle Cluster Registry for cluster has already been initialized Startup will be queued to init within 30 seconds. Adding daemons to inittab root.shExpecting the CRS daemons to be up within 600 seconds. CSS is active on these nodes. ora-rac-1 ora-rac-2 CSS is active on all nodes. Waiting for the Oracle CRSD and EVMD to start Oracle CRS stack installed and running under init(1M) Running vipca(silent) for configuring nodeapps The given interface(s), "ce0, ce2" is not public. Public interfaces should be used to configure virtual IPs.

Page 23 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

/oracle/product/crs/bin/racgons add_config ora-rac-1:6200

ora-rac-2:6200

/oracle/product/crs/bin/oifcfg setif -global ce0/192.168.10.0:public ce1/10.0.0.0:cluster_interconnect ce2/192.168.10.0:public ce3/10.0.0.0:cluster_interconnect /oracle/product/crs/bin/cluvfy stage -post crsinst -n ora-rac-1,ora-rac-2

Page 24 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Check Cluster Status >> ora-rac-1 ORACLE_HOME=/oracle/product/crs/bin export ORACLE_HOME PATH=$ORACLE_HOME/bin:$PATH export PATH $ crs_stat -t Name Type Target State Host ----------------------------------------------------------ora....c-1.gsd application OFFLINE OFFLINE ora....c-1.ons application ONLINE ONLINE ora-rac-1 ora....c-1.vip application ONLINE ONLINE ora-rac-1 ora....c-2.gsd application OFFLINE OFFLINE ora....c-2.ons application ONLINE ONLINE ora-rac-2 ora....c-2.vip application ONLINE ONLINE ora-rac-2

Page 25 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Install crsstat.sh Because of the unreadable output of „crs_stat -t“ setup a wrapper for crs_stat. mkdir /oracle/tools vi /oracle/tools/crsstat.sh

#!/usr/bin/ksh # # Sample 10g CRS resource status query script # # Description: #

- Returns formatted version of crs_stat -t, in tabular

# #

format, with the complete rsc names and filtering keywords - The argument, $RSC_KEY, is optional and if passed to the script, will

#

limit the output to HA resources whose names match $RSC_KEY.

# Requirements: #

- $CRS_HOME should be set in your environment

RSC_KEY=$1 QSTAT=-u AWK=/usr/xpg4/bin/awk # Table header:echo "" $AWK \ 'BEGIN {printf "%-45s %-10s %-18s\n", "HA Resource", "Target", "State"; printf "%-45s %-10s %-18s\n", "-----------", "------", "-----";}' # Table body: $CRS_HOME/bin/crs_stat $QSTAT | $AWK \ 'BEGIN { FS="="; state = 0; } $1~/NAME/ && $2~/'$RSC_KEY'/ {appname = $2; state=1}; state == 0 {next;} $1~/TARGET/ && state == 1 {apptarget = $2; state=2;} $1~/STATE/ && state == 2 {appstate = $2; state=3;} state == 3 {printf "%-45s %-10s %-18s\n", appname, apptarget, appstate; state=0;}'

Output of crsstat.sh HA Resource

Target

State

-----------

------

-----

ora.ora-rac-1.gsd

ONLINE

ONLINE on ora-rac-1

ora.ora-rac-1.ons

ONLINE

ONLINE on ora-rac-1

ora.ora-rac-1.vip

ONLINE

ONLINE on ora-rac-1

ora.ora-rac-2.gsd

ONLINE

ONLINE on ora-rac-2

ora.ora-rac-2.ons

ONLINE

ONLINE on ora-rac-2

ora.ora-rac-3.vip

ONLINE

ONLINE on ora-rac-2

Page 26 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Install ASM Home 10.2.0.1 Run installer >> ora-rac-1 /install/10gR2/database/Disk1/runInstaller

Page 27 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Ora_asmhome

Page 28 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 29 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 30 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 31 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 32 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 33 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 34 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Create ASM Instance Run DBCA /oracle/product/asm/bin/dbca

Page 35 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 36 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 37 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 38 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 39 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 40 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Install Database Home 10.2.0.1 Run installer >> ora-rac-1 /install/10gR2/database/Disk1/runInstaller

Page 41 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 42 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 43 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 44 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 45 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 46 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Install Management Agent Run installer ora-rac-1 /install/10gR2/database/Disk1/runInstaller

Page 47 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 48 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 49 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 50 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 51 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Password: oracle1

Page 52 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 53 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 54 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Create Database ZDA Run DBCA >> ora-rac-1 /oracle/product/db/bin/dbca

Page 55 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 56 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 57 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Password: oracle

Page 58 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 59 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 60 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 61 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 62 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 63 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 64 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 65 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 66 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 67 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 68 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 69 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Create login script for user oracle The script is located at ~/ set_oracle_env.sh ---------------This script starts here -----------#!/usr/bin/bash clear # Menu section. If user does not choose any oracle home, no actions takes place. echo echo echo echo echo echo

"Choose your Oracle Home and hit RETURN" "Enter 1 to use Oracle Home DB" "Enter 2 to use Oracle Home ASM" "Enter 3 to use Oracle Home Agent" "Enter 4 to use Oracle Home CRS" "Hit any other key to skip Oracle Home selection"

read answer if [[ $answer = 1 ]] then ORACLE_HOME=/oracle/product/db ; ORACLE_SID=ZDA1 elif [[ $answer = 2 ]] then ORACLE_HOME=/oracle/product/asm ; ORACLE_SID="+ASM1" ; elif [[ $answer = 3 ]] then ORACLE_HOME=/oracle/product/agent/agent10g ; elif [[ $answer = 4 ]] then ORACLE_HOME=/oracle/product/crs ; else return fi # If the user choose an oracle home, the following actions are performed: ORACLE_BASE=/oracle export ORACLE_BASE export PATH=$ORACLE_HOME/bin:$PATH export ORACLE_SID export ORACLE_HOME EDITOR=vi export EDITOR if [ -t 0 ]; then stty intr ^C fi # Alias section. Some usefull shortcuts ... alias crsi='/oracle/product/crs/bin/crs_stat -t' alias xxx=". ~/set_oracle_env.sh" alias oh='cd $ORACLE_HOME' echo "Tip: Enter xxx to change your current setting"

---------------The script ends here ------------

Page 70 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Install Patchset 10.2.0.2 on CRS Home Stop all oracle processes As oracle: srvctl stop database -d ZDA srvctl stop asm -n ora-rac-1 srvctl stop asm -n ora-rac-2 srvctl stop nodeapps -n ora-rac-1 srvctl stop nodeapps -n ora-rac-2 As root: /etc/init.d/init.crs stop Install binaries ./runInstaller

Page 71 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 72 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 73 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 74 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Restart Custerware >> ora-rac-1 root: /etc/init.d/init.crs start

Page 75 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Install Patchset 10.2.0.2 on ASM Home

Stop Nodeapps >> ora-rac-1: srvctl stop nodeapps -n ora-rac-1 srvctl stop nodeapps -n ora-rac-2

Install binaries ./runInstaller

Page 76 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 77 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 78 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 79 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 80 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Install Patchset 10.2.0.2 on DB Home

Verify service state All Services must be down. crs_stat -t Name Type Target State Host -----------------------------------------------------ora....A1.inst ora....A2.inst ora.ZDA.db ora....SM1.asm ora....-1.lsnr ora....c-1.gsd ora....c-1.ons ora....c-1.vip ora....SM2.asm ora....-2.lsnr ora....c-2.gsd ora....c-2.ons ora....c-2.vip

application application application application application application application application application application application application application

OFFLINE OFFLINE OFFLINE OFFLINE OFFLINE OFFLINE OFFLINE OFFLINE OFFLINE OFFLINE OFFLINE OFFLINE OFFLINE

OFFLINE OFFLINE OFFLINE OFFLINE OFFLINE OFFLINE OFFLINE OFFLINE OFFLINE OFFLINE OFFLINE OFFLINE OFFLINE

Install binaries ./runInstaller

Page 81 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 82 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 83 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 84 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 85 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 86 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Install Interim Patch 5117016 on DB Home Install with opatch >> ora-rac-3 as user oracle #cd /install/10gR2/5117016 #opatch apply -local

Page 87 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

>> ora-rac-4 as user oracle #cd /install/10gR2/5117016 #opatch apply -local

Page 88 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Change /etc/init.d/init.crsd: >> ora-rac-3 and ora-rac-4 as user root from: exec $ORA_CRS_HOME/bin/crsd $BOOTFLAG $* to : exec newtask $ORA_CRS_HOME/bin/crsd $BOOTFLAG $*

Setup system parameters on both Nodes: >> ora-rac-3 and ora-rac-4 as user root #projadd -p 61002 -c "Oracle RAC" -U oracle \

-K 'project.max-sem-ids=(priv,1024,deny)' \ -K 'process.max-sem-nsems=(priv,2048,deny)' \ -K 'project.max-shm-ids=(priv,256,deny)' \ -K 'project.max-shm-memory=(priv,18446744073709551615,deny)' \ MAH cat >> /etc/user_attr oracle::::project=RAC -d

Check all by reboot >> ora-rac-3 as user root #scshutdown -g0 -y

>> ora-rac-3 as user root ok boot

>> ora-rac-4 as user root ok boot

Start services required to perform database upgrade Restart the listener: srvctl start listener -n ora-rac-1 srvctl start listener -n ora-rac-2

Start asm: srvctl start asm -n ora-rac-1 srvctl start asm -n ora-rac-2

Upgrade database ZDA sqlplus / as sysdba STARTUP NOMOUNT ALTER SYSTEM SET CLUSTER_DATABASE=FALSE SCOPE=spfile; shutdown STARTUP UPGRADE SPOOL patch_10202.log @?/rdbms/admin/catupgrd.sql SPOOL OFF

Check log for errors: grep ORA- patch_10202.log

ORA-00600: internal error code, arguments: [qkacon:NFswrwo], [1], [], [], [], This is probably the bug 5089217, fixed in version 11.1 and should be fixed in version 10.2.0.3. Workaround bug 5089217:

Page 89 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Set "_optimizer_connect_by_cost_based" = false shutdown startup @?/rdbms/admin/utlrp.sql select distinct status from dba_objects; => valid ALTER SYSTEM SET CLUSTER_DATABASE=TRUE SCOPE=spfile; ALTER SYSTEM SET "_optimizer_connect_by_cost_based" = false SCOPE=spfile; shutdown; startup;

Restart cluster srvctl srvctl srvctl srvctl srvctl

stop stop stop stop stop

srvctl srvctl srvctl srvctl srvctl

start start start start start

database -d ZDA asm -n ora-rac-1 asm -n ora-rac-2 nodeapps -n ora-rac-1 nodeapps -n ora-rac-2 nodeapps -n ora-rac-1 nodeapps -n ora-rac-2 asm -n ora-rac-1 asm -n ora-rac-2 database -d ZDA

crs_stat -t

How to Setup IPMP as Cluster Interconnect Doc ID: Note:368464.1 In order to setup IPMP as cluster interconnect note 283107.1 should be used.

Page 90 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Add new node to cluster Create links to devices >> ora-rac-4 as root mkdir -p /dev/oracle ln -s /dev/rdsk/c9t600015D00022B3000000000000005012d0s1 /dev/oracle/ocr1 ln -s /dev/rdsk/c9t600015D00022B3000000000000005018d0s1 /dev/oracle/ocr2 ln -s /dev/rdsk/c9t600015D00022B300000000000000501Ed0s1 /dev/oracle/vote1 ln -s /dev/rdsk/c9t600015D00022B3000000000000005024d0s1 /dev/oracle/vote2 ln -s /dev/rdsk/c9t600015D00022B300000000000000502Ad0s1 /dev/oracle/vote3 ln -s /dev/rdsk/c9t600015D00022B3000000000000005042d0s1 /dev/oracle/asm01 ln -s /dev/rdsk/c9t600015D00022B3000000000000005048d0s1 /dev/oracle/asm02 ln -s /dev/rdsk/c9t600015D00022B300000000000000504Ed0s1 /dev/oracle/asm03 ln -s /dev/rdsk/c9t600015D00022B3000000000000005054d0s1 /dev/oracle/asm04 ln -s /dev/rdsk/c9t600015D00022B300000000000000505Ad0s1 /dev/oracle/asm05 ln -s /dev/rdsk/c9t600015D00022B3000000000000005060d0s1 /dev/oracle/asm06 ln -s /dev/rdsk/c9t600015D00022B3000000000000005066d0s1 /dev/oracle/asm07 ln -s /dev/rdsk/c9t600015D00022B300000000000000506Cd0s1 /dev/oracle/asm08 ln -s /dev/rdsk/c9t600015D00022B3000000000000005072d0s1 /dev/oracle/asm09 ln -s /dev/rdsk/c9t600015D00022B3000000000000005078d0s1 /dev/oracle/asm10

Change permissions >> ora-rac-4 as root chown oracle:dba /dev/oracle/*

Page 91 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Modify /etc/hosts Add IP-Adresses. >> ora-rac-1, ora-rac-2, ora-rac-4 as root

127.0.0.1 192.168.10.212 192.168.10.234 192.168.10.233 10.0.0.1 10.0.0.2 10.0.0.3 192.168.10.225

localhost ora-rac-1 ora-rac-1-1 ora-rac-1-2 ora-rac-1-priv ora-rac-1-priv-1 ora-rac-1-priv-2 ora-rac-1-vip

192.168.10.214 192.168.10.232 192.168.10.231 10.0.0.4 10.0.0.5 10.0.0.6 192.168.10.229

ora-rac-2 ora-rac-2-1 ora-rac-2-2 ora-rac-2-priv ora-rac-2-priv-1 ora-rac-2-priv-2 ora-rac-2-vip

192.168.10.223

ora-rac-3-vip

192.168.10.240 192.168.10.241 192.168.10.242 10.0.0.7 10.0.0.8 10.0.0.9 192.168.10.239

ora-rac-4 ora-rac-4-1 ora-rac-4-2 ora-rac-4-priv ora-rac-4-priv-1 ora-rac-4-priv-2 ora-rac-4-vip

loghost ora-rac-1.

loghost ora-rac-2.

loghost ora-rac-4.

Check SSH daemon >> ora-rac-4

ps -ef | grep sshd Enable user equivalence >> ora-rac-4 mkdir /export/home/oracle/.ssh chmod 700 ~/.ssh /usr/bin/ssh-keygen -t rsa /usr/bin/ssh-keygen -t dsa

scp authorized_keys ora-rac-1:/export/home/oracle/.ssh/authorized_keys cat id_dsa.pub >> authorized_keys cat id_rsa.pub >> authorized_keys scp authorized_keys ora-rac-1:/export/home/oracle/.ssh/authorized_keys scp authorized_keys ora-rac-2:/export/home/oracle/.ssh/authorized_keys

Page 92 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Perform cluster check /install/10gR2/clusterware/Disk1/cluvfy/runcluvfy.sh stage -pre crsinst -n ora-rac-1,ora-rac-2,ora-rac-4 -verbose

Workaround OUI Bug >> ora-rac-4 The OUI may fail if it does not find ssh and scp binaries in /usr/local/bin. >> ora-rac-4 as root mkdir -p /usr/local/bin ln -s /usr/bin/ssh /usr/local/bin/ssh ln -s /usr/bin/scp /usr/local/bin/scp

Install clusterware on new node >> ora-rac-1 /oracle/product/crs/oui/bin/addNode.sh

Page 93 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

<Enter node name>

Page 94 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 95 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 96 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

/oracle/product/crs/bin/crs_stat -t Name

Type

Target

State

Host

-----------------------------------------------------------ora....DA1.srv application

ONLINE

ONLINE

ora-rac-1

ora....atch.cs application

ONLINE

ONLINE

ora-rac-1

ora....A1.inst application

ONLINE

ONLINE

ora-rac-1

ora....A2.inst application

ONLINE

ONLINE

ora-rac-2

ora.ZDA.db

application

ONLINE

ONLINE

ora-rac-2

ora....SM1.asm application

ONLINE

ONLINE

ora-rac-1

ora....-1.lsnr application

ONLINE

ONLINE

ora-rac-1

ora....c-1.gsd application

ONLINE

ONLINE

ora-rac-1

ora....c-1.ons application

ONLINE

ONLINE

ora-rac-1

ora....c-1.vip application

ONLINE

ONLINE

ora-rac-1

ora....SM2.asm application

ONLINE

ONLINE

ora-rac-2

ora....-2.lsnr application

ONLINE

ONLINE

ora-rac-2

ora....c-2.gsd application

ONLINE

ONLINE

ora-rac-2

ora....c-2.ons application

ONLINE

ONLINE

ora-rac-2

ora....c-2.vip application

ONLINE

ONLINE

ora-rac-2

ora....c-4.gsd application

ONLINE

ONLINE

ora-rac-4

ora....c-4.ons application

ONLINE

ONLINE

ora-rac-4

ora....c-4.vip application

ONLINE

ONLINE

ora-rac-4

Install ASM home on new node /oracle/product/asm/oui/bin/addNode.sh

Page 97 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 98 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 99 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 100 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Install DB home on new node /oracle/product/db/oui/bin/addNode.sh

Page 101 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 102 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 103 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 104 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Create listener in DB Home /oracle/product/db/bin/netca

Page 105 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Fehler! Hier nur Node ora-rac-4 auswählen!

Page 106 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

LISTENER

LISTENER

Page 107 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

LISTENER

Page 108 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 109 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Create ASM instance on new node >> ora-rac-1 /oracle/product/asm/bin/dbca

Page 110 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 111 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

password: oracle

Page 112 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Create database instance on new node >> ora-rac-1 /oracle/product/db/bin/dbca

Page 113 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 114 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

password sys / oracle

Page 115 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 116 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 117 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Verify that group# is not already used (i.e. by standby redolos. In this case, alter group# to an unused value.

Page 118 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

End of Installation Due to time constraints, the installtion was not completed. To be continued.

Page 119 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Remove Node from Cluster Steps to perform a remove node from cluster following Note 317516.1 Remove instance >> ora-rac-1 /oracle/product/db/bin/dbca

Page 120 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 121 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 122 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 123 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Remove Listener >> ora-rac-1 /oracle/product/db/bin/dbca/netca

Page 124 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 125 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 126 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 127 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Clean-up the ASM >> ora-rac-1 srvctl stop asm -n ora-rac-4 srvctl remove asm -n ora-rac-4

>> ora-rac-4 rm -rf $ORACLE_HOME/admin/+ASM rm -f $ORACLE_HOME/dbs/*ASM*

Edit the /etc/oratab file and remove entries beginning with +ASM. Removing node from the database >> ora-rac-4 /oracle/product/db/oui/bin/runInstaller -updateNodeList ORACLE_HOME= /oracle/product/crs "CLUSTER_NODES={ora-rac-4}" -local

>> ora-rac-4 /oracle/product/crs/oui/bin/runInstaller #=> Choose de-install products and select the dbhome Screenshots here!

>> ora-rac-1 /oracle/product/crs/oui/bin/runInstaller -updateNodeList ORACLE_HOME=/oracle/product/crs "CLUSTER_NODES={node1,node2}"

Removing node from the clusterware: >> ora-rac-1 as root Page 128 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Get Remote Port number: cat /oracle/product/crs/opmn/conf/ons.config /oracle/product/crs/bin/racgons remove_config ora-rac-4:6200

>> ora-rac-4 as root /oracle/product/crs/install/rootdelete.sh

>> ora-rac-1 as root /oracle/product/crs/bin/olsnodes -n /oracle/product/crs/install/rootdeletenode.sh ora-rac-4,3

>> ora-rac-4 /oracle/product/crs/oui/bin/runInstaller -updateNodeList ORACLE_HOME=/oracle/product/crs/ "CLUSTER_NODES={ora-rac-4}" CRS=TRUE -local

/oracle/product/crs/oui/bin/runInstaller #=>

Choose Deinstall software and re-

move the CRSHOME

>> ora-rac-1 /oracle/product/crs/oui/bin/runInstaller -updateNodeList ORACLE_HOME=/oracle/product/crs "CLUSTER_NODES={ora-rac-1,ora-rac-2}" CRS=TRUE /oracle/product/crs/bin/olsnodes -n

Page 129 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Cluster Performance Tuning Overview and best practices for setting up a "zero brownout" configuration to achieve maximum high availability. More info in note 369101.1. Note: Patch 5024639 is not yet available and skipped therefore. Apply Brownout Patches This patches are installed by using OPatch on DB and ASM Home: 5186679 (This became patch 4960705) 5186674 (This became patch 5016142) 5127434 5127472 5127482 5176453 Set the CSS misscount to match the service level 1. Shut down the clusterware stack on all nodes but one 2. On the remaining node, run as root: $ORA_CRS_HOME/bin/crsctl set css misscount 5 $ORA_CRS_HOME/bin/crsctl set css disktimeout 200

3. Reboot the nodes for this to take affect. 4. Confirm the new values for the CSS parameters $ORA_CRS_HOME/bin/crsctl get css misscount $ORA_CRS_HOME/bin/crsctl get css disktimeout

Run CSS in Real Time 1. On each node as root use, crsctl set css priority 4 2. Restart each node in series, using crsctl stop crs followed by crsctl start crs Set the check_interval for VIP addresses >> ora-ra-1 as root 1. Generate the resource profile for the VIP resource cd $ORA_CRS_HOME/crs/profile crs_stat -p ora.ora-rac-1.vip > ora. ora-rac-1.vip.cap crs_stat -p ora.ora-rac-2.vip > ora. ora-rac-2.vip.cap

(current value of check_interval = 60 (default)) 2. Edit ora..vip.cap and set the parameter CHECK_INTERVAL to 5 Page 130 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

vi ora.ora-rac-1.vip.cap vi ora.ora-rac-2.vip.cap

3. Update the resource profile as the root user: crs_register -u ora.ora-rac-1.vip crs_register -u ora.ora-rac-2.vip

Note: The check interval should be long enough to sustain the refresh time for the VIP. Remove the dependency between the VIP and ASM 1. Generate the resource profile for the ASM instance resource cd $ORA_CRS_HOME/crs/profile crs_stat -p ora...asm > ora...asm.cap

2. Edit ora...asm.cap and remove 'ora..vip as the value of the REQUIRED_RESOURCES 3. Change the parameter REQUIRED_RESOURCES vi ora...asm.cap

REQUIRED_RESOURCES= 4. Update the resource profile crs_register -u ora..asm

Note: There is a hard dependency from the ASM to the VIP. This means that when the virtual IP is relo­ cated, ASM is shut down. It is fully supported to remove this CRS dependency, and is the default from 10.2.0.3 and later releases. Reduce the ping timeout 1. Open the $CRS_HOME/bin/racgvip script. (current value of PING_TIMEOUT = 3 (default)) 2. Edit to set PING_TIMEOUT to: PING_TIMEOUT="-w 1 -c 1"

This change shortens the delay between begin of CRS recovery and start of the VIP. Note: If any patches are applied that change the racgvip script, this variable would need to be changed again. alter alter alter alter alter alter

system system system system system system

set set set set set set

disk_asynch_io=true scope=spfile ; fast_start_parallel_rollback=high scope=spfile ; "_imr_disk_voting_interval"=0 scope=spfile; "_first_spare_parameter"=0 scope=spfile ; "_imr_splitbrain_res_wait"=20 scope=spfile ; "_db_mttr_advice"=off scope=spfile ;

Page 131 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Troubleshooting VIP Failover Adjusting Listener Configuration Make the local IP-address the first listening entry in the listener.ora. More infos in note 3589757 and 339107.1 >> All nodes Example on node ora-rac-1. Before: LISTENER_ORA-RAC-1 = (DESCRIPTION_LIST = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP) (HOST = ora-rac-1-vip) (PORT = 1521) (IP = FIRST)) (ADDRESS = (PROTOCOL = TCP) (HOST = 192.168.10.212) (PORT = 1521) (IP = FIRST)) ) ) [...]

After: LISTENER_ORA-RAC-1 = (DESCRIPTION_LIST = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP) (HOST = 192.168.10.212) (PORT = 1521) (IP = FIRST)) (ADDRESS = (PROTOCOL = TCP) (HOST = ora-rac-1-vip) (PORT = 1521) (IP = FIRST)) ) ) [...]

Upgrade Oracle Home of Swingbench Upgrade Oracle Home of Swingbench to Rel. 10.2.0. Adjust SQLNet settings Edit sqlnet.ora of client. Add / change the following line SQLNET.OUTBOUND_CONNECT_TIMEOUT=10

Set the ping timeout back to default 1. Open the $CRS_HOME/bin/racgvip script. (current value of PING_TIMEOUT = 3 (default)) 2. Edit to set PING_TIMEOUT to: PING_TIMEOUT="3"

Page 132 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Addendum Metalink Note 284752.1

Page 133 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Metalink Note 294430.1

Page 134 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 135 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 136 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 137 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 138 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 139 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Metalink Note 294430.1

Page 140 of 141

Document: Installation_Documentation_RAC_ASM.odt Authors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Page 141 of 141