Racwindows2003

RAC on Windows 2003 Julian Dyke Independent Consultant Web Version 1

© 2006 Julian Dyke

Introduction 

2

This presentation describes Windows specific RAC configuration issues including: 

Installation  Pre installation steps  Installing Oracle Clusterware  Installing Oracle Database Software  Patches  Post installation steps



Administration  Services  Networking

© 2006 Julian Dyke

Warning

3



This presentation is intended to supplement information in the Oracle documentation



It is NOT intended to replace the Oracle documentation



Most of the information contained in this presentation was correct for Oracle 10.2  Some information applies to earlier releases  Where appropriate this is highlighted in the slides

© 2006 Julian Dyke

Overview

4



Oracle RAC is available on Windows in  Standard Edition  Enterprise Edition



Standard Edition must use Automatic Storage Management (ASM)



Enterprise Edition can use  Automatic Storage Management (ASM)  Oracle Cluster File System (OCFS)

© 2006 Julian Dyke

Shared Oracle Home

5



Oracle Clusterware (CRS) MUST be installed in a local directory on each node



Oracle Database Software can be installed in a shared Oracle home on the cluster file system

© 2006 Julian Dyke

Installation

6

© 2006 Julian Dyke

Pre-Installation Steps

7

© 2006 Julian Dyke

Pre-Installation Steps 

Pre-Installation tasks include:       

8

Check public interface is first in bind order Disable Windows Media Sensing for TCP/IP Disable Write Caching on Shared Storage Enable Auto-mounting Local drives must be accessible from all nodes Environment variables System clocks must be synchronized

© 2006 Julian Dyke

Public Interface Bind Order

9



The public interface must be the first in the bind order



To check this  Start Windows Explorer  Right click on My Network Places and choose Properties  In the Advanced menu click Advanced Settings...  In the Adapters and Bindings tab  If the public interface is not the first name listed  Click the arrow to move it to the top of the list  Click OK to save the setting

© 2006 Julian Dyke

Windows Media Sensing for TCP/IP  

Windows Media Sensing for TCP/IP must be disabled To check this  Run REGEDT32.EXE  Navigate to HKEY_LOCAL_MACHINE\System\CurrentControlSet\ Services\Tcpip\Parameters  Add the following registry entry:  Right click on right hand pane and select  New -> DWORD value Value Name: Value:



10

DisableDHCPMediaSense 1

NOTE - It is recommended that you backup the registry before making changes using REDEDT32.EXE / REGEDIT.EXE

© 2006 Julian Dyke

Write Caching 

Write Caching must be disabled on shared storage



To disable write caching at operating system level: 

 





11

Start->Settings->Control Panel->Administrative Tools-> Computer Management->Device Manager->Disk Drives Expand the disk drives list Double-click the first drive listed  On the Disk Properties tab uncheck the write cache enabled option Repeat for all remaining disk drives

Not necessary if using HP Array Manager as write caching is automatically disabled at operating system level

© 2006 Julian Dyke

Disk Manager 

To start the Disk Management tool:  Start->Administrative Tools->Computer Management-> Storage->Disk Management



Alternatively the Disk Management tool can be started from the command line using: C:> DISKMGMT.MSC

12

© 2006 Julian Dyke

Auto-Mounting 

Auto-mounting must be enabled when using  RAC with raw partitions  RAC with cluster file system  Oracle Clusterware  Logical drives for ASM



Auto-mounting must be enabled on each node in the cluster



To check if auto-mounting is enabled use: C:> DISKPART DISKPART> AUTOMOUNT Automatic mounting of new volumes disabled

13

© 2006 Julian Dyke

Auto-Mounting 

To enable auto-mounting use: C:> DISKPART DISKPART> AUTOMOUNT ENABLE Automatic mounting of new volumes enabled DISKPART> EXIT

14



Repeat on each node in the cluster



Restart all nodes after configuring this setting

© 2006 Julian Dyke

Local Drive Accessibility 

Local drives must be accessible from all nodes



To check local drive accessibility:  For each node  For each local drive to be used by the Oracle installation  Use NET USE command to confirm accessibility



For example if Oracle will be installed on E$ in a two-node cluster containing nodes RAC1 and RAC2 On RAC1:



NET USE \\RAC2\E$ 

On RAC2: NET USE \\RAC1\E$

15

© 2006 Julian Dyke

Environment Variables

16



On each node the following environment variables must be set  TEMP  TMP



By default these are set to  %USERPROFILE%\Local Settings\Temp



Oracle recommends these are redefined as follows  TEMP=C:\TEMP  TMP=C:\TMP



These settings should be the same on each node

© 2006 Julian Dyke

Time Synchronization 





It is recommended that the system clocks are synchronized between all nodes in the cluster  appears to be less important in Oracle 10g Release 2 than in previous RAC versions To synchronize the system clocks enable the Windows Time Service on one node The current time server for a node can be discovered using: NET TIME /QUERYSNTP The current SNTP server is: node2



The current time server for a node can be set using: NET TIME /SETSNTP:



For example: NET TIME /SETSNTP:SGHRAC1

17

© 2006 Julian Dyke

Time Synchronization 

The current time on a node can be discovered using NET TIME \\NODE1 Current time at \\NODE1 is 20/02/2006 10:50



To initially synchronize the time use NET TIME \\NODE1 /SET Current time at \\NODE1 is 20/02/2006 10:56 The current local clock is 20/03/2006 10:54 Do you want to set the local computer's time to match the time at \\node1? (Y/N) [Y]: y The command completed successfully.

18

© 2006 Julian Dyke

Oracle Clusterware Installation 19

© 2006 Julian Dyke

Oracle Clusterware Installation

20



To install Oracle Clusterware  Login as Local Administrator only



The documentation states that Clusterware can be installed using Domain Administrator  Installation currently fails when attempting to start Clusterware services

© 2006 Julian Dyke

Oracle Clusterware Installation

21



In Oracle 10.1 Oracle Clusterware was called Cluster Ready Services (CRS)



In Oracle 10.1 to set up CRS run CRS/SETUP.EXE



DO NOT run CRS/INSTALL/SETUP.exe  This will cause the installation to fail.  See Metalink Note 277688.1 - CRS Install: Failed to locate service OracleCSSService on second node



Also make sure that the install path does not include any spaces



In Oracle 10.2 there is only one SETUP.EXE program

© 2006 Julian Dyke

Oracle Clusterware Installation

22



During installation the OracleClusterPreInstService is installed on each node to provide a list of available drives



This service uses ORACLECLUSTERPREINSTSERVICE.EXE

© 2006 Julian Dyke

Oracle Clusterware Installation 

To manually delete a failed clusterware installation:



Stop and delete all Oracle services. For example: SC STOP OracleCRService SC DELETE OracleCRService

23



Delete contents of Oracle Clusterware home directory



If required delete Oracle inventory directory  C:\Program Files\Oracle



Use REGEDIT to remove Oracle keys from registry  HKEY_LOCAL_MACHINE\Software\Oracle



Use Oracle Object Manager to delete links to OCR and Voting disk

© 2006 Julian Dyke

Oracle Object Manager 

If using raw devices  Oracle Object Manager is used to create links for  OCR  Voting Disk



Pathname is  %ORACLE_HOME%\bin\GUIObjectOBJManager



Link names are:  ocrcfg  votedsk1



Executed automatically by the OUI during installation Must be executed manually during manual deinstallation



24

© 2006 Julian Dyke

Oracle Object Manager 

25

For example

© 2006 Julian Dyke

Oracle Database Software Installation 26

© 2006 Julian Dyke

Database Installation 

27

The script %ORACLE_HOME%\bin\SelectHome.bat must be executed on all remote nodes to activate the following products:  Oracle Data Provider for .NET  Oracle Provide for OLE DB  Oracle Objects for OLE  Oracle Counters for Windows Performance Monitor  Oracle Administration Assistant

© 2006 Julian Dyke

Automatic Storage Management 28

© 2006 Julian Dyke

Automatic Storage Management  

29

ASM disks must be stamped before they can be used by ASM Can be stamped  Using ASMTOOL command line utility  Using ASMTOOLG GUI utility  During ASM instance creation process in DBCA

© 2006 Julian Dyke

ASMTOOLG

30

© 2006 Julian Dyke

Cluster File System

31

© 2006 Julian Dyke

Cluster File System

32



If selected in the Oracle Universal Installer (OUI) session then the Oracle Cluster File System is installed automatically



It is not necessary to download OCFS separately (as would be the case in Linux for example)

© 2006 Julian Dyke

Cluster File System  

To create partitions use OCFSFORMAT Recommended block sizes are: Software (Shared ORACLE_HOME) Database files



For example: OCFSFORMAT /l R:/C 1024 /V REDO

33

© 2006 Julian Dyke

4 1024

Patches

34

© 2006 Julian Dyke

Patches 

In addition to normal patch sets additional cumulative patches are issued for Oracle on Windows platforms



For example  10.1.0.3 Patch 5  10.2.0.1 Patch 3



Install the latest cumulative patch for the platform



For example in Oracle 10.2 available patches include  10.2.0.1.0 Patch 2 (10.2.0.1.2P) Patch 4751342  10.2.0.1.0 Patch 3 (10.2.0.1.3P) Patch 4751539  10.2.0.1.0 Patch 4 (10.2.0.1.4P) Patch 4923768 At the time of writing Patch 4 was recommended



35

© 2006 Julian Dyke

Patches

36



Current Windows patch sets are documented on Metalink



For Oracle 10.1  See Metalink Note 276548.1 - 10.1.0.x Oracle Database and Networking Patches for Microsoft Platforms



For Oracle 10.2  See Metalink Note 342443.1 10.2.0.x Oracle Database and Networking Patches for Microsoft Platforms



Both reference  Note 161549.1 - Oracle Database Server and Networking Patches for Microsoft Platforms

© 2006 Julian Dyke

Patch Set Installation  



 

37

Installing Oracle 10.1.0.3 Patch 5 The patch set should be installed in both the Clusterware home and the Oracle database home Run Setup.exe in the Oracle database home After installing the CRS Patch on each node stop all CRS services (Service Management)  OracleCRService  OracleEVMService  OracleCSService  OracleClusterVolumeService Run C:\Oracle\product\10.1\crs\install\patch10103.bat After installing Oracle database software patch on one node, run $ORACLE_HOME\bin\SelectHome.bat on remaining nodes

© 2006 Julian Dyke

Patch Set Installation  

 

 

Installing Oracle 10.2.0.1 Patch 3 The patch set should only be installed in the Oracle database home and does not affect CRS Patch is installed using OPATCH OPATCH should be upgraded to 10.2.0.1.1 or above  Patch# : 4898608 Stop all services using %ORACLE_HOME% Install 10.2.0.1.3 using OPATCH APPLY

38

© 2006 Julian Dyke

Patch Set Installation  

 

 

Installing Oracle 10.2.0.1 Patch 4 The patch set should be installed in both the Clusterware and RDBMS home directories  Patch 4: 4923768 Patch is installed using OPATCH OPATCH should be upgraded to 10.2.0.1.1 or above in both Clusterware and RDBMS home directories  Patch# : 4898608 Stop all services using %ORACLE_HOME% Install 10.2.0.1 Patch 4 using OPATCH APPLY

39

© 2006 Julian Dyke

Post-Installation Steps

40

© 2006 Julian Dyke

Post Installation 

41

Post installation tasks include:  Make files node-specific (Shared Oracle home only)  Create ORA_DBA group on remaining nodes  Set Log on as a Batch Job privilege for Enterprise Manager users

© 2006 Julian Dyke

Make Files Node-Specific 

If using shared Oracle Home on OCFS file system  After database creation using DBCA the HC files must be made node-specific.  Applies to Oracle 10.1 (at least)



On each host these files have the pathname: %ORACLE_HOME%\DATABASE\HC_.DAT



For example on instance RAC1 %ORACLE_HOME%\DATABASE\HC_RAC1.DAT

42

© 2006 Julian Dyke

Make Files Node-Specific 

For example on node node1: srvctl stop instance -d RAC -i RAC1 ocfsutil /c NodeSpecificFile /o create /m h: /p oracle\product\10.1.0\db\database\hc_rac1.dat srvctl start instance -d RAC -i RAC1



43

Repeat for remaining instances

© 2006 Julian Dyke

Log on as a batch job 





44

Users must have the Log on as a batch job privilege for Enterprise Manager to work correctly  Start->Administrative Tools->Local Security Policy->  Security Settings->Local Policies->  User Rights Assignment->Log on as a batch job  Assign privilege for each administrative users  Repeat on each node There is a description of this problem in Metalink  Note 279765.1 "Error Message 'RemoteOperationException: ERROR: Wrong password for users' when trying to startup or shutdown the database from the Enterprise Manager console" There is a description of the resolution in Metalink  Note 109188.1 How to Set "Logon as a Batch Job" Privileges on Windows 2000 Systems

© 2006 Julian Dyke

ORA_DBA Group

45



Administrators must be members of this group on each node in order to use / AS SYSDBA



By default ORA_DBA group is only created on installation node  Not created on remaining nodes  Must be created manually



See Metalink Notes  177354.1 RAC: Connect AS SYSDBA Fails with ORA-01031  77665.1 - Guide to setup/enable OS authentication to Connect using OS user account

© 2006 Julian Dyke

ORA_DBA Group 

46

To create new ORA_DBA group and add users  Start->Administrative Tools->Computer Management->  System Tools->Local Users and Groups->Groups  Right click and select New  Name is ORA_DBA  Description is Oracle DBA Group  Add user e.g. XYZDOM\abc.admin

© 2006 Julian Dyke

Administration

47

© 2006 Julian Dyke

Environment Variables

48



Environment variables are stored in the registry



Can be set using:  Start -> Control Panel -> System ->  Advanced-> Environment Variables

© 2006 Julian Dyke

Registry Keys 

Oracle registry entries are stored in 



The inventory location is defined in 



C:\Program Files\Oracle\Inventory

The Oracle Cluster Repository location is stored in 

49

HKEY_LOCAL_MACHINE\Software\Oracle\inst_loc

The default inventory location is: 



HKEY_LOCAL_MACHINE\Software\Oracle

HKEY_LOCAL_MACHINE\Software\Oracle\OCR\ocrconfig_loc

© 2006 Julian Dyke

Registry 

By default CRS-related keys are stored in 



For example  



HKEY_LOCAL_MACHINE\Software\Oracle\KEY_OraDb10g_home

For example   

50

ORA_CRS_HOME ORACLE_HOME

By default database keys are stored in 



HKEY_LOCAL_MACHINE\Software\Oracle\KEY_OraCr10g_home

ORACLE_BASE ORACLE_HOME ORACLE_SID

© 2006 Julian Dyke

Scheduler

51



Windows contains a job scheduler



To schedule a job e.g. overnight backup use  Start->Control Panel->Scheduled Tasks->  Add Scheduled Task



A wizard will step through the configuration steps

© 2006 Julian Dyke

Services

52

© 2006 Julian Dyke

Services

53



Oracle uses Windows services  Similar to Unix daemons  Configured in windows registry



All Oracle services have the "Oracle" prefix



The following services are created during Clusterware installation  Oracle Object Service  OracleClusterVolumeService  OracleCRService  OracleCSService  OracleEVMService

© 2006 Julian Dyke

Services

54



The following services are created by DBCA during ASM instance configuration  OracleOraDb10g_home1TNSListenerLISTENER_<node>  OracleASMService



The following services are created by DBCA during database instance configuration  OracleJobScheduler  OracleService  OracleDBConsole

© 2006 Julian Dyke

Services 

55

Windows services can be controlled in three ways  Using the GUI  Using the NET command  Using the SC command

© 2006 Julian Dyke

Services 

To control services using the GUI  Start->Administrative Tools->Services  Click on a service from the list of available services



Alternatively the Service Management tool can be started from the command line using: C:> SERVICES.MSC



56

Single click allows you to control the service including:  Start / Stop  Pause/Continue  Restart

© 2006 Julian Dyke

Services 

57

Double click allows you to manage the service including:  Start / Stop / Pause / Continue  Specify the startup type  Automatic  Manual  Disabled  Change the login type  Local System Account  Named User  Specify recovery parameters  Specify dependencies

© 2006 Julian Dyke

Services  

Services can be controlled using the NET command To check which services are running use: NET START



Use the MORE command to control the output NET START | MORE





58

Note that this command only prints information about services which are currently running Use the SC command to list services which are currently stopped

© 2006 Julian Dyke

Services 

To start a service use: NET START <service_name>



For example: NET START OracleServiceRAC1



To stop a service use: NET STOP <service_name>



For example: NET STOP OracleServiceRAC1

59

© 2006 Julian Dyke

Services   

Services can also be controlled using the SC command Available on Windows 2003 and Windows XP To print the help message use: SC

 

60

This prints generic help and asks if you wish to see help for the QUERY and QUERYEX options

© 2006 Julian Dyke

Services 

To start a service use: SC START <service_name>



For example: SC START OracleServiceRAC1



To stop a service use: SC STOP <service_name>



For example: SC STOP OracleServiceRAC1

61

© 2006 Julian Dyke

Services 

To query the status of all running services use: SC QUERY



To query the status of all services use: SC QUERY STATE= ALL

 



Note that the syntax for this command is very sensitive You must have  No space between STATE and =  Space between = and ALL Use the MORE command to control the output: SC QUERY STATE= ALL | MORE

62

© 2006 Julian Dyke

Services 

To query the status of an individual service use: SC QUERY <service_name>



For example: SC QUERY OracleServiceRAC1

C:\> SC QUERY OracleServiceRAC1 SERVICE_NAME: OracleServiceRAC1 TYPE : 10 WIN32_OWN_PROCESS STATE : 4 RUNNING (STOPPABLE, PAUSABLE, ACCEPTS_SHUTDOWN) WIN32_EXIT_CODE : 0 (0x0) SERVICE_EXIT_CODE : 0 (0x0) CHECKPOINT : 0x0 WAIT_HINT : 0x0

63

© 2006 Julian Dyke

Services 

To delete an individual service use: SC DELETE <service_name>



For example: SC DELETE OracleCRService



64

This may be necessary if you need to:  tidy up after a failed installation  remove Oracle manually

© 2006 Julian Dyke

Tasklist 

To list relationship between processes and services use: TASKLIST /SVC



For example: C:\> TASKLIST /SVC Image Name --------------------evmd.exe crsd.exe TNSLSNR.EXE OracleOBJService.exe oracle.exe OcfsFindVol.exe nmesrvc.exe oracle.exe ocssd.exe

65

© 2006 Julian Dyke

PID ---2236 2216 4936 832 1964 1552 1580 1628 2412

Services -------------OracleEVMService OracleCRService OracleOraDb10g_home1TNSListenerLISTENER_NODE1 Oracle Object Service OracleASMService+ASM OracleClusterVolumeService OracleDBConsolePDS1 OracleServicePDS1 OracleCSService

Networking

66

© 2006 Julian Dyke

Hosts

67



In Windows the hosts file is typically located in  C:\Windows\System32\Drivers\etc



The format is similar to Unix. For example:

© 2006 Julian Dyke

RAC1 RAC2

10.47.0.101 10.47.0.102

RAC1-vip RAC2-vip

10.47.0.201 10.47.0.202

RAC1-priv RAC2-priv

192.168.0.1 192.168.0.2

HOSTNAME 



The name of the current node can be obtained at the command line using the HOSTNAME command For example C:\> HOSTNAME node1

68

© 2006 Julian Dyke

PING 

 

As in Unix the PING utility can be used to verify network connections By default only four ICMP probes are performed For example: C:\> PING NODE2 Pinging NODE2 [10.131.60.202] with 32 bytes of data: Reply Reply Reply Reply

from from from from

10.131.60.202: 10.131.60.202: 10.131.60.202: 10.131.60.202:

bytes=32 bytes=32 bytes=32 bytes=32

time<1ms time<1ms time<1ms time<1ms

TTL=128 TTL=128 TTL=128 TTL=128

Ping statistics for 10.131.60.202: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss). Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms

69

© 2006 Julian Dyke

PING  



By default PING uses a packet size of 32 bytes. The default TCP/IP packet size is around 1400 bytes  Can be increased by configuring jumbo frames  Maximum packet size with jumbo frames is about 9000 bytes (network card and switch specific) To confirm that the network has been correctly configured to use jumbo frames use the ping command: PING -L <packet_size> -F

 

where <packet_size> is the size in bytes For example PING -L 1600 -F RAC1

70

© 2006 Julian Dyke

PING 

For example assume the default TCP/IP configuration C:\> PING -L 1400 -F NODE2 Pinging NODE2 [10.131.60.202] with 1400 bytes of data: Reply Reply Reply Reply

from from from from

10.131.60.202: 10.131.60.202: 10.131.60.202: 10.131.60.202:

bytes=1400 bytes=1400 bytes=1400 bytes=1400

time<1ms time<1ms time<1ms time<1ms

TTL=128 TTL=128 TTL=128 TTL=128

......

C:\> PING -L 1600 -F NODE2 Pinging NODE2 [10.131.60.202] with 1600 bytes of data: Packet Packet Packet Packet

needs needs needs needs

..... 71

© 2006 Julian Dyke

to to to to

be be be be

fragmented fragmented fragmented fragmented

but but but but

DF DF DF DF

set set set set

IPCONFIG   

IPCONFIG can be used to verify network configurations Similar to ifconfig on Unix systems For example: C:\> IPCONFIG Windows IP Configuration Ethernet adapter Public Team: Connection-specific IP Address. . . . . Subnet Mask . . . . IP Address. . . . . Subnet Mask . . . . Default Gateway . .

DNS . . . . . . . . . .

Suffix . . . . . . . . . . . . . . . . . . . .

. . . . . .

: : : : : :

. . . .

: : 192.168.200.1 : 255.255.255.0 :

10.131.60.201 255.255.0.0 10.131.60.101 255.255.0.0 10.131.60.254

Virtual IP Address Public IP Address

Ethernet adapter Private Team: Connection-specific IP Address. . . . . Subnet Mask . . . . Default Gateway . . 72

© 2006 Julian Dyke

DNS . . . . . .

Suffix . . . . . . . . . . . .

Private IP Address

ARP 

 

As on Unix systems ARP can be used to view the current contents of the ARP cache Maps IP addresses to MAC (Ethernet) addresses For example: C:\> ARP-A

73

Interface: 10.130.60.230 --- 0x10005 Internet Address Physical Address 10.131.60.96 00-0b-cd-3a-19-48 10.131.60.202 00-15-60-55-12-81 10.131.60.203 00-0b-cd-6d-3c-5b 10.131.60.207 00-0f-20-f9-c9-ba 10.131.60.209 00-0f-20-f9-84-65 10.131.60.210 00-0b-cd-41-b5-72

Type dynamic dynamic dynamic dynamic dynamic dynamic

Interface: 192.168.200.1 --- 0x10006 Internet Address Physical Address 192.168.200.2 00-11-0a-5a-42-63

Type dynamic

© 2006 Julian Dyke

Acknowledgements

74



Thanks for help in the preparation of this presentation to:  Stephen Bendall  Tak Tang  John Plowman  Lee Cashmore  Dirk Schmidt  Davy Witvrouwen



Julian Dyke  Independent Consultant  Web Site: www.juliandyke.com  Email: [email protected]  Mobile: +44 7917 360777

© 2006 Julian Dyke