Building Rac Test Environment On Vmware

Building a RAC Test Environment on VMWare For Free Dan Norris d [email protected] i @ i http://www.dannorris.com/ Thanks to Tim Hall @ oracle-base.com

Agenda •Who is Dan? •VMWare Orientation •Importance of prerequisite checking •Configuring node #1 •Cloning (VM) node #1 •Installing Clusterware •Installing ASM and Database •Creating Database •Common C Ch Challenges ll and d IIssues

2

Who is Dan? • Virgo • Scuba Diver (PADI Advanced OW, Nitrox) • Over 21, under 35 g • Oracle DBA & UNIX Admin background • Certifiable: OCM, ACE Director, RHCE • Consultant, mostly fixing things that are broken • Active community participant: RAC SIG, SIG Council, DBA Track Manager, blogger, tweeter • ESA Practice Manager at Piocon Technologies

3

VMWare Orientation •Desktop Products –VMWare VMWare Workstation: “Full” Full product, product $ –VMWare Player: Free, no creation, just use

•Server Server Products –VMWare Infrastructure (ESX Server): $$$, robust,, clustering g capable, p , OS replacement p –VMWare VirtualCenter: $$, management tool for whole environment, Windows –VMWare Server: free, limited, on top of OS

4

VMWare Server Hardware •Almost any server will do, mine is: –Dell SC440: 1 x dual-core 3.0 Ghz –2 x 500Gb internal SATA drives, using LVM –4 Gb RAM

•I use Linux (OEL 5) as host OS •Let Linux use LVM (default), allows disk space to be pooled for 900+ 900 Gb on /u01 •VMWare Server 1.0.5 •Two Virtual Networks: 1 bridged g & 1 hostonly

5

RAC: System Architecture Node1 rac1

Local Storage

Cluster Cl t Interconnect

Shared Storage

Node2 rac2

Local Storage

6

First: Build Node1 Node1

Local Storage

Node2

Shared Storage

Local Storage

7

Node1: Create New VM

8

Node1: Create New VM

9

Node1: Create New VM

10

Node1: Create New VM

11

Node1: Create New VM

12

Node1: Create New VM

13

Node1: Create New VM

14

Node1: Create New VM

15

Node1: Create New VM (Pretend it says RAC1)

16

Node1: Create New VM

17

Node1: Create New VM

18

Node1: Create New VM

19

Node1: Create New VM

20

Node1: Create New VM

21

Node1: Create New VM

22

Node1: Create New VM

23

Install Node1 OS •Plan the network addresses for public and private networks •Start the VM and the CD image should boot •Install the OS OS, ensuring enough swap and /tmp •Assign A i static t ti addresses dd tto each h iinterface t f •See whitepaper for package and setup d t il details

24

Post-Install Steps on Node1 •Modify /etc/hosts •Set Set kernel parameters •Set limits in /etc/security/limits.conf •Add groups (oinstall, dba), oracle user •Set Set up oracle user’s user s profile (shell limits, ORACLE_BASE) •Set up SSH user equivalency

25

Prerequisite Checking •Most issues arise from misconfiguration of OS, OS packages packages, networks networks, or storage •Tools exist to help check prerequisites –CVU CVU –RDA

•Some basic networking knowledge helps •Recipes p can help, p, be sure yyou understand why, not just follow steps

26

Prerequisite Checking - RDA •Metalink Note 314422.1 will orient you to the RDA and has links for download •Download the .zip file, unzip and run (as root or oracle): –./rda.sh –T hcve –provide the release you want to check –answer additional questions –obtain results in ./output/RDA* files

27

Prerequisite Checking - RDA

28

Prerequisite Checking - RDA

29

Prerequisite Checking - CVU •http://www.oracle.com/technology/produ cts/database/clustering/cvu/cvu downloa cts/database/clustering/cvu/cvu_downloa d_homepage.html •Download Download a .zip zip file file, unzip itit, run it: –cluvfy stage -list

30

Prerequisite Checking - CVU

On AMD64 platforms, some packages will be listed twice, once for i386 and once for AMD64

31

On AMD64 platforms, some packages will be listed twice, once for i386 and once for AMD64

32

Done: Build Node1 Node1

Local Storage

Node2

Shared Storage

Local Storage

33

Next: Add Shared Storage Node1

Local Storage

Node2

Shared Storage

Local Storage

34

Node1: Add Shared Storage •Shut down guest OS in Node1 •On On the VMWare server ser er host OS OS, create a new directory for shared disk files: mkdir –p p /u01/vmware/vms/rac /u01/vmware/vms/rac-disks disks

•Back at the VMWare Server Console, add dd new disks di k to t the th Node1 N d 1 (RAC1) VM as shown in the following example.

35

Node1: Add Shared Storage

36

Node1: Add Shared Storage

37

Node1: Add Shared Storage

38

Node1: Add Shared Storage

39

Node1: Add Shared Storage

40

Node1: Add Shared Storage

41

Node1: Add Shared Storage •Repeat to create the following: Size

Virtual Vi t l Device D i Node

/u01/vmware/vms/rac-disks/ocr1.vmdk

10 Gb

SCSI 1:0

/u01/vmware/vms/rac-disks/voting1.vmdk

10 Gb

SCSI 1:1

/u01/vmware/vms/rac-disks/asm1.vmdk

10 Gb

SCSI 1:2

/u01/vmware/vms/rac-disks/asm2.vmdk

10 Gb

SCSI 1:3

/u01/vmware/vms/rac-disks/asm3.vmdk

10 Gb

SCSI 1:4

Path

42

Node1: Add Shared Storage •Now for the tricky part: hand-editing VMWare configuration file file. •The changes we make enable the disks to be shared with other VMs VMs. •Add or update the following parameters i th in the “/ “/u01/vmware/vms/RAC1/Red / / / / Hat Enterprise Linux 4.vmx” file

43

Node1: Add Shared Storage disk.locking = "FALSE" diskLib.dataCacheMaxSize = "0" diskLib.dataCacheMaxReadAheadSize = "0" diskLib.dataCacheMinReadAheadSize = "0" diskLib.dataCachePageSize = "4096" diskLib.maxUnsyncedWrites = "0" scsi1.sharedBus i1 h dB = "VIRTUAL"

44

Node1: Add Shared Storage scsi1:0.deviceType scsi1:0.redo = "" scsi1:1.deviceType scsi1:1.redo = "" scsi1:2 deviceType scsi1:2.deviceType scsi1:2.redo = "" scsi1:3.deviceType yp scsi1:3.redo = "" scsi1:4.deviceType scsi1:4.redo = ""

= "disk" = "disk" = "disk" = "disk" = "disk"

45

Node1: Add Shared Storage •Boot Node1 again and partition the disks to create partition 1 as the whole disk •“Quick and Dirty” setup: add this to /etc/rc local: /etc/rc.local: chown oracle:oinstall /dev/sd[b-f]1 chmod 600 /dev/sd[b-f]1 / /

46

Done: Add Shared Storage Node1

Local Storage

Node2

Shared Storage

Local Storage

47

Next: Clone Node1 to Make Node2 Node1

Local Storage

Node2

Shared Storage

Local Storage

48

Create Node2: Clone Node1 •Node2 is a VMWare clone of Node1 •Shut Sh t do down n Node1 nicel nicely •On host OS, do cp –Rp /u01/vmware/vms/RAC1 \ /u01/vmware/vms/RAC2 mv /u01/vmware/vms/RAC2/RAC1.vmx \ /u01/vmware/vms/RAC2/RAC2 vmx /u01/vmware/vms/RAC2/RAC2.vmx

49

Create Node2: Clone Node1 •Boot Node2 (leave Node1 down) •During power on on, choose Create when prompted about handling the identifer. •Modify –/etc/sysconfig/network –/etc/sysconfig/network-scripts/ifcfg-eth0 –/etc/sysconfig/network-scripts/ifcfg-eth1 / / fi / k i /if f h1

•From the console, run “service network restart”

50

Create Node2: Clone Node1 •Boot Node2 (leave Node1 down) •During During power on on, choose Create when prompted about handling the identifer. •Modifyy – /etc/sysconfig/network – /etc/sysconfig/network-scripts/ifcfg-eth0 / / y g/ p / g – /etc/sysconfig/network-scripts/ifcfg-eth1

•From the virtual machine’s console, run “service network restart” •You can now power up Node1

51

Prerequisite Checking - CVU •Finally able to complete more prerequisite checking •As the oracle user, run CVU to confirm private networking networking, and user equivalence: ./cluvfy / l f stage t –post t hwos h –n rac1,rac2 1 2

52

53

Prerequisite Checking - CVU •Shared storage checks don’t work on VMWare since the disks do not support SMART (used by CVU to verify disk accessibility) •Next, check the pre-crsinst stage like this: ./cluvfy / l f –pre crsinst i t –n rac1,rac2 1 2 \ –r 11gR1

54

Prerequisite Checking - CVU

55

Prerequisite Checking - CVU

56

Prerequisite Checking - CVU •With all CVU checks complete and successful we can move ahead to successful, installing clusterware!

57

Done: Node2 Created Node1

Local Storage

Node2

Shared Storage

Local Storage

58

Next: Install Clusterware Node1

Local Storage

Cluster Cl t Interconnect

Shared Storage

Node2

Local Storage

59

Installing Clusterware •Clusterware installation performed from one node (Node1) (Node1). •Ensure /etc/hosts is configured with host addresses VIP addresses addresses, addresses, and private addresses

60

Installing Clusterware

61

Installing Clusterware

62

Installing Clusterware

63

Installing Clusterware

64

Installing Clusterware

65

Installing Clusterware

66

Installing Clusterware

67

Installing Clusterware

68

Installing Clusterware

69

Installing Clusterware

70

Installing Clusterware

71

Installing Clusterware

72

Installing Clusterware Note: Do not run these scripts in parallel. They should be run sequentially.

73

Installing Clusterware

74

Installing Clusterware

Now go back and click OK on the installer prompt p p box.

75

Installing Clusterware

76

Installing Clusterware

77

Installing Clusterware

78

Installing Clusterware •Once the installation is complete, you can check the status with crs_stat crs stat -t t

79

Finished: Install Clusterware Node1

Local Storage

Cluster Cl t Interconnect

Shared Storage

Node2

Local Storage

80

Next: Install ASM and Database Node1 +ASM1

Local Storage

Cluster Cl t Interconnect

Shared Storage

Node2 +ASM2

Local Storage

81

Installing ASM & Database

82

Installing ASM & Database

83

Installing ASM & Database

84

Installing ASM & Database

85

Installing ASM & Database

86

Installing ASM & Database

87

Installing ASM & Database

88

Installing ASM & Database

89

Installing ASM & Database

90

Installing ASM & Database

91

Installing ASM & Database

92

Installing ASM & Database

93

Installing ASM & Database

94

Installing ASM & Database

95

Installing ASM & Database

96

Installing ASM & Database

97

Installing ASM & Database

98

Finished: Install ASM and Database Node1 +ASM1

Local Storage

Cluster Cl t Interconnect

Shared Storage

Node2 +ASM2

Local Storage

99

Next: Create Database rac1 Node1 +ASM1

Local Storage

Cluster Cl t Interconnect

rac2 Node2 +ASM2

Shared Storage

Local Storage

100

Create Database

101

Create Database

102

Create Database

103

Create Database

104

Create Database

105

Create Database

106

Create Database

107

Create Database

108

Create Database

109

Create Database

110

Create Database

111

Create Database

112

Create Database

113

Create Database

114

Create Database

115

Create Database

116

Create Database

117

Create Database

118

Create Database

119

Create Database

120

Create Database

121

Create Database …after a rather long pause while i t instances are started…(drumroll, t t d (d ll please) l )

122

Done: Create Database rac1 Node1 +ASM1

Local Storage

Cluster Cl t Interconnect

rac2 Node2 +ASM2

Shared Storage

Local Storage

123

Common Issues #1: Firewalls •Firewall and SELinux should be disabled •Can Can be done d during ring install with ith some versions of Linux. If not, then… •Set S t SELINUX=disabled in i /etc/sysconfig/selinux •chkconfig iptables off •Reboot Reboot

124

Common Issues #2: Storage •Shared Storage misconfiguration – different names for disks on each node •For RHEL or OEL 5, use udev for privileges and device mapping •Note Linux v4 and v5 handle things differently •Ensure you use proper device in multipath configurations (not applicable in VMWare)

125

Common Issues #2: Storage, cont’d Note

Title

443996.1 How to map raw device on RHEL5 and OEL5 414897.1

How to setup /etc/udev/rules.d for OCR and Voting for RAC on SLES10/RH5/EL5

371814.1 Using udev with Oracle Architecture (RAC & ASM) – Red Hat 4.0 456239.1 Understanding device-mapper in Linux 2.6 kernel 357472.1 Configuring device-mapper for CRS/ASM 465001 1 465001.1

Configuring raw devices for Oracle 10gR2 (10.2.0) Clusterware on RH/EL 5

126

Common Issues #3: Networking •Networking, especially private network, must be configured properly •NO CROSSOVER CABLES ALLOWED! •In VMWare, VMWare ensure you use correct interface for public, private networks •Use U h host-only t l network t k ffor private, i t bridged b id d for public •Set S t up b bonding di b before f iinstalling t lli clusterware l t

127

Common Issues #4: Filesystems •Don’t “go nuts” with Linux filesystem layout •Avoid too many filesystems (especially on a test system) •Ensure you create sufficient swap space

128

Common Issues #5: Host equivalency •Don’t hate SSH, just set it up properly! ssh keygen –t ssh-keygen t dsa cat ~/.ssh/id_dsa.pub >> \ ~/.ssh/authorized_keys / / scp –rp ~/.ssh othernode:. ssh date •Metalink Note 300548 300548.1 1 is helpful

129

References •Much of this article originated from Tim Hall’s Hall s article at oracle-base oracle-base.com com (with permission--Thanks, Tim!): http://tinyurl com/39ywx2 http://tinyurl.com/39ywx2 •Oracle 11g Documentation •Oracle Metalink, multiple notes and articles

130

131

RAC SIG Events • See www.oracleracsig.org for details –Webcasts: Average 2x per month, live –Conference C f Events: • Customer Panel, Monday @ 10:30a • Experts Panel, Tuesday @ 12:15p • Birds Bi d off a F Feather, th W Wednesday d d @1 1:30p 30

–Forums (via OTN): Lots of participation from RAC SIG as well as Oracle gurus • Join J i th the RAC SIG att www.oracleracsig.org! l i !

132

Building a RAC Test Environment on VMWare For Free S Session i #341

Dan Norris, Norris Twitter: dannorris Piocon booth #1619 dnorris@piocon com [email protected] http://www.dannorris.com/

Legal The information contained herein should be deemed reliable but not guaranteed. The author has made every attempt to provide current and accurate information. If yyou have any p y comments or suggestions, please contact the author at: [email protected] You may request redistribution permission from [email protected]. Copyright © 2008, Piocon Technologies

134