Everything About Rac 10g Linux

OCRCONFIG utility ----------------The OCRCONFIG utility allows you to perform various administrative operations on the OCR. It is locatedin the $ORA_CRS_HOME/bin directory and must be executed by the root user.You can use the OCRCONFIG utility to show details of automatic physical OCR backups, to restore automatic physical OCR backups, and to export and import the contents of the OCR .For Even upgrading and downgraging an OCR this utility can be used for backup -----------OCR is backed up automatically every four hours by the crsd daemon on one instance. The previous three backup copies are retained. A backup is also retained for each full day and at the end of each week. The frequency of backups and the number of files retained cannot be modified Restoring the backup --------------------[root@londonl root]# ocrconfig -showbackup Stop Oracle Clusterware on each node: [root@londonl root]# crsctl stop crs Next, identify the most recent backup: [root@londonl root]# ocrconfig -showbackup Then, restore the backup file: [root^londoni root]# ocrconfig -restore For example, to restore the most recent backup, use the following: [root^londonl root]# ocrconfig -restore $ORA_CRS_HOME/cdata/crs/backupOO.ocr Moving the OCR -------------stop crs edit /etc/oracle/ocr.loc file, and update the ocrconfig_loc parameter: [[ ocrconfig_loc=/u05/oradata/RAC/0CRFile local_only=FALSE ]] Move the OCR file to its new location: [root^londonl root]# mv /u02/oradata/RAC/0CRFile /u05/oradata/RAC/0CRFile upgrade and downgrade should be done under the supervision and advice of Oracle Suport Dumping OCR ---------[rootgilondonl root]# ocrdump -stdout SYSTEM [root^londonl root]# ocrdump -stdout SYSTEM.ess [root@londonl root]# ocrdump -stdout SYSTEM,ess,misscount [rootgilondonl root]# ocrdump -stdout SYSTEM.ess,misscount -xral $ORA CRS HOME/bin/ocrdump.bin -stdout -keyname SYSTEM,ess.misscount -xml

SYSTEM.c5S.misscount UB4 (10) PROCR^ALL^ACCESS PROCR READ PROCR~READ root root

VOTING DISK ADMINISTRATION -------------------------You do not need to shut down the database or Oracle Clusterware before backing up the voting disk dd if = of= LOG FILES ---------The $ORA_CRS_HOME/log/<node> directory also contains of the following subdirectories: � client: Contains log files for various OCR applications, including CLSCFG, CSS, OCRCHECK, OCRCONFIG, OCRDUMP, andOIFCFG � crsd: Contains log files for the crsd daemon, including crsd. log � cssd: Contains log files for ocssd daemon, including ocssd. log � evmd: Contains log files for evmd daemon, including evmd. log � racg: Contains log files for node applications, such as VIP and ONS The log files in these directories grow quite rapidly, so you may wish to back them up and truncate them periodically

Advanced Queing ===============

Scheduler ========= In addition to RAC awareness, DBMSSCHEDULER supports schedule windows, detailed logging within the database of job activity, and external operating system calls, and is fully integrated with Resource Manager.

Jobs are read by a job coordinator process and job slaves are invoked. Job slaves may be internal processes or external programs, and the number of job slaves is controlled by the MAX_DOB_SLAVE_ PROCESSES parameter. In a RAC environment, each instance has one job coordinator. The job table is stored in the database.Therefore, there is only one job table across the database

global cache hit ratio Tuning ======= 1) App tuning 2)db tuning 3)os tuning 4)After tuning each instance individually I would focus on processes that communicate thru the cluster interconnnect they are Global Services Directory Processes GES GCS-oracle makes a GCS req whenever a user accesses a buffer cache to read or modify a data blocks and the block is not in the local cache Monitoring GCS includes monitoring the common wait events Crital rac related waits are 1)Global cache busy->A wait event that occurs whenever a session has to wait for an ongoing operation on the resource to complete 2)Buffer busy global cache-->a wait that is signalled when a process has to wait for a block to become availablr cuz another process is obtained a resource for this block 3)Buffer busy global CR---> waits a consistent read via the global cache Tuning cluster interconnect --------------------------1)check v$session_wait whether ny sessions are experiencing RAC related waits select intd_id event,file_number,block_number,wait time form v$session_wait where event in ('above', 'three', 'waits'); in the output the block number and file number will indicate the object the requesting instance is waitin for 2)Identify the objects that are causing contention for these sessions(for each file_number/block_number combo returned) select owner,segment_name,segment_type

from dba_extents where file_id=9(put file_number from before query) and 150; THE OUTPUT WILL GIVE SEGMENT NAME AND SEGMENT TYPE WITH CONTENTION 3)Modify the objects to reduce contention AFTER OBTAINING THE OBJECT MODIFY BELOW *REDUCE THE NUMBER OF ROWS PER BLOCK *aDJusT THE BLOCK SIZE TO A SMALLER BLOCK SIZE *mODIFY INITRANS AND FREELISTS monitor the block transfer time ================================ GLOBAL CACHE CR BLOCK RECEIVE TIME BLOCK TRASFER TIME = -----------------------------------------GLOBAL CACHE CR BLOCKS RECEIVED BLOCK CONTENTION CAN BE MEASURED BY USING BLOCK TRANSFER TIME this can be achieved with a self join on gv$SYSSTAT Things to check 1)High transfer time 2)One node showing excessive transfer time 3)use os commands to verify cluster interconnect *Large no of prcs in run state waiting for cpu/scheduling *Platform specifig OS parameter setting that affect IPC buffering/process scheduing *slow busy faulty interconnects.Look for dropped packets/retransmits /crc eroors *Ensure u hane a provate network *Ensure inter instance tranffic is not routed thru public network USE NETSTAT -L/-S VMSTAT IOSTAT monitor the block service time ================================ GLOBAL CACHE CR BLOCK service time = GLOBAL CACHE CR BLOCK read timex + GLOBAL CACHE CR BLOCK build time+ GLOBAL CACHE CR BLOCK log flush time + GLOBAL CACHE CR BLOCK send time

lms process this can be achieved with a self join on gv$SYSSTAT for avg service times the one with least service time is on the slowwer node otherwise on a faster node and serves blocks faster monitor the gc convert and get times ====================================

GCS resources are converted null-exclusive shared-exclusive null-shared gc convert time---> accumilated time a session requires for a gc convertion of GCS resource gc converts gc get time gc gets If converstime > 10-20 ms these usually vary with CPUs on each instance use gv$system_event view to review TIME_WANTED stats or various gcs events iF\ the get/convert times become significant stats pack can be used to view these events HIGH CONVERT TIME instance swapping a lot of blocks over interconnect RAPID INCREASE GCS contention HIGH LATENCIES OF RESOURCE OPS excessive system loads *REDUCE THE LOAD ON THE SERVER BY REDUCING THE NUM OF PROCS ON db SERVER *INCREASE CAPACITY BY ADDING CPUS *ADD NODES TO THE CLUSTER DATABASE Non zero GC convert time sytem congestion/serious perf problem

other wait events ---------------these may show up in the top 5 timed wait events on stats pack report determine the cause of wait event gc open s/x --> a session has to wait for receiving permissions for shared/exclusive access to req resource NOT MUCH CAN BE DONE.CONSIDER PRELOADING HEAVILY USED TABLES gc null to x-->happens when 2 nodes exchange a same block back and forth REDUCE NUM OF ROWS PER BLOCKTO ELIMINATE THE NEEED FOR BLOCK SWAPPING BETWEEN INSTANCES gc null to s gc cr request-->happens when an insrance reqs a CR data block and the block to be transfered hasnt arrived at the requesting instance MODIFY OBJECTS FOR POSSIBLE CONTENTIONS *REDUCE THE NUMBER OF ROWS PER BLOCK *aDJusT THE BLOCK SIZE TO A SMALLER BLOCK SIZE *mODIFY INITRANS AND FREELISTS GCS utilization for Logical Reads hIGH gCS Time per req ---------------------u can get this in statspack report CAUSES:

1)Contention of blocks FIX: *REDUCE THE NUMBER OF ROWS PER BLOCK

*aDJusT THE BLOCK SIZE TO A SMALLER BLOCK SIZE *mODIFY INITRANS AND FREELISTS 2)SYSTEMLOADS FIX: *REDUCE THE LOAD ON THE SERVER BY REDUCING THE NUM OF PROCS ON db SERVER *INCREASE CAPACITY BY ADDING CPUS *ADD NODES TO THE CLUSTER DATABASE 3)NETWORK ISSUES *Check os logs for nw link congestion *ensure packets are being routed thru private interconnect running STATSPACK ----------------the report contains valuable statistics that can help you identify sql related problems memory problems, IO problems latch contention and enqueue issues shows stats of instance on which it is run run perfstat.statspack.snap procedure and spreport.sql script on individual node instances

to monitor

and compare to other

OUTPUT TOP 5 TIMED EVENTS waits time Elapsed Time ---------------------- ----- -----------gc cr request 820 154 72.50 cpu time 54 24.34----->cpu time shud be predominant but cr req exceeds therefore needs investigation gc null to x 478 1 .52 control file seq read 600 1 .52 control file parallel write 141 1 .28 transfer times are excessive from other instances in this cluster to this instance could be due to nw problem/bufer cache sizing issue network changes were made an index was added Now it lookss like TOP 5 TIMED EVENTS waits time Elapsed Time --------------------------- -----------cpu time 54 24.34 gc cr request 820 154 72.50 gc null to x 478 1 .52 control file seq read 600 1 .52 control file parallel write 141 1 .28 gv$ PERFOMANCE VIEWS FOR rac ============================= gv$sysstat --->contains stats tht indicate perf of RAC global cache blocks lost-->indicates n/w problem global cache blockscorrupt-->blocks corrupted during transfer ,indicates ipc/nw/ardware problem

SELECT A.VALUE "GC BLOCKS LOST 1", B.VALUE "GC BLOCKS CORRUPT 1", C.VALUE "GC BLOCKS LOST 2", D.VALUE "GC BLOCKS CORRUPT 2" FROM GV$SYSSTAT A,GV$SYSSTAT B,GV$SYSSTAT C,GV$SYSSTAT D WHERE A.INST_ID=1 AND A.NAME='GLOBAL CACHE BLOCKS LOST' WHERE B.INST_ID=1 AND B.NAME='GLOBAL CACHE BLOCKS LOST' WHERE C.INST_ID=2 AND C.NAME='GLOBAL CACHE BLOCKS LOST' WHERE D.INST_ID=3 AND D.NAME='GLOBAL CACHE BLOCKS LOST' the O/P OF aboVe stats should near zero IF NOT ZERO THAT NODE INSTANCE IS EXPERIENCING LOST/CORRUPT BLOCKS gv$dml_misc gv$system_event gv$session_wait gv$system_event toolkit ------racdiag.sql can be downloaded from metalink statspack

Copy Right . Indus DB Pro ,Vikas Motkur