Configuring Snapmirror

Configuring Snapmirroring

Configuring Snapmirroring Eircomnet Version 1.0 Owner EircomNet Prepared by Chander Sangra DBA Team

11 July 2008

DBATeam/ NetApp/ Snapmirroring

Page 1 of 20

Configuring Snapmirroring

1 Document Control 1.1

Revision History

Revision version 1.0

1.2

Previous revision date

Summary of Changes

Author Chander Sangra

Distribution

This document has been distributed to Name Dervla Monaghan

11 July 2008

Title Operations

Date of Issue 21-April-2009

DBATeam/ NetApp/ Snapmirroring

Version 1.0

Page 2 of 20

Configuring Snapmirroring

2 Table of Contents 1 Document Control......................................................................... 2 1.1 Revision History................................................................................... ................2 1.2 Distribution............................................................................................ ..............2 2 Table of Contents.........................................................................3 3 Overview...................................................................................... 5 3.1 Preface................................................................................................................ .5 3.2 Purpose and Scope....................................................................... .......................5 3.3 Prerequisites and Assumptions.......................................................... ..................5 4 Snapmirroring..............................................................................6 4.1 Type of Snapmirroring ................................................................................. ........6 4.1.1 Asynchronous........................................................................ ................6 4.1.2 Synchronous................................................................................. .........6 4.1.3 Semi-Synchronous..................................................................... ...........6 5 Snapmirroring Requirements........................................................7 5.1 Assumptions.................................................................................... ....................7 5.2 Requirements ....................................................................... ..............................7 5.2.1 Network..................................................................................... ............7 5.2.2 Root Volume................................................................... .......................8 5.2.3 Rsh access......................................................................................... ....8 5.2.4 Permissions............................................................. ..............................8 5.2.5 Mount................................................................................ ....................8 5.2.6 Ownership..................................................................... ........................8 5.2.7 Others...................................................................... .............................8 6 Architecture.................................................................................9 7 Configuring Snapmirroring..........................................................10 7.1 Configuring Database................................................................... .....................10 7.2 Configuring Snapmirror.................................................................................. ....11 7.2.1 Identify Source & Target filer...................................... .........................11 7.2.2 Identify Volumes...................................................................... ............11 7.2.3 Identify Schedule........................................................................ .........11 7.2.4 Enable SnapMirror........................................................ .......................12 7.2.5 Create Baseline......................................................... ..........................12 7.2.6 Update SnapMirror............................................................................. ..12 7.2.7 Check the Status......................................................... ........................12 8 Database Recovery Using SnapMirror Async and Sync..................13 8.1 8.2 8.3 8.4

Generate Data............................................................................... ....................13 Snapshot Backup........................................................................... ....................13 Simulate Disaster.............................................................. ................................13 Perform Recovery................................................................ ..............................14 8.4.1 Recovery from SnapMirror Update Point of Data Volumes...................15 8.4.2 Recovery from Snapshot Backup of the Data volumes........................15

9 Special Consideration.................................................................17 9.1 9.2 9.3 9.4 9.5 9.6

Initialisation Phase................................................................................... ..........17 Throttle Rate.............................................................................................. ........17 Incremental Phase......................................................................... ....................17 Stopping & Restarting Snapmirroring................................................................ .17 Breaking the Mirror........................................................................... .................17 Re-synching the Mirror.............................................................................. .........17

11 July 2008

DBATeam/ NetApp/ Snapmirroring

Page 3 of 20

Configuring Snapmirroring 10 Appendices...............................................................................19 10.1 Appendix A (Scripts)................................................................. .......................19 10.2 Appendix B .......................................................................... ...........................19 11 References...............................................................................20

11 July 2008

DBATeam/ NetApp/ Snapmirroring

Page 4 of 20

Configuring Snapmirroring

3 Overview 3.1

Preface The base of this document is NetApp’s document on “Disaster Recovery With SnapMirror Technology” (Ref. TR 3057). The document is prepared to fit the existing database environment in eircomnet. The document is prepared based on the theoretical knowledge of the DBA without any practical experience. The next version of this document would be produce once the testing is performed in the test environment.

3.2

Purpose and Scope The purpose of this document is to presents an overview of SnapMirror technology, its importance in the database environment, evaluating the pros & cons of its implementation and finally the recommendations to assist the DBA/Ops team in designing an optimal SnapMirror solution.

3.3

Prerequisites and Assumptions This document is useful to the DBA and the Operations team members, the following assumptions are made that the reader has: - Minimal knowledge of NetApp platforms and products, particularly in the area of data protection - General knowledge of disaster recovery (DR) solutions - Working knowledge of the NetApp Snapshots and Snaprestore

11 July 2008

DBATeam/ NetApp/ Snapmirroring

Page 5 of 20

Configuring Snapmirroring

4 Snapmirroring SnapMirror software provides automated file system replication from a source to a destination system. The NetApp storage system from which data is transferred is referred to as the SnapMirror source, and the NetApp storage system to which the data is transferred is referred to as the SnapMirror destination. The SnapMirror source and destination can be miles apart, provided that both NetApp storage systems can communicate with each other across a network. The SnapMirror configuration details are maintained in a configuration file called snapmirror.conf, which resides on the destination NetApp storage system. This file, along with information entered via the snapmirror.access option or the snapmirror.allow file, is used to establish relationships between specified source volumes or qtrees and the destination volume or qtree where the mirrored data is kept. More details on Snapmirroring can be found at a separate document “Proposal on Snapmirroring” at the following location: http://obsscs/cgibin/display.pl?file=configuringsnapmirror_v1.0.doc&type=download

4.1

Type of Snapmirroring

4.1.1 Asynchronous In Asynchronous mode, SnapMirror performs incremental, block-based replication as frequently as once per minute. Performance impact on the source FAS system is minimal, as long as the system is configured with sufficient CPU and disk I/O resources.

4.1.2 Synchronous SnapMirror in synchronous mode is a mode of replication that sends updates from the source to the destination as they occur, rather than according to a predetermined schedule. This guarantees that data written on the source system is protected on the destination even if the entire source system fails. In synchronous mode, SnapMirror immediately replicates all data written to the source file system. This guarantees zero data loss in the event of a failure, but can have a significant performance impact. It is not necessary or appropriate for all applications.

4.1.3 Semi-Synchronous A semi-synchronous mode is also provided which minimizes data loss in a disaster while also minimizing the extent to which replication impacts the performance of the source system. Unlike asynchronous mode, which can replicate either volumes or quota trees, synchronous and semisynchronous modes work only with volumes.

11 July 2008

DBATeam/ NetApp/ Snapmirroring

Page 6 of 20

Configuring Snapmirroring

5 Snapmirroring Requirements 5.1

Assumptions This document covers Disaster Recovery Solutions offered by NetApp for Oracle 10gR2 single database instance using SnapMirror Async as well as Sync technology. The following assumptions are made in this document:

5.2

-

Source: o Server: testmsstaging00 o NetApp filer: "bfiler03" o Oracle Admin User: "SNAPDBA" o NetApp Volume: “/vol/dbasnaptestdata” for Oracle data “/vol/dbasnaptestlog” for Oracle logs o NetApp Mount Points): “/snaptestdata” for Oracle data “/snaptestlog” for Oracle data o Aggregate on the NetApp storage system used for database storage: “dbaggr” (The flexible volumes snaptestdata & snaptestlog reside in aggregate dbaggr)

-

Target: o Server: testdba00 o NetApp filer: "bfiler03" o Oracle Admin User: "SNAPDBA" o NetApp Volume: “/vol/testdba/testdba00” for Oracle data “/vol/ora_data_testdba00” for Oracle logs o NetApp Mount Points: “testdba” for Oracle data “oradata” for Oracle logs o Aggregate on the NetApp storage system used for database storage: “dbaggr” (The flexible volumes testdba & oradata reside in aggregate dbaggr)

-

Both Target & Source: It is also assumed that the database host used for accessing the database at the Disaster Recovery site has a similar setup to the database host on the primary site and has all required privileges to access the volumes on the SnapMirror destination NetApp storage system.

Requirements

5.2.1 Network Ensure good connectivity between the following: - Source Database Host and Source NetApp filers o Make entries to the “/etc/hosts” files - Target Database Host and Target NetApp filers o Make entries to the “/etc/hosts” files - Others

11 July 2008

DBATeam/ NetApp/ Snapmirroring

Page 7 of 20

Configuring Snapmirroring 5.2.2 Root Volume The root on the database server must have access to the root volume /vol/vol0. For example, to grant access on a root volume named /vol/vol0 on a NetApp storage system to the user root on the database host system <servername>, execute the following command on the storage systems: exportfs –p rw=<servername>,root=<servername>,anon=0 /vol/vol0 exportfs -a

5.2.3 Rsh access Enable ‘rsh’ access from the database server if it is going to be used. Add the IP address of the database host to the “/etc/hosts.equiv” file.

5.2.4 Permissions Ensure the user has permissions on all the volumes to be used for the database exportfs –p rw=,root=,anon=0 /vol/ exportfs -a

5.2.5 Mount Mount the volumes to be used for the database on database host

5.2.6 Ownership Ensure the file system on the mount points is owned by the user ‘oracle’ on the database server. Chown –R oracle:dba <mount point>

5.2.7 Others

11 July 2008

DBATeam/ NetApp/ Snapmirroring

Page 8 of 20

Configuring Snapmirroring

6 Architecture The following network & storage model is to be followed in this exercise:

11 July 2008

DBATeam/ NetApp/ Snapmirroring

Page 9 of 20

Configuring Snapmirroring

7 Configuring Snapmirroring 7.1

Configuring Database Prepare database according to the following setup: Database Configuration: -

Database: Oracle 10g R2 Instance: Single Instance (Instance Name) Oracle Home: Local on database host (/home/oracle/ora10g/product/10.2.0/db_1) Data volume for database data: Log volume for database logs: Softlinks to follow the OFA directory structure o Ln –s /mnt/ /home/oracle/ora10g/ o Ln –s /mnt/ /home/oracle/ora10g/

Database Layout: Control files: //control01.ctl //control02.ctl //control03.ctl Redo logs:

Datafiles:

//redo_01.log //redo_02.log //*.dbf

Database Setup: DB Structures: Execute ‘create_tables_demo.sql’ to create the tables DB Data: Execute ‘create_data_demo.sql’ to generate data into the demo tables

11 July 2008

DBATeam/ NetApp/ Snapmirroring

Page 10 of 20

Configuring Snapmirroring 7.2

Configuring Snapmirror Configure the SnapMirror using the following steps:

7.2.1 Identify Source & Target filer Identify the SnapMirror Source and the Destination NetApp storage systems. The source NetApp storage system should already have an Oracle 10g R2 database created on it. We are using “testmsstaging00.bfiler03” as SnapMirror Source that has Oracle 10g R2 database on it and “testdba00.bfiler03” as SnapMirror destination.

7.2.2 Identify Volumes Identify the volumes on the source NetApp storage system that is to be replicated across the Target NetApp storage systems using SnapMirror We are using “/vol/dbasnaptestdata” and “/vol/dbasnaptestlog” as the two volumes to be mirrored across Target system.

7.2.3 Identify Schedule Define a schedule for each SnapMirror relationship in the /etc/snapmirror.conf file, which resides on the SnapMirror destination’s NetApp storage system. The /etc/snapmirror.conf file controls where data is replicated from and how often the mirror is updated. The entry in the /etc/snapmirror.conf file should be in the following format: <Source Filer>: : ‘-‘ <Schedule> Where: The <Schedule> is defined by [minute] [hour] [days of month] [days of week]

A snippet from the /etc/snapmirror.conf file:

bfiler03:/vol/dbasnaptestdata bfiler03:/vol/testdbavol/testdba00 – 0,30 917 * 1-5 bfiler03:/vol/dbasnaptestlog bfiler03: /vol/testdba/testdba00 – 0,30 * * 1-5 In the above snippet, - The volume named “/vol/dbasnaptestdata” on bfiler03 is replicated to the volume named “/vol/testdbavol/testdba00” on bfiler03. - The argument dash ‘-‘ represents that data is replicated at the fastest rate possible. - The NetApp storage system updates the volume “/vol/dbasnaptestdata” at every 30-minute interval between 9 a.m. and 5 p.m., Monday through Friday. - The asterisk (*) in the example means that the mirror is updated on every day of the month.

11 July 2008

DBATeam/ NetApp/ Snapmirroring

Page 11 of 20

Configuring Snapmirroring

7.2.4 Enable SnapMirror Enable the SnapMirror feature on the Source and the Target by issuing the following command: option snapmirror on

7.2.5 Create Baseline Create one-time baseline transfer by initialising the SnapMirror relationship for each volume. Issue the following command from the SnapMirror destination: snapmirror initialize –S <Source Filer>: : for example: snapmirror initialize –S bfiler03:/vol/dbasnaptestdata bfiler03:/vol/testdbavol/testdba00

7.2.6 Update SnapMirror After the baseline transfer, the SnapMirror updates can be triggered based on a predefined schedule in the /etc/snapmirror.conf file. It is also possible to trigger a SnapMirror update manually by executing the following command from the destination: snapmirror update For example, execute the following command on the destination to start the manual SnapMirror update for a volume named /vol/testdbavol/testdba00: snapmirror update /vol/testdbavol/testdba00

7.2.7 Check the Status Monitor the SnapMirror status by executing the following command on the Destination filer: snapmirror status

11 July 2008

DBATeam/ NetApp/ Snapmirroring

Page 12 of 20

Configuring Snapmirroring

8 Database Recovery Using SnapMirror Async and Sync To simulate a Disaster Recovery strategy using mixed modes of SnapMirror, we configured the data volume replication using SnapMirror Async and the transaction logs volume replication using SnapMirror Sync mode. The frequency of SnapMirror and Snapshots are summarized as: Volume vol/dbasnaptestdata vol/dbasnaptestlog

SnapMirror 2 hrs Sync

Snapshot Backup 30 min n/a

The snippet from the /etc/snapmirror.conf file looks somewhat similar to the following:

bfiler03:/vol/dbasnaptestdata bfiler03:/vol/testdbavol/testdba00 – 0 0-22/2 * 1-5 bfiler03:/vol/dbasnaptestlog bfiler03: /vol/testdba/testdba00 – sync The argument ‘sync’ in the above entries controls whether a volume is Sync snapmirrored with its source or Async. If the argument is left out of the conf file, then the volume will be Async snapmirrored. Perform the following steps to simulate disaster on the primary site (Source) and perform recovery on the Disaster Recovery site (Target):

8.1

Generate Data Execute the following database command to run the load: Execute scott.demobld(25000000);

8.2

Snapshot Backup Take -

8.3

snapshot backup of the database. Move the database into the ‘Backup Mode’ Take Snapshot of the data & log volumes Move the database out of ‘Backup Mode’

Simulate Disaster Let us assume disaster strikes at 3:05 p.m. In order to simulate the disaster, we made the SnapMirror source NetApp storage system unavailable by executing the following command:

11 July 2008

DBATeam/ NetApp/ Snapmirroring

Page 13 of 20

Configuring Snapmirroring

After a disaster, the SnapMirror destination volumes corresponding to the database data will be in SnapMirror Snapshot consistent state. That means the data volumes at the Target site will have the same data, including Snapshot Backup copies, as the corresponding source volume had at the time of successful SnapMirror update. The transaction logs volume is Sync snapmirrored; therefore, the corresponding volume at the Disaster Recovery site will have all the changes up to the point the disaster started. For the Test Scenario, the data volume named dbasnaptestdata is Async snapmirrored and the transaction logs volume named vol/dbasnaptestlog is Sync snapmirrored; therefore, these two volumes will be in the following states: dbasnaptestdata – consistent with the source volume at the time of SnapMirror update at 2.00 p.m. dbasnaptestlog – consistent with the source volume at the time disaster started at 3.05 p.m.

8.4

Perform Recovery After the baseline transfer, the destination volumes are available to clients, but in a read-only state. The status of a destination will show that it is snapmirrored. A disaster makes the source unavailable; to use the destination volumes for writing as well as reading, you would need to end the SnapMirror relationship by executing the following command on the destination NetApp storage system: snapmirror break This command changes the destination volume’s status from snapmirrored to broken-off, thus making it writable. After a disaster, in this type of configuration, there are two possible options for the database recovery: 1) Recovery from the SnapMirror update point of the data volumes. 2) Recovery from an application-coordinated Snapshot copy of the data volumes.

11 July 2008

DBATeam/ NetApp/ Snapmirroring

Page 14 of 20

Configuring Snapmirroring 8.4.1 Recovery from SnapMirror Update Point of Data Volumes The database’s transaction logs volume is sync snapmirrored; as a result, all the changed data up to the point the disaster started is replicated to the destination volume. Therefore, after database recovery, there shouldn’t be any data loss; the RPO (Recovery Point Objective) will be zero. The RTO (Recovery Time objective) will vary based on the transaction logs that need to be applied for recovery. To perform recovery from the SnapMirror update point of the data volumes, we need to follow the following steps on the database server: Mount each volume used for the database to a database host by executing the following command : mount –o rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,vers=3, timeo=600 : <mount point> The control file that resides on the transaction logs volume will have the latest database information, so copy that control file from the dblogs volume to the dbdata volume by executing the following commands on the database server: <> Start the database instance and perform database recovery by executing the following commands on the database server: Startup mount Set autorecovery on Recover database Alter database open

8.4.2 Recovery from Snapshot Backup of the Data volumes This type of recovery may become necessary in rare cases, such as when database recovery is not possible from the SnapMirror update point. In this kind of situation, the database can be recovered using applicationcoordinated Snapshot copies of the data volume. In this configuration, the database transaction logs volume is sync snapmirrored; as a result, all the changed data up to the point the disaster started is replicated to the destination volume. Therefore, after database recovery, there shouldn’t be any data loss; RPO (Recovery Point Objective) will be zero. The RTO (Recovery Time objective) will vary based on the transaction logs that need to be applied for recovery. In order to perform database recovery using an application-coordinated Snapshot copy of the volume holding the database’s data, you would need to complete the following steps: First you need to restore the volumes that hold data for the database from the application-coordinated Snapshot copy by executing the following command on the NetApp storage system: snap restore -s <snapshot name>

11 July 2008

DBATeam/ NetApp/ Snapmirroring

Page 15 of 20

Configuring Snapmirroring

Mount the NetApp storage system volumes that hold the database’s data and transaction logs to a database host by executing the following command : mount -o rw,bg,hard,nointr,rsize=32768,wsize=32768, tcp,vers=3,timeo=600 : <mount point>

The control file that resides on the transaction logs volume will have the latest database information, so copy that control file from the dblogs volume to the dbdata volume by executing the following commands on the database server: <> Start the database instance and perform database recovery by executing the following commands on the database server: Startup mount Set autorecovery on Recover database Alter database open After completing the above steps, the database recovery will be complete.

11 July 2008

DBATeam/ NetApp/ Snapmirroring

Page 16 of 20

Configuring Snapmirroring

9 Special Consideration Several special scenarios should be considered as:

9.1

Initialisation Phase If an initialization (level 0) replication event is interrupted for more than 9 minutes (ex. a network outage, filer reboot) it will abort. Partial level 0 events are not recoverable, so the process must be restarted.

9.2

Throttle Rate The KB per second maximum throttle parameter in /etc/snapmirror.conf can be changed at any time, and the edited values will take effect within two minutes. The exception to this is mirror initialization where a bandwidth throttle is already in effect. In this case, the bandwidth throttle cannot be modified until the initialization phase is complete or the process is interrupted.

9.3

9.4

Incremental Phase -

Incremental updates will not start until the level 0 initialization is complete.

-

Any incremental updates which are missed are simply skipped. Subsequent incremental updates transmit any data which was skipped, so no data is lost.

-

Incremental updates in progress will run to completion according to the configuration settings and available network bandwidth. If a new incremental update is scheduled to start while an existing update is in progress, it is considered a schedule miss and skipped.

Stopping & Restarting Snapmirroring The SnapMirror process can be stopped and restarted at any time by issuing the following command sequence on either filer: vol snapmirror off [arbitrary time interval] vol snapmirror on. As long as the target volume remains read only during the time between turning the SnapMirror process off and on, the mirror remains intact.

9.5

Breaking the Mirror To break the mirror, you issue the following command on the target filer: vol options oracle_mirror snapmirrored off As a result the target volume is still online but changed from read-only to read/write mode.

9.6

Re-synching the Mirror Currently the ability to restart a broken mirror is not available, although this feature is planned for a future release. Presently, the method for reestablishing a mirror is to reinitialize the mirror. If the target volume has been brought online and then written to, this may need to be done twice. -

11 July 2008

Once to replicate the data back to the source volume. (In this case the mirror relationship between the two filers is flipped.) Once to reinitialize with the mirror relationship back to normal.

DBATeam/ NetApp/ Snapmirroring

Page 17 of 20

Configuring Snapmirroring An alternative is to back up the target volume's data to tape, and then restore that data onto the source filer. This avoids the necessity to reinitialize the mirror twice.

11 July 2008

DBATeam/ NetApp/ Snapmirroring

Page 18 of 20

Configuring Snapmirroring

10Appendices 10.1 Appendix A (Scripts) Stored procedure script ----------------------------------------------------------------- Script Name: create_demobld.sql -- Purpose: This script generate database load by inserting specified number of rows in the table.

---------------------------------------------------------------CREATE OR REPLACE PROCEDURE demobld( rcount IN number DEFAULT 10) IS --- Purpose: Generate database load by inserting specified number of -rows in the table BEGIN for i in 1..rcount loop insert into scott.tab1 values (i, 'ABDFADFAFAFAFASFDASDFFFFFFFFFFFFFFFFFFFFFFFFFF'); if mod(i,1000)=0 then commit; end if; end loop; END; /

Snapshot Backup ----------------------------------------------------------------- Script Name: snapshot_backup.ksh -- Purpose: This script perform the following tasks: o o o

Put the database in hot backup mode (backupmodein.sh) Create snapshot (create_snapshot.ksh) Put the database out of backup mode (backupmideout.sh)

----------------------------------------------------------------

10.2 Appendix B

11 July 2008

DBATeam/ NetApp/ Snapmirroring

Page 19 of 20

Configuring Snapmirroring

11References -

SnapMirror Best Practices Guide (TR-3446)

-

SnapMirror Async Overview and Best Practices Guide (TR-3446)

-

SnapMirror Sync and SnapMirror Semi-Sync Overview and Design Considerations Guide (TR-3326)

EircomNet: Backup & Recovery guideline document for the Mediasurface databases where snapshot based backup & recovery solution is implemented: http://obsscs/cgibin/display.pl?file=backuppolicy_v1.0.doc&type=download -

-

EircomNet: Proposal on Net-App based backup & recovery solution: http://obsscs/cgi-bin/display.pl?file=b_rproposal.doc&type=download

EircomNet: Practical demonstration of Net-App based database recovery: http://obsscs/cgi-bin/display.pl?file=recoverydemo1.doc&type=download -

11 July 2008

DBATeam/ NetApp/ Snapmirroring

Page 20 of 20