Vmware Runbook.docx
This document was uploaded by user and they confirmed that they have the permission to share it. If you are author or own the copyright of this book, please report to us by using this DMCA report form. Report DMCA
Overview
Download & View Vmware Runbook.docx as PDF for free.
More details
- Words: 3,488
- Pages: 28
Information Technology WFM – Run-book [WFM] Revision #: [1] Created Date:7/18/2011 Last Revised:
Requested By: [VNOC Support] Author: [Rajnish Tahiliani]
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Runbook Version #: [ ]
Table of Contents Table 1. 2. 3.
of Contents ......................................................................................... OVERVIEW ......................................................................................... ROLES AND RESPONSIBILITIES ......................................................... ESCALATIONS ....................................................................................
3.1 3.1.1 3.1.2
Severity Level Definitions ................................................................ 4 Severity 1 ................................................................................ 4 Severity 2 ................................................................................ 4
3.2.1
Escalation Procedures ...................................................................... 5 Time Frames ............................................................................ 6
3.2 3.3
Call Tree ........................................................................................ 6 3.3.1 User Call’s VMSD for a WFM Problem ........................................... 6 3.3.2 VNOC Gets a Monitoring Alarm ................................................... 6
3.4
GSS FAST Team Contact Info: .......................................................... 6
3.5
Infrastructure & Interfaces Team Contact Info: .................................. 7
3.6
How to Contact IEX Support ............................................................. 8
3.7
IEX Support Escalations: ................................................................. 9 ARCHITECTURE ................................................................................ 11
4.1
Architecture Diagrams ................................................................... 11
4.2
Dependencies ............................................................................... 13 HOST INFORMATION ........................................................................ 13
5.1
Infrastructure ............................................................................... 13 5.1.1 Network Diagram .................................................................... 13 5.1.2 IP Configuration Information .................................................... 13 5.1.3 Host Information .................................................................... 13 OPERATIONAL PROCEDURES ............................................................ 14
6.1
Admin Procedures and Maintenance – GSS WFM DB .......................... 20
6.2
Backup and Recovery – GSS WFM DB.............................................. 21
6.3
Troubleshooting Scenarios - Informatica .......................................... 21 FUNCTIONAL PROCEDURES - Informatica ........................................ 21 7.1.1 Functional Support Scenarios ................................................... 21 7.1.1.1 Running WFM Process ................................................................................21 7.1.2 SOX Evidence ......................................................................... 25 7.1.3 Emergencies .......................................................................... 25 7.1.4 Exceptions ............................................................................. 25
7.2
Failure Contingencies .................................................................... 25 SERVICE LEVEL AGREEMENT (SLA)................................................... 25
4.
5.
6.
7.
8.
2 4 4 4
VMware
CONFIDENTIAL
Page 2 of 28
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Runbook Version #: [ ] 9.
10.
MONITORING AND REPORTS ............................................................ 25 9.1
Types of Monitoring ...................................................................... 25
9.2
Monitoring Configurations .............................................................. 25
9.3
Weekly Reports ............................................................................ 25
9.4
Monthly Reports ........................................................................... 25
9.5
Ad-hoc Reports............................................................................. 25
9.6
Audit Logs ................................................................................... 25 Security and Compliance .................................................................. 25
10.1
Change Control ............................................................................ 25
10.2
SDLC .......................................................................................... 25
10.3
Known Deficiencies ....................................................................... 25
10.4
User Management ......................................................................... 25
10.5
Links to Access Control .................................................................. 25
10.6 Appendix Appendix Appendix Appendix
VMware
SOX Evidence ............................................................................... 25 A: Document Change History ...................................................... 26 B: Open and Closed Issues ......................................................... 26 C: Glossary/ Definitions .............................................................. 26 D: Review and Sign-off ............................................................... 27
CONFIDENTIAL
Page 3 of 28
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 4 of 28 Runbook Version #: [ ]
1. OVERVIEW The GSS Business team must effectively manage and track the time of its customer-facing agents, which include Technical Support Engineers (TSEs), Customer Service Representatives (CSRs) and Licensing Representatives. In the long-term VMware may consider expanding the user base to external contractors and service providers. With a global operations footprint, it has become critical to be able to optimally manage available capacity to match customer demand. VMware GSS currently has very limited capabilities for forecasting and scheduling without significant overhead. Additionally, the organization lacks mature resource forecasting for its short/long term needs and the ability to effectively manage sudden peak demand. Rudimentary performance tracking and resource availability are done with legacy tools like Excel.
2. ROLES AND RESPONSIBILITIES Informatica Team: Resolve Informatica maintenance, performance, and troubleshooting on production Informatica server and data flow from the informatica server to the Work-force management application server. Telecom Team: Responsible for providing support during a data flow issue from CMs to Work-force management application server VNOC Team: Response and escalation for Severity 1 and 2 issues for WFM Application DBA Team: Resolve database maintenance, performance, and troubleshooting on production Informatica server Network Team: Resolve network maintenance, performance, and troubleshooting on production Informatica server Linux Server Team: Resolve server maintenance, performance, and troubleshooting on production Informatica server Windows Server Team: Resolve server maintenance, performance, and troubleshooting on production workforce management application servers
3. ESCALATIONS 3.1 Severity Level Definitions 3.1.1 Severity 1 HelpZilla: Critical issues affecting large numbers of people (e.g., entire campus, building, or regional location). Response time: 30 minutes or less. VNOC: LEVEL 1 Incident. Telecom System (CMS) is down. Examples of LEVEL 1 Incident: Gss team member are not getting RTA on WFM Application
VMware
Gss team member will not be able to see SLA report
Impacting entire site or greater
Severe business impact
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 5 of 28 Runbook Version #: [ ]
3.1.2 Severity 2 HelpZilla: Issues affecting an entire department (e.g., Engineering, Sales, Support, Finance, etc). Response time: 1 hour or less. VNOC: LEVEL 2 Incidents. Informatica is down or severely degraded. Examples of LEVEL 2 Incidents: Any of the above systems are down or the performance has degraded.
No workarounds
Impacting certain department or group.
Partial site functional loss.
3.2 Escalation Procedures Please note this escalation process is only if there is an infrastructure / operational level outage within VMWare. All application level issues will be handled by the FAST GSS team. If users call with a problem please contact FAST GSS Team.
Alert Type WFM Server Issue ( Monitoring ) Informatica Availability CPU Threshold
Level 1 (Primary) Contact Person/Team Please call IEX Support off-site (Prama B) or onsite (on-call support resource) Linux/Windows on-call appropriate to the system.
When to Notify Level 2 Person/Team Please notify FAST GSS Team Please notify FAST GSS Team
Memory Threshold
Linux/Windows on-call appropriate to the system
Please call FAST GSS Team
Disk Capacity
Please call FAST GSS Team Linux/windows on-call appropriate to the system
If the server is Unix box and has a database hosted on it, please call DBA and Informatica admin. All other cases notify System Admin
Database Temp space size monitoring
DBA on-call
Database Availability
DBA on-call
Listener Availability
DBA on-call
VMware
Please call FAST GSS Team
Please call FAST GSS Team Please call FAST GSS Team
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 6 of 28 Runbook Version #: [ ]
3.2.1 Time Frames Primary on-call person – please give 15 minutes to respond and call the secondary on-call person after the 15 minutes are over for the primary on-call person. If there is no response from the primary/secondary on-call person, escalate it to the manager in the first 30 minutes.
3.3 Call Tree GSS FAST Team will handle all application level issues. GSS FAST team will notify the users and also call into IEX support as appropriate. 1. GSS End users (managers/supervisors, front line staff) would report application issues (such as with forecasts, schedules, agent data corrections) with the FAST GSS team. The issues will be reported via Help-SFDC-Zilla or can be sent directly to [email protected]. Issues created in the SFDCZilla are sent to this email address as well. The FAST GSS WFM team (Bhaskar, Karthik, Ashwin) and the Futuresoft consultants subscribe to this email alias. 2. The FAST GSS WFM team (IEX secondary contacts) can directly report an issue with IEX that is related to IEX application functionality or behavior. VMware issues can be tracked on IEX web portal using our credentials.
Level 1 Support:
3.3.1
User Call’s VMSD for a WFM Problem
Contact the FAST GSS Team as per escalation process defined above. In case of an operations / infrastructure issue please contact the Infrastructure & Interfaces Team Contact Info:
3.3.2
VNOC Gets a Monitoring Alarm
Please contact as per Escalations
3.4 GSS FAST Team Contact Info: Escalation Path
Name
Office Phone
Primary
GSS FAST
During APAC Business Hours
Bhaskar Babu
None
“
Ashwin Prasad
None
“
Karthik Venkatraman
None
During EMEA Business Hours
John McDermott
During Americas Business Hours Manager Escalation VMware
Cell Phone
+91 966 387 4003 +91 959 114 4998 +91 994 584 4255
353 21 4660290
353 86 7714575
Jon Campbell
303 942 4867
303 601 7812
Lynn Adamson
650 427-4317
415 794-4762 CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 7 of 28 Runbook Version #: [ ]
3.5 Infrastructure & Interfaces Team Contact Info: Informatica Issue: Escalation Path
Name
On-site Primary (Level 1 and 2)
Prama B
Off-site Primary (Level 1 and 2)
On-call Informatica resource
On-site/Off-site Manager (Level 1 and 2)
Rajesh Karale
Office Phone
Cell Phone
Office Phone
Cell Phone
Telecom Issue: Escalation Path
Name
Primary
Hector Herrera
Manager Escalation
Will Clark
1-650-427-5528 1-650-427-5775
Informatica - DNBIM Database Issue: Escalation Path
Name
Primary
On-call Oracle DBA resource
Manager Escalation
Varinder Kumar
Office Phone
Cell Phone
(650) 427-5938
(408) 203-8602
Office Phone
Cell Phone
(650) 427-5605
(408) 203-8103
Office Phone
Cell Phone
(650) 427-5938
(408) 203-8602
Office Phone
Cell Phone
(650) 427-5938
(408) 203-8602
Office Phone
Cell Phone
Informatica - Oracle Apps Issue: Escalation Path
Name
Primary
On-call Oracle Apps resource
Manager Escalation
Kiran Chinya
Informatica - Oracle DB issue: Escalation Path
Name
Primary
On-call Oracle DBA resource
Manager Escalation
Varinder Kumar
WFM DB issue: Escalation Path
Name
Primary
On-call Oracle DBA resource
Manager Escalation
Varinder Kumar
Network issue: Escalation Path
VMware
Name
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 8 of 28 Runbook Version #: [ ] Primary
On-call Network matrix
Manager Escalation
Alan Holman Will Stokes
(650) 427-3231
(408) 316-7071
Business SMEs: SME’s in any S1/S2 issues related to their subject matter systems SME Interfaces
IT BA Name
IT BA Email
Telecom – Hector Herrera VMStar – Deepa Shah
3.6 How to Contact IEX Support
Support Process 1 – Please click on image below for detailed Support Process VMWare is provided with 24*7*365 Support Coverage with IEX.
VMware
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 9 of 28 Runbook Version #: [ ]
Support Help Desk Line: 1.800.367.4390 Site # for Vmware 35420 Customer Interaction Center: http://cic.iex.com. Please use site # as username Password = “Virtualize” [email protected]: Send documentation on an open incident. (This may include screen shots, workstation traces or other pertinent information. Please reference the incident number within your email.) serial number wfm-35420
3.7 IEX Support Escalations: Support Engineers and Help Desk Queue1.800.367.4390 IEX WFM Support [email protected] Engineers and Incident [email protected] Please follow this order for escalations Jim Draucker Manager, IEX WFM Support(O) 972.643.4617 [email protected](M) 214.934.9863 Scott McLaughlin Manager, IEX WFM Support (O) 972.301.1783 [email protected](M) 972.983.4290 Carl Mullins Manager, IEX WFM Support (O) 972.301.1781 [email protected](M) 972.983.4297 Michael Darby Director, IEX WFM Support, Americas(O) 972.301.1245 [email protected](M) 214.641.4827 Randy Price Vice President, Customer Care(O) 972.301.4890 [email protected](M) 214.563.8824
Customer Interaction Centerhttp://cic.iex.com IEX Customer Services Websitehttp://totalview.iex.com
VMware
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 10 of 28 Runbook Version #: [ ]
VMware
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 11 of 28 Runbook Version #: [ ]
4. ARCHITECTURE 4.1 Architecture Diagrams
User
User
User
User
User
User
User
· Forecasts · Schedules
User
User
User
User
User
User
User
User
User
WFM Application Server
Smart Sync
Smart Sync
Real time reports ( 30 Minute Interval )
Real time SR Information from MySQL
WFM Application Server
WFM Application Server
CMS MySQL
SFDC
Avaya CM WFM Database Server
The charts below show the end-to-end functionality of Informatica throughout all the Applications.
VMware
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 12 of 28 Runbook Version #: [ ] WFM Architecture Flow – Smart Sync SFDC The following diagram depicts the end-to-end flow of data during the work flow management process and the interaction of Informatica with mysql database and sending the file to the target location.
MYSQL
Staging tables
Informatica
Data File
VMware
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 13 of 28 Runbook Version #: [ ]
4.2 Dependencies WFM is dependent upon the following: · WFM Application Server · WFM DB Server · Avaya CMS · Informatica o Oracle database (server and applications) o Mysql o it’s own host: the Informatica repository database
5. HOST INFORMATION Host information for WFM PROD environment in SC9.
5.1 Infrastructure 5.1.1 Network Diagram There is no network diagram.
5.1.2 IP Configuration Information Informatica Server ora-prod-inf-d2.vmware.com WFM Application Servers wfm-prod-app-1
IP 10.113.61.44 IP 10.113.61.232 10.113.61.233 10.113.61.234 DB/PORT WFMPROD/1521 IP 10.19.69.130
wfm-prod-app-2 wfm-prod-app-3 WFM DB Servers ora-prod-gsswfm-d1 CMS Server Avaya CMS Server
5.1.3 Host Information Server
Location
IP Address
OS
Ho st Typ e
Informatica App
inf-prod-appd2.vmware.com
SC9
10.113.61.44
Linux OEL5
VM
Informatica (internally dedicated) DB
ora-prod-infd2.vmware.com
SC9
10.113.61.44
Linux OEL5
VM
WFM Application Server 1
wfm-prod-app-1
Win2008 r2
VM
Description
VMware
SC9
10.113.61.232
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 14 of 28 Runbook Version #: [ ] 10.113.61.233 wfm-prod-app-2 WFM Application Server SC9 Win2008 r2 VM 2 WFM Application Server 3 DB Server
wfm-prod-app-3
ora-prod-gsswfm-d1
SC9 SC9
10.113.61.234
Win2008 r2
VM
Linux
VM
6. OPERATIONAL PROCEDURES Steps to Start WFM process – Informatica Interface:
Step 1: Start ->Programs->Informatica Power Center 8.6.1 -> Client-> Power Center Workflow Manager.
VMware
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 15 of 28 Runbook Version #: [ ]
Step 2: Login to repository by clicking on repository and type the necessary credentials. Please refer to following screen shot below Username: (Credentials given to the User who runs the Loader) Password: (Credentials given to the User who runs the Loader)
VMware
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 16 of 28 Runbook Version #: [ ]
Step 3: After entering Username and Password, Click on Connect as shown in the screen shot below.
Note: The above credentials may not be the exact credentials given to the user who runs the Loader. Step 4: infarep_PROD -> WFM-> Click on open.
VMware
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 17 of 28 Runbook Version #: [ ]
Step 5: Click on Workflows -> wf_WFM_CASES -> Right click on Workflow and Click on Start Workflow option.
VMware
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 18 of 28 Runbook Version #: [ ]
The above step will kick off WFM job .
6.1.2 Steps to modify parameter value: When there is any failure in Informatica job the jobs will be in unscheduled mode. For ex: If the job is failed which is scheduled at 12.40 PM and its observed at 2 PM. Then you can restart the job which will run for 12.00 to 12.30 interval. But remaining missed 30 mins interval should be rerun manually updating the parameter table. After that reschedule the jobs to run current time interval. VMware
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 19 of 28 Runbook Version #: [ ]
NOTE : The time interval should be in GMT format
Following are steps to update parameter value in Parameter file. 1) Login Informatica staging database. (SFA3PROD) 2) SELECT * FROM WFM_INFA_PARAMETERS ; 3) UPDATE WFM_INFA_PARAMETERS SET PAR START_DATE=’ ‘ AND END_DATE=’ ‘ (Modify Parameter_Value) 4) Commit;
VMware
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 20 of 28 Runbook Version #: [ ]
6.1 Admin Procedures and Maintenance – GSS WFM DB Shutdown procedure 1. Login to ora-prod-gsswfm-d1 with your login 2. sudo to oracle 3. set env variable to WFMPROD by running set_DB.sh ( source set_DB.sh) 4. Verify env setted properly by echo $ORACLE_SID and it should return WFMPROD 5. Initiate sqlplus session Sqlplus ‘/ as sysdba’; 6. Select name,open_mode from v$database < -- to make sure you are at correct database 7. Shutdown Instance by typing Shutdown immediate; < -- it make take 2-10 min depending on number of active sessions in database
Startup procedure
1. Login to ora-prod-gsswfm-d1 with your login 2. sudo to oracle 3. set env variable to WFMPROD by running set_DB.sh ( source set_DB.sh) 4. Verify env setted properly by echo $ORACLE_SID and it should return WFMPROD 5. Initiate sqlplus session Sqlplus ‘/ as sysdba’; 6. Startup Select name,open_mode from v$database < -- to make sure it is correct database and open_mode is read write. Database Server Tns Entry WFMPROD = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = ora-prod-gsswfm-d1)(PORT = 1521)) ) (CONNECT_DATA = (SERVICE_NAME = WFMPROD) ) )
Catalog Server tns entry PRODRCAT = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = ora-prod-rcat-d1.vmware.com)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = PRODRCAT.VMWARE.COM) ) VMware
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 21 of 28 Runbook Version #: [ ] )
6.2 Backup and Recovery – GSS WFM DB RMAN Backup: The DB backup involves backing up of the 2 WFM Schema’s (WFMCUSTOMER & WFMYSTEM) RMAN Backup job name : BACKUP_WFMPRD_ONLINE_DAILY scheduled daily at 11 PM Export backup: Export backup for wfmcustomer and wfmsystem user is scheduled via cron. 10 04 * * * /oracle/scripts/exp-wfmprod-user.sh 1>/oracle/scripts/log/exp-wfmprod-user.log 2>&1
6.3 Troubleshooting Scenarios - Informatica Informatica Application Support troubleshooting scenario: Scenario/Reported Condition
Directives
Informatica workflow/session failure
Production Support person will receive Email Notification and Session log attached with it to identify the issue. Or Open workflow monitor and open the corresponding workflow/session logs to identify the issue
Informatica session did not start
Open workflow monitor and open the corresponding workflow/session logs to identify the issue
7.FUNCTIONAL PROCEDURES - Informatica 7.1.1 Functional Support Scenarios 7.1.1.1 Running WFM Process This process would be required to run every thirty minutes:
Drop the target files on Informatica server . (path : /apps/infa/PowerCenter8.6.1/server/infa_shared/tgtFiles/WFM)
2.1 Steps to Start WFM: VMware
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 22 of 28 Runbook Version #: [ ]
Step 1: Start ->Programs->Informatica Power Center 8.6.1 -> Client-> Power Center Workflow Manager.
VMware
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 23 of 28 Runbook Version #: [ ]
Step 2: Login to repository by clicking on repository and type the necessary credentials. Please refer to
following screen shot below Username: (Credentials given to the User who runs the Loader) Password: (Credentials given to the User who runs the Loader)
Step 3: After entering Username and Password, Click on Connect as shown in the screen shot below.
VMware
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 24 of 28 Runbook Version #: [ ]
Note: The above credentials may not be the exact credentials given to the user who runs the Loader. Step 4: infarep_PROD -> WFM-> Click on open.
1. Run the workflow under WFM folder - wf_WFM_CASES which will
VMware
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 25 of 28 Runbook Version #: [ ] Generate the output file. The output file is generated on the Informatica server at /apps/infa/PowerCenter8.6.1/server/infa_shared/TgtFiles/WFM And with the naming convention GMT format Team IT –Informatica Development
Primary Contact
Primary Contact Info
Meghana Kawari
Manager Name Ajit Deshpande
7.1.2 SOX Evidence Not applicable
7.1.3 Emergencies Contact appropriate tier-one resource (see call tree) Route Helpzilla tickets to Informatica queue
7.1.4 Exceptions Not applicable
7.2 Failure Contingencies Contact appropriate tier-one resource (see call tree)
8. SERVICE LEVEL AGREEMENT (SLA) 9. MONITORING AND REPORTS 9.1 Types of Monitoring 9.2 Monitoring Configurations 9.3 Weekly Reports 9.4 Monthly Reports 9.5 Ad-hoc Reports 9.6 Audit Logs
10. Security and Compliance 10.1 10.2 10.3 10.4 10.5 VMware
Change Control SDLC Known Deficiencies User Management Links to Access Control CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 26 of 28 Runbook Version #: [ ]
10.6 SOX Evidence Not applicable
VMware
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 27 of 28 Runbook Version #: [ ]
Appendix A: Document Change History Version No. 1
Date 7/3/2011
Name (Alias)
Description of Change
Tapash Jena
Created the document
Appendix B: Open and Closed Issues ID
Issue
Resolution
Responsibility
Target Date
Impact Date
Appendix C: Glossary/ Definitions Term
Definition
WFM ETL Infa DB
Workflow management process Extract Transform Load Informatica Database
VMware
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 28 of 28 Runbook Version #: [ ]
Appendix D: Review and Sign-off Below is a list of the required reviewers and approvers: Name
Role
Contact
Signoff Date
I have carefully assessed the Runbook Document for the (System Name). This document has been completed in accordance with the requirements of the VMware System Development Life Cycle methodology. I fully accept and signoff on this document.
NAME & Signature IT Operations (Director Level or above)
DATE
NAME & Signature IT Production Support (Director Level or above)
DATE
VMware
CONFIDENTIAL
Requested By: [VNOC Support] Author: [Rajnish Tahiliani]
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Runbook Version #: [ ]
Table of Contents Table 1. 2. 3.
of Contents ......................................................................................... OVERVIEW ......................................................................................... ROLES AND RESPONSIBILITIES ......................................................... ESCALATIONS ....................................................................................
3.1 3.1.1 3.1.2
Severity Level Definitions ................................................................ 4 Severity 1 ................................................................................ 4 Severity 2 ................................................................................ 4
3.2.1
Escalation Procedures ...................................................................... 5 Time Frames ............................................................................ 6
3.2 3.3
Call Tree ........................................................................................ 6 3.3.1 User Call’s VMSD for a WFM Problem ........................................... 6 3.3.2 VNOC Gets a Monitoring Alarm ................................................... 6
3.4
GSS FAST Team Contact Info: .......................................................... 6
3.5
Infrastructure & Interfaces Team Contact Info: .................................. 7
3.6
How to Contact IEX Support ............................................................. 8
3.7
IEX Support Escalations: ................................................................. 9 ARCHITECTURE ................................................................................ 11
4.1
Architecture Diagrams ................................................................... 11
4.2
Dependencies ............................................................................... 13 HOST INFORMATION ........................................................................ 13
5.1
Infrastructure ............................................................................... 13 5.1.1 Network Diagram .................................................................... 13 5.1.2 IP Configuration Information .................................................... 13 5.1.3 Host Information .................................................................... 13 OPERATIONAL PROCEDURES ............................................................ 14
6.1
Admin Procedures and Maintenance – GSS WFM DB .......................... 20
6.2
Backup and Recovery – GSS WFM DB.............................................. 21
6.3
Troubleshooting Scenarios - Informatica .......................................... 21 FUNCTIONAL PROCEDURES - Informatica ........................................ 21 7.1.1 Functional Support Scenarios ................................................... 21 7.1.1.1 Running WFM Process ................................................................................21 7.1.2 SOX Evidence ......................................................................... 25 7.1.3 Emergencies .......................................................................... 25 7.1.4 Exceptions ............................................................................. 25
7.2
Failure Contingencies .................................................................... 25 SERVICE LEVEL AGREEMENT (SLA)................................................... 25
4.
5.
6.
7.
8.
2 4 4 4
VMware
CONFIDENTIAL
Page 2 of 28
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Runbook Version #: [ ] 9.
10.
MONITORING AND REPORTS ............................................................ 25 9.1
Types of Monitoring ...................................................................... 25
9.2
Monitoring Configurations .............................................................. 25
9.3
Weekly Reports ............................................................................ 25
9.4
Monthly Reports ........................................................................... 25
9.5
Ad-hoc Reports............................................................................. 25
9.6
Audit Logs ................................................................................... 25 Security and Compliance .................................................................. 25
10.1
Change Control ............................................................................ 25
10.2
SDLC .......................................................................................... 25
10.3
Known Deficiencies ....................................................................... 25
10.4
User Management ......................................................................... 25
10.5
Links to Access Control .................................................................. 25
10.6 Appendix Appendix Appendix Appendix
VMware
SOX Evidence ............................................................................... 25 A: Document Change History ...................................................... 26 B: Open and Closed Issues ......................................................... 26 C: Glossary/ Definitions .............................................................. 26 D: Review and Sign-off ............................................................... 27
CONFIDENTIAL
Page 3 of 28
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 4 of 28 Runbook Version #: [ ]
1. OVERVIEW The GSS Business team must effectively manage and track the time of its customer-facing agents, which include Technical Support Engineers (TSEs), Customer Service Representatives (CSRs) and Licensing Representatives. In the long-term VMware may consider expanding the user base to external contractors and service providers. With a global operations footprint, it has become critical to be able to optimally manage available capacity to match customer demand. VMware GSS currently has very limited capabilities for forecasting and scheduling without significant overhead. Additionally, the organization lacks mature resource forecasting for its short/long term needs and the ability to effectively manage sudden peak demand. Rudimentary performance tracking and resource availability are done with legacy tools like Excel.
2. ROLES AND RESPONSIBILITIES Informatica Team: Resolve Informatica maintenance, performance, and troubleshooting on production Informatica server and data flow from the informatica server to the Work-force management application server. Telecom Team: Responsible for providing support during a data flow issue from CMs to Work-force management application server VNOC Team: Response and escalation for Severity 1 and 2 issues for WFM Application DBA Team: Resolve database maintenance, performance, and troubleshooting on production Informatica server Network Team: Resolve network maintenance, performance, and troubleshooting on production Informatica server Linux Server Team: Resolve server maintenance, performance, and troubleshooting on production Informatica server Windows Server Team: Resolve server maintenance, performance, and troubleshooting on production workforce management application servers
3. ESCALATIONS 3.1 Severity Level Definitions 3.1.1 Severity 1 HelpZilla: Critical issues affecting large numbers of people (e.g., entire campus, building, or regional location). Response time: 30 minutes or less. VNOC: LEVEL 1 Incident. Telecom System (CMS) is down. Examples of LEVEL 1 Incident: Gss team member are not getting RTA on WFM Application
VMware
Gss team member will not be able to see SLA report
Impacting entire site or greater
Severe business impact
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 5 of 28 Runbook Version #: [ ]
3.1.2 Severity 2 HelpZilla: Issues affecting an entire department (e.g., Engineering, Sales, Support, Finance, etc). Response time: 1 hour or less. VNOC: LEVEL 2 Incidents. Informatica is down or severely degraded. Examples of LEVEL 2 Incidents: Any of the above systems are down or the performance has degraded.
No workarounds
Impacting certain department or group.
Partial site functional loss.
3.2 Escalation Procedures Please note this escalation process is only if there is an infrastructure / operational level outage within VMWare. All application level issues will be handled by the FAST GSS team. If users call with a problem please contact FAST GSS Team.
Alert Type WFM Server Issue ( Monitoring ) Informatica Availability CPU Threshold
Level 1 (Primary) Contact Person/Team Please call IEX Support off-site (Prama B) or onsite (on-call support resource) Linux/Windows on-call appropriate to the system.
When to Notify Level 2 Person/Team Please notify FAST GSS Team Please notify FAST GSS Team
Memory Threshold
Linux/Windows on-call appropriate to the system
Please call FAST GSS Team
Disk Capacity
Please call FAST GSS Team Linux/windows on-call appropriate to the system
If the server is Unix box and has a database hosted on it, please call DBA and Informatica admin. All other cases notify System Admin
Database Temp space size monitoring
DBA on-call
Database Availability
DBA on-call
Listener Availability
DBA on-call
VMware
Please call FAST GSS Team
Please call FAST GSS Team Please call FAST GSS Team
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 6 of 28 Runbook Version #: [ ]
3.2.1 Time Frames Primary on-call person – please give 15 minutes to respond and call the secondary on-call person after the 15 minutes are over for the primary on-call person. If there is no response from the primary/secondary on-call person, escalate it to the manager in the first 30 minutes.
3.3 Call Tree GSS FAST Team will handle all application level issues. GSS FAST team will notify the users and also call into IEX support as appropriate. 1. GSS End users (managers/supervisors, front line staff) would report application issues (such as with forecasts, schedules, agent data corrections) with the FAST GSS team. The issues will be reported via Help-SFDC-Zilla or can be sent directly to [email protected]. Issues created in the SFDCZilla are sent to this email address as well. The FAST GSS WFM team (Bhaskar, Karthik, Ashwin) and the Futuresoft consultants subscribe to this email alias. 2. The FAST GSS WFM team (IEX secondary contacts) can directly report an issue with IEX that is related to IEX application functionality or behavior. VMware issues can be tracked on IEX web portal using our credentials.
Level 1 Support:
3.3.1
User Call’s VMSD for a WFM Problem
Contact the FAST GSS Team as per escalation process defined above. In case of an operations / infrastructure issue please contact the Infrastructure & Interfaces Team Contact Info:
3.3.2
VNOC Gets a Monitoring Alarm
Please contact as per Escalations
3.4 GSS FAST Team Contact Info: Escalation Path
Name
Office Phone
Primary
GSS FAST
During APAC Business Hours
Bhaskar Babu
None
“
Ashwin Prasad
None
“
Karthik Venkatraman
None
During EMEA Business Hours
John McDermott
During Americas Business Hours Manager Escalation VMware
Cell Phone
+91 966 387 4003 +91 959 114 4998 +91 994 584 4255
353 21 4660290
353 86 7714575
Jon Campbell
303 942 4867
303 601 7812
Lynn Adamson
650 427-4317
415 794-4762 CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 7 of 28 Runbook Version #: [ ]
3.5 Infrastructure & Interfaces Team Contact Info: Informatica Issue: Escalation Path
Name
On-site Primary (Level 1 and 2)
Prama B
Off-site Primary (Level 1 and 2)
On-call Informatica resource
On-site/Off-site Manager (Level 1 and 2)
Rajesh Karale
Office Phone
Cell Phone
Office Phone
Cell Phone
Telecom Issue: Escalation Path
Name
Primary
Hector Herrera
Manager Escalation
Will Clark
1-650-427-5528 1-650-427-5775
Informatica - DNBIM Database Issue: Escalation Path
Name
Primary
On-call Oracle DBA resource
Manager Escalation
Varinder Kumar
Office Phone
Cell Phone
(650) 427-5938
(408) 203-8602
Office Phone
Cell Phone
(650) 427-5605
(408) 203-8103
Office Phone
Cell Phone
(650) 427-5938
(408) 203-8602
Office Phone
Cell Phone
(650) 427-5938
(408) 203-8602
Office Phone
Cell Phone
Informatica - Oracle Apps Issue: Escalation Path
Name
Primary
On-call Oracle Apps resource
Manager Escalation
Kiran Chinya
Informatica - Oracle DB issue: Escalation Path
Name
Primary
On-call Oracle DBA resource
Manager Escalation
Varinder Kumar
WFM DB issue: Escalation Path
Name
Primary
On-call Oracle DBA resource
Manager Escalation
Varinder Kumar
Network issue: Escalation Path
VMware
Name
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 8 of 28 Runbook Version #: [ ] Primary
On-call Network matrix
Manager Escalation
Alan Holman Will Stokes
(650) 427-3231
(408) 316-7071
Business SMEs: SME’s in any S1/S2 issues related to their subject matter systems SME Interfaces
IT BA Name
IT BA Email
Telecom – Hector Herrera VMStar – Deepa Shah
3.6 How to Contact IEX Support
Support Process 1 – Please click on image below for detailed Support Process VMWare is provided with 24*7*365 Support Coverage with IEX.
VMware
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 9 of 28 Runbook Version #: [ ]
Support Help Desk Line: 1.800.367.4390 Site # for Vmware 35420 Customer Interaction Center: http://cic.iex.com. Please use site # as username Password = “Virtualize” [email protected]: Send documentation on an open incident. (This may include screen shots, workstation traces or other pertinent information. Please reference the incident number within your email.) serial number wfm-35420
3.7 IEX Support Escalations: Support Engineers and Help Desk Queue1.800.367.4390 IEX WFM Support [email protected] Engineers and Incident [email protected] Please follow this order for escalations Jim Draucker Manager, IEX WFM Support(O) 972.643.4617 [email protected](M) 214.934.9863 Scott McLaughlin Manager, IEX WFM Support (O) 972.301.1783 [email protected](M) 972.983.4290 Carl Mullins Manager, IEX WFM Support (O) 972.301.1781 [email protected](M) 972.983.4297 Michael Darby Director, IEX WFM Support, Americas(O) 972.301.1245 [email protected](M) 214.641.4827 Randy Price Vice President, Customer Care(O) 972.301.4890 [email protected](M) 214.563.8824
Customer Interaction Centerhttp://cic.iex.com IEX Customer Services Websitehttp://totalview.iex.com
VMware
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 10 of 28 Runbook Version #: [ ]
VMware
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 11 of 28 Runbook Version #: [ ]
4. ARCHITECTURE 4.1 Architecture Diagrams
User
User
User
User
User
User
User
· Forecasts · Schedules
User
User
User
User
User
User
User
User
User
WFM Application Server
Smart Sync
Smart Sync
Real time reports ( 30 Minute Interval )
Real time SR Information from MySQL
WFM Application Server
WFM Application Server
CMS MySQL
SFDC
Avaya CM WFM Database Server
The charts below show the end-to-end functionality of Informatica throughout all the Applications.
VMware
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 12 of 28 Runbook Version #: [ ] WFM Architecture Flow – Smart Sync SFDC The following diagram depicts the end-to-end flow of data during the work flow management process and the interaction of Informatica with mysql database and sending the file to the target location.
MYSQL
Staging tables
Informatica
Data File
VMware
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 13 of 28 Runbook Version #: [ ]
4.2 Dependencies WFM is dependent upon the following: · WFM Application Server · WFM DB Server · Avaya CMS · Informatica o Oracle database (server and applications) o Mysql o it’s own host: the Informatica repository database
5. HOST INFORMATION Host information for WFM PROD environment in SC9.
5.1 Infrastructure 5.1.1 Network Diagram There is no network diagram.
5.1.2 IP Configuration Information Informatica Server ora-prod-inf-d2.vmware.com WFM Application Servers wfm-prod-app-1
IP 10.113.61.44 IP 10.113.61.232 10.113.61.233 10.113.61.234 DB/PORT WFMPROD/1521 IP 10.19.69.130
wfm-prod-app-2 wfm-prod-app-3 WFM DB Servers ora-prod-gsswfm-d1 CMS Server Avaya CMS Server
5.1.3 Host Information Server
Location
IP Address
OS
Ho st Typ e
Informatica App
inf-prod-appd2.vmware.com
SC9
10.113.61.44
Linux OEL5
VM
Informatica (internally dedicated) DB
ora-prod-infd2.vmware.com
SC9
10.113.61.44
Linux OEL5
VM
WFM Application Server 1
wfm-prod-app-1
Win2008 r2
VM
Description
VMware
SC9
10.113.61.232
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 14 of 28 Runbook Version #: [ ] 10.113.61.233 wfm-prod-app-2 WFM Application Server SC9 Win2008 r2 VM 2 WFM Application Server 3 DB Server
wfm-prod-app-3
ora-prod-gsswfm-d1
SC9 SC9
10.113.61.234
Win2008 r2
VM
Linux
VM
6. OPERATIONAL PROCEDURES Steps to Start WFM process – Informatica Interface:
Step 1: Start ->Programs->Informatica Power Center 8.6.1 -> Client-> Power Center Workflow Manager.
VMware
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 15 of 28 Runbook Version #: [ ]
Step 2: Login to repository by clicking on repository and type the necessary credentials. Please refer to following screen shot below Username: (Credentials given to the User who runs the Loader) Password: (Credentials given to the User who runs the Loader)
VMware
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 16 of 28 Runbook Version #: [ ]
Step 3: After entering Username and Password, Click on Connect as shown in the screen shot below.
Note: The above credentials may not be the exact credentials given to the user who runs the Loader. Step 4: infarep_PROD -> WFM-> Click on open.
VMware
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 17 of 28 Runbook Version #: [ ]
Step 5: Click on Workflows -> wf_WFM_CASES -> Right click on Workflow and Click on Start Workflow option.
VMware
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 18 of 28 Runbook Version #: [ ]
The above step will kick off WFM job .
6.1.2 Steps to modify parameter value: When there is any failure in Informatica job the jobs will be in unscheduled mode. For ex: If the job is failed which is scheduled at 12.40 PM and its observed at 2 PM. Then you can restart the job which will run for 12.00 to 12.30 interval. But remaining missed 30 mins interval should be rerun manually updating the parameter table. After that reschedule the jobs to run current time interval. VMware
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 19 of 28 Runbook Version #: [ ]
NOTE : The time interval should be in GMT format
Following are steps to update parameter value in Parameter file. 1) Login Informatica staging database. (SFA3PROD) 2) SELECT * FROM WFM_INFA_PARAMETERS ; 3) UPDATE WFM_INFA_PARAMETERS SET PAR START_DATE=’ ‘ AND END_DATE=’ ‘ (Modify Parameter_Value) 4) Commit;
VMware
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 20 of 28 Runbook Version #: [ ]
6.1 Admin Procedures and Maintenance – GSS WFM DB Shutdown procedure 1. Login to ora-prod-gsswfm-d1 with your login 2. sudo to oracle 3. set env variable to WFMPROD by running set_DB.sh ( source set_DB.sh) 4. Verify env setted properly by echo $ORACLE_SID and it should return WFMPROD 5. Initiate sqlplus session Sqlplus ‘/ as sysdba’; 6. Select name,open_mode from v$database < -- to make sure you are at correct database 7. Shutdown Instance by typing Shutdown immediate; < -- it make take 2-10 min depending on number of active sessions in database
Startup procedure
1. Login to ora-prod-gsswfm-d1 with your login 2. sudo to oracle 3. set env variable to WFMPROD by running set_DB.sh ( source set_DB.sh) 4. Verify env setted properly by echo $ORACLE_SID and it should return WFMPROD 5. Initiate sqlplus session Sqlplus ‘/ as sysdba’; 6. Startup Select name,open_mode from v$database < -- to make sure it is correct database and open_mode is read write. Database Server Tns Entry WFMPROD = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = ora-prod-gsswfm-d1)(PORT = 1521)) ) (CONNECT_DATA = (SERVICE_NAME = WFMPROD) ) )
Catalog Server tns entry PRODRCAT = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = ora-prod-rcat-d1.vmware.com)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = PRODRCAT.VMWARE.COM) ) VMware
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 21 of 28 Runbook Version #: [ ] )
6.2 Backup and Recovery – GSS WFM DB RMAN Backup: The DB backup involves backing up of the 2 WFM Schema’s (WFMCUSTOMER & WFMYSTEM) RMAN Backup job name : BACKUP_WFMPRD_ONLINE_DAILY scheduled daily at 11 PM Export backup: Export backup for wfmcustomer and wfmsystem user is scheduled via cron. 10 04 * * * /oracle/scripts/exp-wfmprod-user.sh 1>/oracle/scripts/log/exp-wfmprod-user.log 2>&1
6.3 Troubleshooting Scenarios - Informatica Informatica Application Support troubleshooting scenario: Scenario/Reported Condition
Directives
Informatica workflow/session failure
Production Support person will receive Email Notification and Session log attached with it to identify the issue. Or Open workflow monitor and open the corresponding workflow/session logs to identify the issue
Informatica session did not start
Open workflow monitor and open the corresponding workflow/session logs to identify the issue
7.FUNCTIONAL PROCEDURES - Informatica 7.1.1 Functional Support Scenarios 7.1.1.1 Running WFM Process This process would be required to run every thirty minutes:
Drop the target files on Informatica server . (path : /apps/infa/PowerCenter8.6.1/server/infa_shared/tgtFiles/WFM)
2.1 Steps to Start WFM: VMware
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 22 of 28 Runbook Version #: [ ]
Step 1: Start ->Programs->Informatica Power Center 8.6.1 -> Client-> Power Center Workflow Manager.
VMware
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 23 of 28 Runbook Version #: [ ]
Step 2: Login to repository by clicking on repository and type the necessary credentials. Please refer to
following screen shot below Username: (Credentials given to the User who runs the Loader) Password: (Credentials given to the User who runs the Loader)
Step 3: After entering Username and Password, Click on Connect as shown in the screen shot below.
VMware
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 24 of 28 Runbook Version #: [ ]
Note: The above credentials may not be the exact credentials given to the user who runs the Loader. Step 4: infarep_PROD -> WFM-> Click on open.
1. Run the workflow under WFM folder - wf_WFM_CASES which will
VMware
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 25 of 28 Runbook Version #: [ ] Generate the output file. The output file is generated on the Informatica server at /apps/infa/PowerCenter8.6.1/server/infa_shared/TgtFiles/WFM And with the naming convention GMT format Team IT –Informatica Development
Primary Contact
Primary Contact Info
Meghana Kawari
Manager Name Ajit Deshpande
7.1.2 SOX Evidence Not applicable
7.1.3 Emergencies Contact appropriate tier-one resource (see call tree) Route Helpzilla tickets to Informatica queue
7.1.4 Exceptions Not applicable
7.2 Failure Contingencies Contact appropriate tier-one resource (see call tree)
8. SERVICE LEVEL AGREEMENT (SLA) 9. MONITORING AND REPORTS 9.1 Types of Monitoring 9.2 Monitoring Configurations 9.3 Weekly Reports 9.4 Monthly Reports 9.5 Ad-hoc Reports 9.6 Audit Logs
10. Security and Compliance 10.1 10.2 10.3 10.4 10.5 VMware
Change Control SDLC Known Deficiencies User Management Links to Access Control CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 26 of 28 Runbook Version #: [ ]
10.6 SOX Evidence Not applicable
VMware
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 27 of 28 Runbook Version #: [ ]
Appendix A: Document Change History Version No. 1
Date 7/3/2011
Name (Alias)
Description of Change
Tapash Jena
Created the document
Appendix B: Open and Closed Issues ID
Issue
Resolution
Responsibility
Target Date
Impact Date
Appendix C: Glossary/ Definitions Term
Definition
WFM ETL Infa DB
Workflow management process Extract Transform Load Informatica Database
VMware
CONFIDENTIAL
Error! Unknown document property name.Error! Unknown document property name.Error! Unknown document property name.[Project Name] Page 28 of 28 Runbook Version #: [ ]
Appendix D: Review and Sign-off Below is a list of the required reviewers and approvers: Name
Role
Contact
Signoff Date
I have carefully assessed the Runbook Document for the (System Name). This document has been completed in accordance with the requirements of the VMware System Development Life Cycle methodology. I fully accept and signoff on this document.
NAME & Signature IT Operations (Director Level or above)
DATE
NAME & Signature IT Production Support (Director Level or above)
DATE
VMware
CONFIDENTIAL
