APPLICATION MANAGEMENT
ANALYZING PERFORMANCE AND AVAILABILITY: BEST PRACTICES FOR ROOT-CAUSE ANALYSIS AND TRIAGE
EXECUTIVE OVERVIEW Many applications and monitoring tools in today’s market will send an alert when they think there is a systems problem. However, most of these alerts are due to minor systems variations or issues that don’t actually impact end users in any meaningful way. You can best work through this noise to identify real problems and quickly resolve them by using a top-down approach from the end-user perspective. By watching what your business processes are trying to accomplish instead of how they do it, you can improve your service levels and reduce the number of false alerts you have to handle. This white paper describes how you can use Mercury Business Availability Center™ to monitor performance and availability from an end-user perspective to improve your production triage and root-cause analysis processes. By applying these basic principles, Mercury can help you fundamentally simplify the complex task of managing IT.
TABLE OF CONTENTS Executive Overview …………………………………………………………………2
Problem Identification and Triage Example ………………………………………5
Analyzing Application Performance and Availability ……………………………3
Mercury Managed Services …………………………………………………………7
Mercury Business Availability Center………………………………………………3
Summary ………………………………………………………………………………7
Application Management Dashboard and Reports …………………………4
Appendix: Performance Analysis Decision Tree …………………………………7
Mercury End User Management ………………………………………………5
APPLICATION MANAGEMENT
Analyzing Application Performance and Availability Today’s IT departments are faced with the daunting task of ensuring that the enterprise’s mission-critical applications are always available and performing at peak levels. This challenge is made more difficult by the growing complexity of today’s heterogeneous computing environments and the increased pressure to make applications available around-the-clock, and around the world. Uncontrolled change and complexity within your company’s operations and infrastructure can result in unplanned downtime, sub-standard application performance, and unmet business goals — all of which present unacceptable consequences to today’s enterprises. In order to survive in this difficult environment, you need sophisticated, integrated application management solutions that can quickly and easily master this constant change and ever-growing complexity. Effective application management solutions must provide the ability to: • Monitor all of the enterprise’s mission-critical applications. • Detect customer problems before they impact the business or the end user. • Monitor complex, heterogeneous computing environments. • Prioritize IT operational activities based on business and end-user needs. • Effectively measure and manage application service-level agreements (SLAs).
Mercury Business Availability Center Mercury Business Availability Center provides complete visibility into and control over the end-user experience and availability of applications running in complex, distributed application environments. It is the only solution that offers an end-to-end, business-centric approach to application management — providing the ability to map, measure, and manage application, system, and infrastructure performance and availability according to end-user, service-level, and business goals. Mercury Business Availability Center provides a comprehensive set of tools, processes, reports, and best practices to facilitate all of the steps involved in a complete application management program. With Mercury Business Availability Center offerings, you can: • Obtain real-time, end-to-end visibility into the complex and changing relationships between applications and their underlying infrastructure. • Align IT with the enterprise’s business priorities and expectations. • Receive proactive alerts for all service disruptions. • Define and measure business-centric service levels. • Reduce security breaches with real-time service-level visibility. • Prioritize problem resolution and perform triage based on potential business impact and service-level compliance.
WWW.MERCURY.COM
3
APPLICATION MANAGEMENT
BUSINESS AVAILABILITY CENTER Application Management Dashboard Business Availability
End User Management
Problem Resolution
System Availability Management (SiteScope)
Service Level Management
Application Mapping Diagnostics (J2EE, .Net, SAP, Siebel)
Analytics
Application Management Foundation Administration Alerts and Configuration
Automatic Discovery Applications & Infrastructure
End User Monitoring Business Process, Client, Real User
Infrastructure Monitoring SiteScope, 3rd Party Adapters
Solutions available for: J2EE, .Net, SAP, Siebel, Oracle, PeopleSoft MERCURY SERVICES Managed Service, Consulting, Education, Support Mercury Business Availability Center offerings.
Application Management Dashboard and Reports
Mercury Business Availability Center offers an intuitive user interface that facilitates application management. All data and reports generated by Mercury Business Availability Center are available for viewing through Mercury Application Management Dashboard™. This console enables IT and business users to view a comprehensive display of performance and availability metrics from one unified control center. The top level of the dashboard gives an integrated view of all applications and business processes that are essential to the enterprise. The intuitive interface enables quick identification of all areas in the infrastructure that need attention — with green indicating OK, yellow for warning, and red indicating full alert or complete failure of a process or component. When you move the cursor over any item on the dashboard, additional information is displayed, allowing you to click on the process or component to view more detailed reports. You can drill down from the main display to see more detail on the underlying infrastructure associated with these key business processes. Drill-down views and reports are easily customized, and can be organized by data center, technology cluster, or by geography. Mercury Business Availability Center also offers a wide selection of application management reports that reveal real-time, end-user data and can be leveraged to assess and prioritize resources to meet SLAs or ensure a positive end-user experience.
WWW.MERCURY.COM
4
APPLICATION MANAGEMENT
Mercury Application Management Dashboard and performance reports facilitate application monitoring, end-user and system availability management, service-level management, application relationship mapping, diagnostics, analytics, and problem resolution. This comprehensive approach to application management enables enterprises to accelerate time-to-value, lower TCO, and maximize ROI. Mercury End User Management
One key Mercury Business Availability Center offering is Mercury End User Management™. Mercury Application Management Dashboard’s color-coded indicators display green for components or processes that are behaving correctly, yellow for areas that need immediate attention, and red if the process or component has failed completely.
Mercury End User Management monitors application availability in real-time, from the end-
user perspective. It proactively emulates end-user business processes against applications on a 24x7 basis for over 50 protocols — web, client/server, mainframe, and wireless — as well as packaged applications including Siebel, SAP, PeopleSoft, Oracle, Citrix, and others. Core to the Mercury End User Management application is its ability to accurately measure the end-user experience of business processes embedded within applications. You can drive transactions from multiple locations, both from inside the company firewall or from external locations leveraging Mercury’s worldwide measurement locations. Mercury End User Management provides the ability to manage performance from a wide number of supported desktops, handheld, and devices, enabling IT to accurately gauge the impact of outages on customers and assess the business impact on company productivity and revenue.
Problem Identification and Triage Example
BUSINESS AVAILABILITY
PROBLEM RESOLUTION
Application management is comprised of many different stages and processes, including the
Align Business and IT Operations
alignment of business and IT operations,
Define Key Objectives & Controls
monitoring application availability and
Business Impact
Service Levels
performance, problem resolution, and the
Resolve Problem Tier 1 Classification & Escalation
Tier 2 Application Triage
Tier 3 Application Diagnostics
Business Availability Plan
prevention of application downtime. The following example will help illustrate some
Monitor Availability and Performance
of the processes and reports that can be used
MAP Applications to Underlying Infrastructure
to diagnose an application performance
Measure Users, Applications, Infrastructure
problem. (Note: For a complete “Decision Tree” that will help in the design of a comprehensive
Manage Customer Impact, Service Contracts, Problem Escalation
Prevent Application Downtime Performance and Capacity Planning
Change and Configuration Management
performance analysis process, please see the This diagram illustrates the many phases and stages of an application management process.
end of this paper.)
WWW.MERCURY.COM
5
APPLICATION MANAGEMENT
Step 1: Operations personnel are receiving intermittent end-user performance alerts from Mercury Application Management Dashboard’s proactive business process monitors. At this point, they see that their systems monitors are all reporting that the servers are performing as expected. This leads the operations team to drill down into the locations report to try to isolate the performance problem. Step 2: Looking at the average response time over time report, we see that some locations (e.g., Singapore) have a much longer response time then would normally be expected — in this case more than double the normal end-user response time. At this stage, the operations team suspects a network issue, but drills into the
Step 1
transaction breakdown report to be certain. Step 3: We first want to look at the average of all transactions across all locations to be able to understand what the normal enduser performance is. In this case, we see that we have an average transaction baseline of about eight seconds, with about 5.5 seconds of that being consumed by the network time to first buffer. Step 4: Next we focus only on Singapore. What we quickly see is that Singapore in general has a much worse end-user response time, almost double the normal amount at 15 seconds. Most of this time is spent on the network. We also see a clear spike in
Step 2
download time and network time to first buffer. Let’s drill into network analysis isolate the problem. Conclusion: The graphs clearly show a spike in network latency at the same time as the end users’ performance degraded. The spike affects a number of routers all in Level 3’s network. At this point we have identified that the performance problems are caused by the network at Level 3. We now have actionable information we can hand over to our ISP partners to enable them to more quickly resolve the problem our users in Singapore have experienced.
Step 3
Step 4 Conclusion WWW.MERCURY.COM
6
APPLICATION MANAGEMENT
Mercury Managed Services
Performance Analysis Decision Tree
Deploying software through Mercury Managed View ‘Average response time over time’ graph for all transactions. Note if all transactions are experiencing an increased time.
Services™ enables your enterprise to generate value in the shortest time, while reducing total cost of
No
Is increased time occurring from all locations?
ownership (TCO). Even if you want to move to an
[might be network or agent issue]
Yes [might be application issue]
in-house solution eventually, Mercury Managed Services provides the fastest time-to-implementation
View transaction breakdown over time on daily. Note which sections of the transaction time take more time. For details on transaction breakdown, check the “Understanding the Transaction Breakdown Reports” section.
and is the best way to achieve the greatest value
Make a note of which locations are having response time problems. Note if there are any patterns in locations or ISPs. Set Active Filters to only focus on those locations which are having the problem.
from all Mercury Business Availability Center offerings.
Has server time increased?
Mercury Managed Services includes everything Yes
needed for a rapid, successful deployment of
Has client time increased?
Has network time increased?
Yes
Yes
Server time includes: - DNS Resolution - Connection Time - SSL Handshaking Time
Network time includes: - Network Time to First Buffer - Download Time
- Server Time to First Buffer - Download Time
Mercury Business Availability Center, including
Inform appropriate personnel that server (typically application server) is having problems. Note transaction in question and time problem started.
implementing the products, enabling the IT staff, and optimizing all processes. Mercury provides an
Alert your operations group to problem. Contact MMS support. Have them verify that agent is functioning correctly.
Go to Webtrace. View daily Webtrace data from several of the locations in question.
infrastructure that is available 24x7, saving the time and expense of setting up and running
Yes
DONE
Have Webtrace times increased?
Problem resolved?
N No
B
Mercury Business Availability Center implementations in-house.
Summary Mercury Business Availability Center enables your
No No
Yes Yes
Submit Service Request to MMS to modify script or debug the problem (include error message & screenshots).
Go to Webtrace individual hops. Make note of IP addresses showing problems. Forward it along with error messages to Network Engineering & appropriate personnel.
enterprise to quickly diagnose and proactively fix potentially harmful performance and availability B
issues before end users experience them. With Mercury Business Availability Center offerings, you
No
can reduce mean time to problem identification, View transaction breakdown during problem period and prior to problem time.
improve service-level performance, minimize application downtime, and lower the TCO of the managing applications and infrastructure — all leading to improved application performance with
Has data size increased
higher customer satisfaction.
Site may have changed. Verify site change with appropriate personnel.
No
For more information about Mercury Business Does your server use HTTP 1.1 data chunking?
Availability Center, Mercury End User Management, Mercury Managed Services, or other Mercury
No
Contact MMS support to try to determine cause for shift in network time.
Yes
services and solutions, contact your local Mercury sales representative or visit www.mercury.com.
Network time increase may be due to server lag later in the response. Contact MMS support to verify
Basic Performance Analysis Process.
WWW.MERCURY.COM
Yes
7
Problem resolved?
No
Submit Service Request to MMS to modify script or debug the problem (include error message & screenshots)
Yes
DONE
Mercury is the global leader in business technology optimization (BTO). We are committed to helping customers optimize the business value of IT. WWW.MERCURY.COM © 2005 Mercury Interactive Corporation. Patents pending. All rights reserved. The Mercury logo, Mercury Business Availability Center, Mercury Application Management Dashboard, Mercury End User Management, and Mercury Managed Services are are trademarks or registered trademarks of Mercury Interactive Corporation in the United States and/or other foreign countries. All other company, brand, and product names are marks of their respective holders. WP-1397-0405