Performance Testing

Summary: Performance in a Web application is of utmost importance. This paper discusses the various types of testing that should be performed while measuring the performance of a Web application, the different workload models, and the ways of measuring response times using automated tools. A good performance tool aids the testing process and can be the smartest way of measuring performance in a project. End-to-end performance is the last stage of the project in a typical Web development iteration cycle. This paper also talks about some crucial design factors that improve the performance of a system. 1. What Is the Difference between Load and Stress Testing? A load test is a simulation of real-life use of a Web application, performed to assess how the application will work under actual conditions of use. The test is performed under expected conditions both prior to application deployment and during the life cycle of the application. A stress test is designed to determine how heavy a load the Web application can handle. A huge load is generated as quickly as possible to stress the application to its limit. To guarantee a realistic simulation of the workload created by actual users, the test environment must be as close to the real environment as possible. A good load test should generate different types of traffic expected, using actual computer software and hardware. 2. What Is a Workload? A workload is the total burden of activity placed on the Web application to be tested. This burden consists of a certain number of virtual users who process a defined set of transactions in a specified time period. Assigning a proper workload is at the crux of performance testing. 3. Typical Workload Models: • • • •

A steady state workload is the simplest workload model used in load testing. A constant number of virtual users are run against the Web application for the duration of the test. An increasing workload model helps testers to find the limit of a Web application’s work capacity. At the beginning of the load test, only a small number of virtual users are run. Virtual users are then added to the workload step by step. In a dynamic workload model, you can change the number of virtual users in the test while it is being run and no simulation time is fixed. In a queuing workload model, transactions are scheduled following a prescribed arrival rate. This rate is a random value based on an average interval calculated from the simulation time and the number of transactions specified.

4. Understanding the Architecture and Test Environment Before choosing the right model for performance tests, it is important for us to understand the architecture. Testing in an environment that does not mimic production provides wrong results. It’s crucial to know the specifications of the Web server, databases, or any other external dependencies the application might have. Building the test environment as close to production as possible makes the testing effort a lot easier and is the key to providing accurate results. 5. Writing Automated Scripts In any tool, record and playback is a common feature. But automated testers should realize that recording and playing back the Web traffic that you recorded from the application won’t help you simulate the workload. Once have your basic recorded script, you have to: parameterize the script by declaring variables and creating data files associated with the script; call functions to parse through the data files; store the values; and input them to the script. A good example to

consider would be to simulate multiple logins in an application. In this case you might want to store the user names and passwords in a comma-separated value file or some other format. It’s also a good idea to randomize certain data wherever possible to ensure complete coverage of data. You can also add think times to the script while performing a stress test. Verifying the headers and HTML or XML content returned from the server involves writing additional functions in the automated script. Once the basic framework for the script is complete, the next step would be to structure the script based on the scenarios that you want to include in the load or stress tests. Typical Randomization Factors

6. Structure and Format the Automated Script Depending on the mix of transactions that you want to include in the automated script, you can create different types of virtual users by defining user types within the automated script. Once you have designed the user types and the actions that each user type should perform, you are pretty much ready to run the script. This enables you to incorporate these user types while running different workload models, giving you the capability to add different flavors to your tests. Keeping all functions and methods together in a separate file makes the automated script shorter and easy to read and maintain. 7. Baselining and Benchmarking the Tests A benchmark test is basically conducted to profile performance for different software releases. Benchmarking helps you to run tests under similar conditions in order to compare results for different software, to verify that new features and functions do not impact response times of the Web application under test, and to find system bottlenecks. Defining what each virtual user does and measuring response times for a single virtual user performing a certain transaction is what baselining is all about. Once you have data for a single user, you can then ramp up the number of users linearly to "n" users and measure the response times to look for deviations. 8. Determining the Capacity of the System In our case, once we had the baseline test in place, we increased the number of virtual users to find the system limit. We encountered a good number of errors when we ramped up to about ten users. Typical errors were socket errors, access denied, socket connection refused, etc., depending on the configuration of the Web server. This may happen if a simulated version of software is used or any of the dependencies for the application are not available at the time of performing the test. This may pose a serious problem, because the response times would not be an exact replication of the production environment. Under these circumstances we should know when to stop. Once we know the threshold number of virtual users the Web server can handle, we can specify our workload scenarios for our load and stress tests.

9. Designing Test Variables Test variables can be a good way of altering the workload by controlling these variables. These variables will depend on the maturity of the software that’s being tested. The variables are named Vx (e.g. V1, V2, etc).

10. What to Measure? Most performance tools measure round-trip response times. "Round trip" can be defined as the time taken between the request being sent from the client and the response received from the server. Round-trip time includes the network time that is not a constant depending on the environment in which the test is conducted, the external dependencies on databases, and network latency. Hence, round-trip response times are not an accurate indicator of the performance of the application. You should measure the server-side response times from the server logs and compare them with the round-trip times to chart how the response times vary depending on the network traffic during different times of the day. It might also be a good idea to run an all-day workload and get measurement of response times at sample intervals of time to get an average number. In our case, we also measured parameters like average number of transactions/sec, hits/sec, throughput, etc., as shown in Graphs A, B, and C. Graph A

Graph B

Graph C

11. Reporting Performance Numbers Performance numbers are typically reported as average, minimum, and maximum response times. If the performance test runs for a specified time, the average response time would be the most important indicator of performance. Maximum and minimum response times provide a means to investigate why a certain transaction with a certain input indicated a good or bad response time. Graph D helped us in analyzing response times. Graph D

12. Monitoring the Web Server Monitoring the Web server is an integral part of performance engineering. While running different workloads with multiple users, we always logged performance counters like processor time, private bytes, thread count, handle count, etc. In most cases, our hardware/processor was not able to handle the load and the CPU time exceeded 100%; hence, the results were not accurate. It’s almost imperative that performance counters be logged and analyzed. This can prevent unpleasant situations when the application is being tested in the production environment, and you also get an estimate of how many virtual users your environment is able to handle during your performance integration tests. 13. Conclusion End-to-end testing of performance in Web applications is a complex and time-consuming task. The proposed methodical approach in this paper not only ensures a structured way of testing, but also provides a practical way of enhancing performance of a Web application by applying these techniques at every phase or iteration in a Web development life cycle. References "Designing Performance into Your Web-Based Applications," IBM Performance Management and Capacity Planning Services