Building A User Interface Test Automation Framework

Building a user interface test automation framework Izzat Alsmadi and Kenneth Magel Department of Computer Science North Dakota State University Fargo, ND 58105 { izzat.alsmadi, kenneth.magel}@ndsu.edu Abstract This article suggests a fully test automation tool with the least amount of time and decisions required from testers. Using an automated tool should not be offset by testers spending a good amount of time for setup and guidance. The proposed framework for user interface test automation uses the information stored in the application assembly to build the tested model. The GUI model is parsed from the application under test at run time. The GUI model is built as an XML tree. Test cases are then generated from the XML tree using different proposed algorithms. Some techniques for test case prioritization, critical path testing, execution and verification are suggested. The goal is to do all those activities dynamically as part of a GUI test automation framework. since they don’t know how much time it will require upfront for setup. They also don’t know how much of test automation tasks can be fully or partially automated. We all agree that complete coverage in test automation is impossible as we also agree that complete coverage in testing is impossible too. No matter what software methodology a company follows, software testing is going to play a major role in terms of project success or resource allocation. Automated test refers to testing by a computer (no human tester). We choose the option of automating the GUI testing when tests are repeated several times [1]. Software test automation tasks include selecting and generating the test cases, building the test oracle (test cases with expected results), executing, reexecuting, reporting and validating the results. GUIs manage controls. Controls are reusable objects with which users can interact. The term "control" is used as a generic term for any graphical object or widget that an application may produce. Controls have common elements that should be considered before writing a program that interacts with a GUI [2]. • Controls are categorized according to Window classes. Each control belongs to a window class (making it possible to search them by class). • Controls have an organizational hierarchy; every GUI has at least one root control, and every control may have child controls. Controls and forms make a tree. We can generate a valid test case by selecting several controls that are children of each other.

General Terms Software testing, user interface testing and test automation.

Keywords Software test automation, test case generation, Test execution, GUI testing, GUI modeling, and verification.

1. Introduction Software testing is an important stage in software projects. It is one of the most expensive stages that usually stops after using the available resources (and not completed). Test automation takes more time in the initial stages. Eventually, this is expected to reduce the cost of testing through reducing the required time and resources. GUI is increasingly taking a larger portion of the overall program’s size as newer programs are more interactive with the user inputs. and GUI testing is taking a major rule in the whole project’s validation. GUI test automation is a major challenge for test automation activities. Most of the current GUI test automation tools are partially automated and require the involvement of users or testers in several stages. Test automation tools are still complex and expensive. They don’t fully replace testers. They can be usually used to re-execute those repeated tasks. Companies consider taking the decision to buy, or not, a GUI test automation tool as a tough decision

1

to compete with. However, GUI test automation didn’t reach a mature level were those tools can be implemented in all scenarios with no problems. User Interfaces evolves rapidly with more rich and new components. The automation processes in general use artificial intelligence algorithms to simulate user activities. The richness and complexity of this space make it possible for new research ideas to compete with long time existed test automation tools in the industry. There are several research papers about GUI testing using the data model, [3] [4] [5] [7] [8] [9] [10]. The overall goals and approach for this work is very similar to their goals. The GUI testing framework described, as a GUI test automation structure, is generic and should be applied to any testing or GUI testing model. Since research projects in general have limited resources, each paper discusses a specific area of GUI test automation. Without having all components, it is very hard to use such ideas in the industry. There is a need for using universal standards for the output formats in all testing activities. If each tool is producing several intermediate outputs (generated test cases, recorded scripts, log files, etc), it will be impossible to use those tools or ideas in other research projects. Some papers follow a complex procedure in test case generation and do not consider any state reductions. Assuming that changing any property in any GUI object changes the GUI state is an assumption that generated a very large number of possible states for even small applications. State reduction techniques are considered here to improve the effectiveness of the track. We intended to follow the same GUI testing framework for our future work and expect the overall results to be more practical and easier to apply on actual projects. The second category[13][6] is related to semi test automation using some capture/reply tools like WinRunner, QuickTest pro, Segui silk, QARun, Rational Robot, JFCUnit, Abbot and Pounder to creates unit tests for the AUT. Capture/reply tools have been existed and used for years. This may make them currently more reliable and practical as they have been tested and improved through several generations and improvements. There are several problems and issues in using record/play back tools [11]. The need to reapply all test cases when the GUI changes, the complexity in editing the scripts code, and the lack of error handlings are some examples of those issues. The reuse of test oracles is not very useful in the case of using a capture/replay tool [12]. We expect future software projects to be more GUI complicated that may make the test automation data model more practical. Many researches and

• Controls have different ways to be located or identified at run time. They can be identified by their names, parents, unique location or by the window handle associated to them at run time. We present a GUI testing framework that does not require the involvement of testers throughout the different stages. The tests do not need to be revisited during any change in the GUI structure. Since the application uses reflection to get the actual model at run time, those changes are included in the current test. Generating test cases can happen from requirements, design or the actual GUI implementation. Although it is expected that those three should be consistent and related, yet they have different levels of abstraction. Requirements and design are usually of a high level of abstraction to generate from them the test cases. On the other hand the task of generating test cases from the GUI implementation model is delayed until GUI is implemented, which is usually occurred in the late implementation. We should not have any problems in delaying GUI testing giving the fact that a tool will automate the generation and execution processes. A tool is built in C# that uses reflection to serialize the GUI control components. Certain control properties are selected to be serialized. These properties are relevant to the user interface testing. The application then uses the XML file that is created to build the GUI tree or the event flow graph. Test cases are then generated using the GUI XML file. Generating the test cases takes into consideration the tree structure to select the test cases that cover unique branches. In test execution and verification, test cases are executed on the application and compared with the original test suite. The advantages of the object oriented approach over the widely used capture/replay back one, is in the fact that the model is generated at run time which makes it represents the current state of the GUI model. In record/play back cases, we have to retest the application in case of any change in the functionalities or the GUI of the program. Once utilized, this object oriented approach is expected to be less expensive as it does not require users to manually test the application or make decision for pre- and post-conditions.

2. Related work There are several GUI test automation tools available in the market. Some of the larger vendors for such tools includes: IBM Rational, Mercury, and Segue. Trying to build a GUI test automation tool is like trying to make a new operating system in such that the available resources for the existed ones are hard

2

True <Enabled>True <ShortCut>None Figure1: a screen shot from a generated XML file.

improvements need to be done for the suggested data model to be more practical and usable.

3. Goals and approaches Our approach uses two constraints: The tool should develop all test automation activities without or with the least user involvement. The second constraint is that those dynamically generated test case should provide a value for test adequacy and should represent actual test case scenarios. Since the test cases will be dynamically generated with no need for a user to define pre- or postconditions, algorithms are developed for test case generation. One important aspect of GUI’s that helps in state reduction is the fact that the GUI is hierarchical by nature. For example, in MS Notepad, to reach the event “Print”, the event “File” should be executed first. The GUI is analyzed and its hierarchical tree of controls is extracted using reflection. The decision to use the XML file as a saving location for the GUI model is in the fact that XML is a universal format and it supports hierarchy. This hierarchy can be extracted easily from the XML file. We used the information about the parent of each control in the assembly to build the GUI tree. Figure1 is a screen shot from the XML output file extracted from a simple Notepad application.

The above information about the controls and their properties are extracted directly from the assembly. We added two more properties; control level and control unit for each control. Those properties are used along with the parent property to locate each control during test generation, selection and execution. The user has the ability to select certain controls from the tree and gives them more than the typical equally distributed weight. This will have an effect on test case selection algorithms. For example, a control that is selected manually by the user will be selected whenever it is possible or valid to do so. Rather than defining the GUI state change as a change in any control or control property [2], which produces a large amount of possible test cases, we define the GUI state as the state that is represented by the generated XML file from that GUI. If any of the parent-child relations in the GUI is changed, or any property of those parsed controls is changed, then that is considered a GUI state change. Some abstraction is used removing those properties that less relevant or important to the GUI state to reduce the large number of possible states. The manually selected properties are: the control name, parent name, text, locationX (horizontal location), locationY (vertical location), text, or caption, forecolor, backcolor, enabled, visible, control unit and control level. In the process of developing test generation techniques, several test generation algorithms are developed. All algorithms check for a valid selection of a tree edge. For example, using Notepad AUT, if the current control is “File”, then a test algorithm may select randomly a valid next control from the children of the File control ( e.g. Save, SaveAs, Open, Exit, Close, Print). In another algorithm, we processed the selected test scenarios to ensure that no test scenario will be selected twice in a test suite. Figure2 is a sample output generated from one of the test generation algorithms. In the algorithm, each test case starts from the root or the main entry “Notepad Main”, and then selects two or more controls randomly from the tree. The algorithm verifies that the current test case is not existed in the already generated test cases.

GUI-Forms Form1 <Parent-Form>Form1 Form1 0 <Parent-Form>Form1 textBox1 1 0 0 24 Color [DarkBlue] Color [Linen] <Enabled>True False <MenuItem> System.Windows.Forms.MenuItem, Items.Count: 4, Text: File 1 0 <Parent>System.Windows.Forms.MainMenu, Items.Count: 3 File

1,NOTEPADMAIN,SAVE,SAVEFILEBUTTON1,,, 2,NOTEPADMAIN,EDIT,FIND,TABCONTROL1,T ABFIND,

3

File new Mouse Up 10/3/2007 11:51:23 AM New txtbody Menu Click 10/3/2007 11:51:23 AM New txtbody Mouse Down 10/3/2007 11:51:23 AM New txtbody Mouse Up 10/3/2007 11:51:23 AM TxtBody Mouse Move 10/3/2007 11:51:23 AM TxtBody Key Down 10/3/2007 11:51:23 AM TxtBody Key Up 10/3/2007 11:51:23 AM (Test) is written in the document 10/3/2007 11:51:23 AM (Test) is written in the document 10/3/2007 11:51:24 AM SaveFilebutton1 Mouse Move 10/3/2007 11:51:24 AM SaveFilebutton1 Mouse Button Down 10/3/2007 11:51:24 AM SaveFilebutton1 Mouse Button Up 10/3/2007 11:51:24 AM File SAVE AS Menu Click 10/3/2007 11:51:24 AM File SAVE AS Mouse Down 10/3/2007 11:51:24 AM File SAVE AS Mouse Up 10/3/2007 11:51:24 AM SaveFilebutton1 Mouse Move 10/3/2007 11:51:24 AM SaveFilebutton1 Mouse Button Down 10/3/2007 11:51:24 AM SaveFilebutton1 Mouse Button Up 10/3/2007 11:51:24 AM

3,NOTEPADMAIN,VIEW,STATUS BAR,,, 4,NOTEPADMAIN,FIND,TABCONTROL1,TABR EPLACE,REPLACETABTXTREPLACE, 5,NOTEPADMAIN,FIND,TABCONTROL1,TABR EPLACE,REPLACETABBTNREPLACE, 6,NOTEPADMAIN,FIND,TABCONTROL1,TABR EPLACE,REPLACETABLABEL2, 7,NOTEPADMAIN,EDIT,CUT,,, 8,NOTEPADMAIN,EDIT,FIND,TABCONTROL1,T ABREPLACE, 9,NOTEPADMAIN,OPEN,OPENFILECOMBOBO X4,,, Figure2: A sample from a test case generation file. To evaluate test generation efficiency in the generated test cases, the total number of arcs visited in the generated test cases is calculated to the total number of arcs or edges in the AUT. File-Save, EditCopy, Format-Font are examples of arcs or edges. An algorithm is developed to count the total number of edges in the AUT by using the parent info for each control. (This is a simple approach of calculating test efficiency. More rigorous efficiency measuring techniques are planned in future work). Of those tested applications, about 95% of the application paths can be tested using 200-300 test cases.

Figure 4 Log file output of a sample test case. Since the test execution process is subjected to several environment factors, the verification process is divided into three levels. 1. In the first level the tool checks that every control in the test suite is successfully executed. This step is also divided into two parts. The first part is checking that all controls executed are existed in the test suite. This is to verify that the execution process itself does not cause any extra errors. The second part that ensures all controls in the test suites are executed tests the execution and its results. In the implementation of this level, some controls from the test scenarios were not executed. This is maybe the case of some dynamic execution or time synchronization issues where some controls are not available the time they are expected. 2. In the second level the tool checks that the number of controls matches between the two suites. 3. In the third level, the tool checks that the events are in the same sequence in both suites. The verification process is automated by comparing the test cases’ file with the log file. Time stamp is important to verify the correct sequence of events. The controls in the test case suites are written to a sorted list and the execution sequence is also written to another sorted lists. To verify the results of the test execution, the two lists are compared with each other. Upon testing several applications, a small percent of controls generated in the test cases and not executed. Timing synchronization causes some controls to be unavailable or invisible within their execution time. Regular users “wait” for a new menu to be opened. This time varies depending on the application, the computer and the environment. Time synchronization and some other dynamic issues are part of the future research goals. One approach for the GUI test oracle is to have event templates. For each test scenario listed, expected results are written in a way that can be automatically

For test verification, a log file is created to track the events that are executed in the tool during the execution process. In a simple example, figure 3 shown below, two test cases are generated that write a text in Notepad and save it to a file. Those test cases are generated using the tool. 1,NOTEPADMAIN,FILE,NEW,TXTBODY,, 2,NOTEPADMAIN,FILE,SAVE AS,SAVEFILEBUTTON1,,, Figure 3: a sample of generated test cases. The first test case opens a new document and writes to it. As part of the default input values, we set for each control a default value to be inserted by the tool through execution. A textbox writes the word “test” or the number “0” whenever it is successfully called. A menu item is clicked, using its parent, whenever it is successfully called. For example, if Save is called as a control, File-Save as an event is triggered. We should have tables for valid and invalid inputs for each GUI control. The second test case opens the save file dialogue and clicks the OK or accept button (Savefilebutton1), to save the document. Here is the corresponding log file output for the above test cases. Control Event

Date Time

File new Menu Click 10/3/2007 11:51:23 AM File new Mouse Down 10/3/2007 11:51:23 AM

4

controls in the categories with a proportion to their weight or occurrence. This algorithm is considered as a hybrid technique that uses some of the capture/play-back processes. In a capture/ play-back tool, the same user session that is captured in the manual testing is executed. In this approach the controls’ weights are extracted from the manual testing to guide test case generation and execution.

verified. This requires some level of abstraction where similar events are abstract into one event (like the saving process). This proposal does not mean exhaustively testing all probable events. By selecting critical events for automation and abstracting similar events we will get a valuable state reduction that makes the number of GUI states more practical in terms of testing. Some algorithms are developed to calculate critical paths automatically. Those algorithms are: 1. Critical path using node weight. In this approach, each control has a metric weight that represents the count of all its children. For example if the children of File are: Save, SaveAs, Close, Exit, Open, Page Setup, and Print, then its metric weight will be 7 (another alternative is to sum all the decedents). For each generated scenario, the weight of that scenario is calculated as the sum of all the weights of its individual selected controls. 2. Critical path level reduction. In this approach, a test scenario is randomly selected. The selected scenario includes controls from the different levels. Starting from the lowest level control, we exclude from selection all those controls that share the same parent with the selected control. This reduction shouldn’t exceed half of the tree depth. For example if the depth of the tree is four levels, we should exclude controls from levels three and four. We assume that each test scenario should start from the main entry point and that three controls is the least required for a test scenario (like Notepad – file – exit). We select five consecutive test scenarios using the same reduction process described above. 3. The previously described algorithms for critical paths’ selection depend on statistics pulled from the implementation model. As an alternative, we can analyze several user captured sessions (e.g. from testers or users in beta testing) to automatically weight the GUI controls. User session data is the set of user actions performed on the AUT from entering the application until leaving it. The session logs all the controls that are executed in the different scenarios. A simple count or percentage is given to each control depending on how many times it is listed in those scenarios. The test scenarios should include all primary and major use cases for the tested application. The controls’ weights (calculated from user sessions) can drive the test case generation and execution. Theoretically all controls should get the same weight in the generated test suite. However, in real scenarios this may not be true. We may cluster the controls according to their usage from user sessions into three levels; heavily used, medium and low. Depending on the availability of the resources to testing, we may choose one or two categories and generate test cases that cover those

4. Conclusion and future work Developing a user interface test automation tool faces several challenges. Some of those challenges are: Serializing the GUI widgets, test results’ verification, time synchronization issues, handling dialog boxes, testing data dependent systems, and building error logging and recovery procedures. Some techniques, in test case generation, execution and verification, are explained in principles in this article. Test case generation from user sessions is explored that represent real usage of the application and focuses on the application areas where they can be heavily used by users. A logging procedure is implemented to compare the executed suite with the generated one. A formal verification procedure is presented that compare and verifies the output of the execution process with its input. Another track of verification is suggested. This track requires building templates for events. For each event pre conditions, post conditions and expected results are included. More elaborations and verifications are required to prove the effectiveness of the suggested approaches. Automation of the first few test cases is expensive; beyond that they become much cheaper to develop and execute. Test automation can only be successful when we keep in mind that testing in general, and particularly automated testing, is easily made obsolete by some changes in the application and environment. In GUI, it is difficult to reach a high level of test adequacy in generating test cases that cover all possible combinations. GUI state reduction is needed to make testing all or most possible states feasible. In this approach, there is state reduction from selecting specific properties to parse. Those properties are more relevant and critical than the rest for the testing process. Total properties of less than 10 are selected. The idea of test automation is not to automate everything; we automate to save time and effort. The other issue that causes state reduction is considering the hierarchy. In flat state diagrams, we assume that any state is reached from another state once certain preconditions are achieved. In GUI terms it means that any control is accessible from any

5

evolving software. IEEE Transactions on Software Engineering, 31(10):884-896, 2005. 5. A. M. Memom, I Banerejee, and A. Nagarajan. GUI Ripping: Reverse Engineering Of Graphical User Interfaces For Testing. In Proceedings of the 10th Working Conference on Reverse Engineering ( WCRE’03), 10951350/03. 2003. 6. A. K. Ames and H Jie. Critical Paths for GUI Regression Testing. University of California, Santa Cruz. http://www.cse.ucsc.edu/~sasha/ proj/ gui_testing.pdf. 2004. 7. A. M. Memon. Developing Testing Techniques for Event-driven Pervasive Computing Applications. Department of Computer Science. University of Maryland. 8. A. M. Memon. GUI testing: Pitfall and Process. Software Technologies. August 2002. Pages 87-88. 9. A. M Memon. A Comprehensive Framework For Testing Graphical User Interfaces. Ph.D. thesis, Department of Computer Science, University of Pittsburgh, July 2001. 10. A. M. Memon, and M. Soffa. Regression Testing of GUIs. In Proceedings of ESEC/FSE’03. Sep. 2003. Saket Godase. An introduction to software automation. http://www.qthreads.com/articles/testing/an_introduction_t o_software_test_automation.html. 2005. 12. Brian Marick. When should a test be automated. http://www.testing.com/writings/automate.pdf. (Presented at Quality Week '98.). 13. L. White and H. Almezen. Generating test cases from GUI responsibilities using complete interaction sequences. In Proceedings of the International Symposium on Software Reliability Engineering, pages 110-121, Oct 2000. 14. A. Mitchell and J. Power. An approach to quantifying the run-time behavior of Java GUI applications.

other one. This is not true in GUI states, as usually controls are only accessible through their parent controls. We can also get GUI states reduction by abstracting the processes. For example, the saving process has several intermediate processes (file-save-as, file name, type… ok), but what is important is the end state where there are certain results if the process is successful (e.g. a file with certain content is saved to a certain location) and other results if it is not. We don’t automate testing everything. We automate testing those frequently repeated tasks. We don’t automate those tests that require user validation or those tests that require special dynamic or time synchronization consideration (such as connecting to a database or verifying the overall acceptance of a GUI).

5. References 1. Yury Makedonov. Managers guide to GUI test automation. Software test and performance conference. 2005. http://www.softwaretestconsulting.com/ Presentations_slides/Manager_sGuide_GUI_TestAutomati on_11wh.pdf. 2. George Nistorica. Automated GUI testing. http://www.perl.com/pub/a/2005/08/11/win32guitest.html. 2005. 3. Q. Xie. Developing Cost-Effective Model-Based Techniques for GUI Testing. In Proceedings of The International Conference of Software Engineering 2006 (ICSE’06). 2006. 4. A. M. Memon and Q. Xie . Studying the fault detection effectiveness of GUI test cases for rapidly

6