Automating Lolcalization Testing

Automation Programs Automating localization testing can be very tricky. Most automation programs include a Scenario Recorder/Playback utility. This will be useless for localization testing for these reasons:

•

Some of the automation programs will record by default mouse clicks on the screen, by position. The position of the items in localized software is different from the US software, because most language strings will be shorter or longer than English strings. So the next sentence (from a common automation program) will only work in the original US software: Play "{Click 278, 34, Left}"

•

The second most common default uses text. Obviously, this will not be useful for localization testing purposes. For instance, if we are checking a Web site, in some instances we would like our automated script to navigate to the home page. Most automation programs would write something like this: WebLinkClick (http://www.bowneglobal.com, "Home") The Spanish string for Home is "Inicio," and the French is "Accueil," so that sentence will never work in the Spanish or French environments, not to mention any Asian language (although it would work for German or Italian, as they also use "Home").

So we will need to forget about the "recording" and any other default and start writing the scripts line by line, making sure what we write can be "understood" in any language. There are several ways to do this: a) Ordinal Position The first approach would be the ordinal position of the objects. Ideally, the items are placed in the same ordinal sort in every language. Let’s check the main menu of any Microsoft application: File-Edit-View-InsertFormat. Any other language will translate those terms, but the ordinal sort will be the same. So, while this command will not work: wmenuselect ("Edit") this one will work in any language: wmenuselect (_ord (2)) Do we have a definitive solution? Unfortunately not. We have a solution for menus and submenus, in a very controlled environment, and maybe some dialogs, but this approach will not work in other dialogs. For example, check these screenshots from Microsoft Outlook (New Contact-Insert-Item):

Let’s say we want to click in Calendar. According to the successful test we wrote for the Edit menu, we would type something like this: WtreeItemClk (_ord(1)) This would open the Calendar in the US machine, but the Inbox folder in the Spanish environment. As the elements are alphabetically sorted, Calendar is ord (1) in the US, but it is ord (4) in Spanish, and the order will be different for French, Russian, and German, etc. b) ID Numbers We need to go one step further: Recording doesn’t work, text references are also a waste of time, and ordinal position only works sometimes. We need to check the ID numbers of the objects. ID numbers always work: Calendar, as a tree object, will have a unique ID for any language, so we can find that ID out with any Window Information utility and we would write: WtreeItemClk (_id(83411528)) This will work. But again, do we have a definitive solution? And again, the answer is no. First of all, not all the objects have a unique ID. Some objects are just containers for other objects, so they are not identified by a unique ID. Also, assigning IDs to objects is not fully automatic during development. And you cannot trust that all the developers took their time to assign unique IDs to the objects they were creating. There is even one more problem: the ID can be just a number, or a whole string containing a lot of information, from the main window to the deepest object information. This is the ID of the Tools-Options OK button in Microsoft Word: =132820.Options{dialog}.#3. In most automation programs, it is not enough to say, "Hey, click the button ID (3), please." You have to pass the whole string as a parameter of your command, such as WbuttonClick (_id (=132820.Options{dialog}.#3.)) The number "#3" will be the same for all languages, but the string "Options{dialog)" will not work in the localized software, as it is English text. We will have to create a whole function to capture during run-time the equivalent localized strings, and then create a string variable for the ID. That number 132820 is the "handle," a unique number identification for the session opened in the software, and it is different each time you launch Word. But we can isolate that number easily, so we shouldn’t worry a lot about it. To summarize, using IDs will require much more programming than using ordinal positions, but sometimes we will need to use them. c) Keyboard Events There is one more thing we can use that will work in the US/English machines and the localized machines: keyboard events. The philosophy is more or less the same as for the ordinal positions: Within the program, clicking Tab, Arrow keys and Enter can lead to the same screens in any environment, when the objects are placed in the same positions. When we are talking about menus, it will be better to use the ord parameter. But when we talk about trees, list boxes, combo boxes, check boxes, or radio buttons, if they are not alphabetically sorted, it will be a good idea to simplify our script by writing things like: Play "{TAB}"

Play "{DOWN}" Play "{ENTER}" This is especially useful if we need to click on a button or any other object we know that has the "focus." For example, you can launch a Wizard, and you know the default button is "Finish." If you type "Play "{ENTER}"," the wizard will start working, in any language. d) Resource Files There is finally a more professional approach to the whole picture: creating resource files for each language. The process for this approach would be as follows: 1. 2. 3.

We first create and record the scripts against the original software. Then we make our scripts "international" by changing all the software strings and other literals to constants, stored in a resource file. Finally, we create the proper resource files for the languages we want to test, by assigning the localized terms to the constants created in the previous steps.

Once we have done this, we can forget about ordinal positions, IDs, and keyboard events—we can simply call the constant in the proper resource file and the script will work fine. But there are some requirements to performing this approach:

• •

•

Using translation software is a must. If you don’t have an application that can create memories and glossaries of your translations, the creation of localized resource files will take too long. Usually, localization testing starts as soon as the translation files are delivered and built, or implemented in some server. If you want to create localized resource files, you will need a longer time frame between the translation and the testing phases. So you will need your management team to plan this from the very beginning of the project. Even if you are using a software like Trados, which will offer you glossaries and translation memories, you need time and a lot of care when building the resource files. In most languages, the same English word can be, and will be, translated in different ways. This can break any script, so you have to make sure you are using the proper translation.

There is no unique solution to automating localization testing. We know we cannot use most of the defaults the automation programs offer. And we know several tricks that can work in particular situations. We even have a professional approach able to automate 70 to 80 percent of the testing by building resource files. But each time we write a script we want to run in a set of languages, we will face different issues, and the solution will have to be adapted to the situation. One good option to automate localization testing is skipping the automation programs on the market and developing your own software using Visual Basic, but this would be the subject of another article.

When Do You Automate? First of all, automation cannot substitute for manual testing when we talk about localization. It can be useful to avoid human errors when trying to reach a particular dialog, and it is very good as help for translators not very skilled in computers. It also saves a lot of time, as you can run setups during the night, so all the environments are ready by the time testers arrive in the morning. But you need the tester or the translator to manually check the final dialog your automated script can reach. Also, if you want to automate, you will need to fulfill some previous requirements: 1. 2.

You need the original software well in advance of the testing start day. Depending on the number of tests you want to automate, you could need two weeks or two months. You need to test a pilot language first. You have created and run the tests against the original language software. The first time you run your tests against a localized version, you will get a lot of

3. 4.

errors. The pilot language will help you identify those errors and create a revised set of scripts for the remaining languages. You need a dedicated team of automation engineers to handle the tester and translator issues with your tests. The size of the team will depend on the number of testers and translators, but one automation engineer for every ten users is usually a good ratio. You need to use a "controlled" environment for the testing. The localized machines must be a "mirror" of the original machine: no extra software, no different drivers or graphical cards. The more similar they are, the more chance of success you’ll have.

We can think of at least four scenarios where the efforts to automate localization testing are worthwhile: 1) When you are running a lot of test cases in three or more languages. Let’s say you have to run 1,000 test cases in six languages. The automation will help you to reach the dialogs you need to test, avoiding human errors during the steps you have to perform to reach those dialogs. You can write a test that performs all the operations needed to find that dialog your tester has to review. You can even check the duplicated hotkeys (most automation programs have a function just for this). But you need the tester to check if the translation is precise; if the concatenation of different strings makes sense; if there are truncations or inconsistencies between translations in that dialog and other parts of the software; and any other particular check you need for your project. There is no way to fully automate those tasks with enough reliability But if you are able to automate those 1,000 tests, so your team doesn’t need to learn about the program functionality, you will gain a lot of time. 2) When you want to create a set of screenshots of localized software for a native translator revision. You can also face a situation where your client wants a complete revision of the translation, made by native people, of the running software. You can think of several solutions such as

• • •

hiring all those native speakers and moving them to your office during the testing creating remote connections, so those translators can connect to your servers and machines remotely to check the software taking screenshots of every possible dialog of the software and sending it to the translators so they can review it

Most probably you will go for the last solution. It is the cheapest one, and you avoid the technical problems and waste of time the second solution can cause. So if you decide to take the screenshots, automation is a very good option. All the automation programs have a utility to take screenshots. If you are able to create tests that run properly in all languages, you will be able to take and save all the screenshots you need, with minimum human support. 3) When you have to perform a lot of setup testing. Launching setup.exe files from different locations (CD, Network, HDD, etc.) is probably the easiest thing you can automate. If you need to do customized setups, most programs can do that just by adding some parameters to the setup.exe command. If your testing involves a lot of setups, then automate it. You will save your team from boring hours in front of a progress bar, and you will save time and money. 4) When you are in charge of localization testing of a huge application, in a lot of languages, and you expect to be testing later versions of the program. In my opinion, this is the only scenario that makes sense for the approach mentioned in the above section (d) Resource Files. The effort of creating and testing the resource files is a project itself, involving programmers, testers, translators, and managers to make it cost effective. If you want your investment in time and money back, you need to apply this approach to a large project in content and time.

Conclusion Automating software testing is always a difficult decision. Many elements must be taken into account to

ensure a successful and profitable automation process. When it comes to localization testing, the task is even more difficult, because to the original elements we must add new ones. To summarize:

• • 1. 2. 3.

Automation is a complement to localization testing. It can save a lot of money and resources, but it cannot fully substitute for manual testing. There are many things we need to carefully consider before we proceed to automate our localization testing project: We will need to program our scripts almost line by line. We need programmers to automate. "Record and Playback" utilities will not work. Do we have the needed people on our team, or can we hire them at a reasonable price? We might need to create resource files linked to translation memories and glossaries, and also linked to the automated scripts. Is the extra programming worth the effort? Will we reuse our work in the future? We will need automation engineers supporting our manual testers. Are we ready to have a stable team of automation engineers? Do we have enough projects to keep them busy and saving money for us?

If you can answer "yes" to the previous questions, you have a good chance of success in your automation project, but you still need to carefully plan what tool or tools you are going to use. For localization purposes, you probably will not need a whole suite of testing tools and trackers—a scripting program with the utility to schedule scripts running might be enough. Finally, if you have planned carefully, reached the proper conclusions, and your automation analysis and development is good, automating localization testing will be worth the effort, and you will save time and money. One thing is true: automation is worth the effort in the long-run. Initially, it may take a period of time to develop scripts for an early build of the software. But execution time of the scripts will be short and modification of scripts for later builds will also take less time. If this software is being tested across a lot of platforms/languages, then the financial and time savings will be even greater.