True Brew Process Overview

TRUE BREW® Process Overview

QUALCOMM Proprietary

QUALCOMM Incorporated 5775 Morehouse Drive San Diego, CA. 92121-1714 U.S.A. Copyright © 2003 QUALCOMM Incorporated All Rights Reserved Printed in the United States of America. All data and information contained in or disclosed by this document are confidential and proprietary information of QUALCOMM Incorporated, and all rights therein are expressly reserved. By accepting this material, the recipient agrees that this material and the information contained therein are held in confidence and in trust and will not be used, copied, reproduced in whole or in part, nor its contents revealed in any manner to others without the express written permission of QUALCOMM Incorporated. Export of this technology may be controlled by the United States Government. Diversion contrary to U.S. law prohibited. BREW, BREW SDK, and MobileShop are trademarks of QUALCOMM Incorporated. QUALCOMM, Binary Runtime Environment for Wireless, TRUE BREW, and The Grinder are registered trademarks of QUALCOMM Incorporated. All trademarks and registered trademarks referenced herein are the property of their respective owners.

TRUE BREW® Process Overview 80-D4299-1 Rev. D December 17, 2003

Contents 1 Introduction 4 1.1 Abbreviations and acronyms 4

2 What is TRUE BREW Testing? 5 3 TRUE BREW Testing Overview 6 3.1 Documentation needed 6 3.2 Process overview 7 3.2.1 Preparing for application or extension submission 7 3.2.2 Submitting an application or extension 8 3.2.3 Testing overview 8 3.3 Phases of TRUE BREW Testing 10 3.3.1 Entrance Criteria 11 3.3.2 Exploratory 11 3.3.3 Handset/AUT Interaction 12 3.3.4 BREW Features 12 3.3.5 BREW User Interface 12 3.3.6 Adversarial 12 3.3.7 Application Download 13

4 Operator Testing 14 5 Next Steps 15

iii QUALCOMM Proprietary

1 Introduction This document provides information on the TRUE BREW® Testing (TBT) process for a Binary Runtime Environment for Wireless® (BREW) application. The objective of this document is to identify the following: The overall TBT process All supporting information that may be of benefit to the reader Note that this document should be considered as an introduction to the TRUE BREW Test Guide and related documents that provides detailed testing requirements and procedures. The TRUE BREW Test Guide can be found at the following address: http://www.qualcomm.com/brew/developer/resources/ad/documentation.html

1.1

Abbreviations and acronyms

The following is a list of abbreviations and acronyms used in this document. ADS

Application Download Server.

AUT

Application under test.

BREW™

Binary Runtime Environment for Wireless®. BREW is an application development environment that is layered upon public services offered by mobile ASICs.

BREW SDKTM

BREW Software Development Kit. This allows application developers to write and test applets and modules in a familiar Windows desktop environment.

MIF

Module Information File. The MIF Editor generates this binary file, which contains information regarding the list of classes and applets supported by the modules.

TBT

TRUE BREW® Testing.

4 QUALCOMM Proprietary

2 What is TRUE BREW Testing? TBT is a BREW application testing program. Its goal is to ensure that tested applications meet a specified level of stability and compliance with the BREW platform. QUALCOMM works with third party test organizations to implement the TBT program and assure impartiality to each developer. An application must pass TBT before it can be commercialized by operators participating in the TBT program. The goal of TBT is not to evaluate the quality of an application in a subjective sense. For example, TBT does not comment on the marketability or business case for an application. It does not make any judgment on whether an application might be popular with a certain audience, or whether it is “fun” to use. This is left to the discretion of BREW operators who may choose to perform some additional testing once the application has passed TBT.

5 QUALCOMM Proprietary

3 TRUE BREW Testing Overview This document references two entities in the TBT process: The third party test lab implements the TBT requirements, manages application submission, and provides feedback to developers. The test lab grants TRUE BREW status. A QUALCOMM entity represents QUALCOMM on TRUE BREW issues. QUALCOMM makes the TRUE BREW application available for distribution to the operators. This entity also maintains the list of test requirements and collects more general performance measures for the TBT process. This information is used to improve the efficiency and effectiveness of the entire process, as well as providing developers with information to improve their own development and testing efforts.

3.1

Documentation needed

The following documents are used by both the developer and the third party test lab to evaluate and test a BREW application: BREW™ User Interface Design Guidelines: Provides information on user interface design considerations and usability principles especially targeted at developers of applications for BREW devices. TRUE BREW® Test Guide: Specifies the test requirements that a BREW application must meet in order to acquire the TRUE BREW status. TRUE BREW® Test Procedures: Provides a spreadsheet that presents procedures for testing applications against the requirements. The spreadsheet includes columns for noting test results, issues, and comments. TRUE BREW® Application Specification Template: Provides the base for the creation of the Application Specification (App. Spec.) document that is required when submitting an application to TBT. It contains information that will assist the testers in getting to know and evaluating the BREW application.

6 QUALCOMM Proprietary

TRUE BREW Testing Overview

The above documents are available under the BREW Documentation section of the BREW internet web site at: http://www.qualcomm.com/brew/developer/resources/ad/documentation.html Additional information is available under the Application and Extension Testing section of the BREW internet web site at: http://www.qualcomm.com/brew/developer/resources/bd/ext_test.html

3.2

Process overview

This section provides an overview of the TBT process. It also documents the steps necessary to prepare an application for submission to TBT.

3.2.1 Preparing for application or extension submission Before submitting an application for TBT, you must perform all of the following tasks: Download the BREW Software Development Kit (BREW SDK™) and create a BREW application. Sign the application developer legal agreement and complete the questionnaire providing access to the BREW Developer Extranet (Developer Extranet). Download the BREW Testing and Commercialization package from the Developer Extranet. Develop the application. In addition, you should test the application both in the BREW Emulator (Emulator) and on the handset by following the TRUE BREW Test Guide. The process is slightly different for BREW extensions: Download the BREW SDK and create a BREW extension. Sign the extension developer legal agreement and complete the questionnaire providing access to the Developer Extranet. Download the BREW Testing and Commercialization package from the Developer Extranet. Develop a testing application that uses your extension. The third party test lab will use this application to test your extension. Note that the testing application should cover all interfaces provided by your extension. The sample application must also provide testers with a user interface in order to interact with it. The user interface 7 QUALCOMM Proprietary

TRUE BREW Testing Overview

does not have to be sophisticated, but it has to be functional and to conform to the TBT requirements. In addition, you should test the extension using the testing application both in the Emulator and on the handset by following the TRUE BREW Test Guide.

3.2.2 Submitting an application or extension Go to the third party test lab’s web site and submit your application. For example, NSTL operates an approved TBT lab that can be accessed at the NSTL web site (http://www.nstl.com/logoprogram/qualcomm_logoprogram.html). You must ensure that the submitted application is signed using a VeriSign™ Class 3 Certificate. Note that TBT labs may ask you to sign a legal agreement before you can submit your application. You are required to submit the following materials: A version of the application compiled for the target handset’s processor. BREW currently runs on commercial devices using an ARM processor, and therefore an ARM-compiled version is necessary. A Windows-compiled version of the application designed to run on the Emulator. The App. Spec. document. For extensions, you must submit the following material: A version of the extension and testing application compiled for the target handset’s processor. BREW currently runs on commercial devices using an ARM processor, and therefore ARM-compiled versions are necessary. A Windows-compiled version of the extension and testing application to run on the Emulator The App. Spec. document for both the extension and sample application For more detailed submission requirements, refer to the Developer Extranet at: https://brewx.qualcomm.com/developer/operations/submitapp.jsp

3.2.3 Testing overview Figure 3-1 is a high-level illustration of the TBT process. 8 QUALCOMM Proprietary

TRUE BREW Testing Overview

Figure 3-1. TRUE BREW test process Submission is canceled. Status and testing report sent to developer by email. Rework required.

Developer creates BREW application

Developer submits application package to 3rd party lab submission site

No

Pass entrance criteria?

Entrance criteria executed

YES Full TRUE BREW testing executed Status and testing report sent by email to the developer. Rework required

NO

Pass TRUE BREW testing?

YES

Minor Errors Application receives “Passed with Notes” status

Application receives TRUE BREW status

Status and testing report sent by email to the developer

Application enters BREW distribution system. Developer is notified about next step.

The application developer submits applications to be tested through the third party test lab’s application submission web page. After logging into the third party TBT program web page, the application developer fills out a web-based form to proceed with testing. This form includes a TRUE BREW-specific questionnaire that collects application-specific and testingspecific information. Note that developers will incur TBT charges as described on the following web page: http://www.qualcomm.com/brew/developer/resources/bd/ext_test.html#fees When an application is submitted, Entrance Criteria testing begins (see section 3.3.1 Entrance Criteria). If the application fails Entrance Criteria testing, the application submission is canceled, and an email is sent to the developer. A testing report explaining the

9 QUALCOMM Proprietary

TRUE BREW Testing Overview

issues is attached to the email. The developer has to fix these issues and resubmit to the third party test lab. If the application passes Entrance Criteria, the test lab runs the different phases of TBT as described in sections 3.3.2 through 3.3.7. After the testing is completed, the test results are evaluated; in particular, the errors detected during testing. Based upon this analysis, an application will fall into one of the following three categories: TRUE BREW status: The application meets all TBT requirements. No errors are detected. Pass-with-notes status: The application meets all TBT requirements with minor errors that are noted in the testing report. These errors have a low probability of occurrence and/or have minimal impact on the end-user experience. See TRUE BREW Testing: Developer’s Guide to the TRUE BREW “Passed With Notes” Process, which can be found at: http://www.qualcomm.com/brew/developer/resources/ad/documentation.html Fail: The application does not meet the TBT requirements. The application cannot be commercialized. The developer has to fix the reported errors and resubmit to the third party test lab. In addition to errors that cause an application to fail or pass-with-notes, the testing of an application may produce observations, made by the tester, that are not part of the TRUE BREW pass/fail criteria. These observations are referred to as issues. Though not part of the pass/fail criteria, these issues may be documented in the test report. Operators may consider these issues when evaluating an application. It is important to note that the identification and documentation of these issues is not the goal of TBT. They are documented and identified as time permits.

3.3

Phases of TRUE BREW Testing

The TBT process is composed of several phases that can be run in a sequential or parallel fashion. In the process of preparing your application for submission to TBT, it is strongly recommended that you run these phases in order to ensure that your application will pass upon first submission. You should not limit the time allocated to this phase, and you should execute each test case for every screen display, function, and feature provided by your application. Each type of testing is described in Table 3-1 and in the following paragraphs.

10 QUALCOMM Proprietary

TRUE BREW Testing Overview

Table 3-1. Application testing Item

Type of test

Phase

1.

Entrance Criteria

Prior to starting TBT

2.

Exploratory Testing

TBT

3.

Handset/AUT Interaction

TBT

4.

BREW Features

TBT

5.

BREW User Interface

TBT

6.

Adversarial

TBT

7.

Application Download

TBT

3.3.1 Entrance Criteria This portion of the test process is intended to catch easily discoverable errors so that they can be corrected before more thorough testing begins. Entrance Criteria testing is largely performed by inspection of the package submitted by the developer, including the MIF (privileges, notifications, and so forth), directory structure, files in the submission package, and the Application Specification. The last step of the entrance criteria consists of downloading the application from an ADS and starting it up on the BREW device. This cannot be performed by the developer. However, if the developer was able to cable-load the application and start it up on the handset, it is expected that the application will pass this phase of the testing process with no issues. An application that does not meet these requirements receives the “cancelled” status and not the “failed” status. The developer will not be charged a testing fee if this occurs.

3.3.2 Exploratory Exploratory Testing is performed manually by using the application installed on the handset. The tester uses the Application Specification as the basis for determining product functionality and then verifies the existence and operation of the specified functionality. The tester goes through the application screens and features just as the end-user would do.

11 QUALCOMM Proprietary

TRUE BREW Testing Overview

A good introduction to exploratory testing can be found in the document General Functionality and Stability Test Procedure (James Bach, Satisfice, Incorporated) at: http://www.satisfice.com/tools/procedure.pdf.

3.3.3 Handset/AUT Interaction These tests focus on evaluating the behavior of the application when suspended and the ability of the application to resume activity at or near the point of suspension. A combination of incoming calls, SMS messages, and alerts is used during this phase to suspend the application. The application behavior should be independent of the particular type of suspension (for example, SMS suspend versus incoming phone call suspend). The suspend/resume behavior should be evaluated on all screens and all state transitions (for example, setting up data calls, tearing down data calls, transitioning from one screen to another, and so forth).

3.3.4 BREW Features These tests are performed by evaluating specific capabilities or functions of the application (for example, saving data, downloading ring tones, and setting up a network connection). All of these capabilities would have been evaluated to some degree during Exploratory Testing. The purpose of these tests is to focus in on significant BREW features used by the application as advertised in the Application Specification document.

3.3.5 BREW User Interface These tests focus on how the application reacts to Clear and Send key inputs. The Clear key, for instance, has a special meaning in terms of navigation and the user interface: It should take the user back to the previous screen. This phase also focuses on response time of the application as the user navigates through the screens and functions of the application.

3.3.6 Adversarial These tests are performed using both manual and automatic procedures. Manual procedures include placing the handset in a shielded box to ensure that the application can handle loss-of-service events while performing certain functions. They also include using the MaxFileCount tool and the Shaker tool to create low memory and low file handle conditions on the handset and see how the application reacts to those conditions. 12 QUALCOMM Proprietary

TRUE BREW Testing Overview

Automatic procedures consist of running the application in the Emulator and on the handset in conjunction with The Grinder® tool, which sends random key events to the application.

3.3.7 Application Download These tests are performed through interaction between the handset and the ADS. They include disabling and then restoring the application, and having the ADS recall the application. Note that it is not possible for the developer to perform this phase.

13 QUALCOMM Proprietary

Operator Testing

4 Operator Testing After an application has been made available to the operators, an operator may choose to test the application as part of its acceptance program. The operator test is conducted solely at the discretion of the operator. The operator typically interacts directly with the developer at this point to identify and resolve any issues. QUALCOMM encourages operators to disclose all operator-specific application requirements and test cases to developers. If the application fails this last stage of the testing process, the following scenarios may occur: The application fails for a specific operator but is available to other operators. If the application developer is required to modify the application in response to operator testing, the application is required to repeat the TBT application submittal process. If QUALCOMM determines that the reason for failure may be applicable to other operators, the TRUE BREW status may be revoked for the application.

14 QUALCOMM Proprietary

5 Next Steps After your application passes TBT, you will be notified by email and receive instructions about what the next step should be. You may have to complete an export compliance questionnaire and submit it to QUALCOMM before your application can be uploaded into the BREW Distribution System (BDS). Once your application is uploaded into the BDS, you will have to complete the distribution and pricing information found on the Developer Extranet. A complete guide to application distribution can be found under the Documents section of the following web page: https://brewx.qualcomm.com/developer/operations/operations.jsp

15 QUALCOMM Proprietary