The Benefits Of Outsourced Testing

The Benefits of Outsourced Testing Why choose to Outsource Testing? For the majority of organisations, the testing effort is often cyclical, with high and low peaks of activity. However, because of this it is not efficient or practical to maintain a fully resourced test team all year round, leaving excess capacity during the low peaks. One solution is to use a specialist outsourcing testing company to provide the help you need, either in the form of attaining in-house assistance (consultants onsite), or, as discussed in this article, outsourcing this function completely. Other reasons for outsourcing their testing includes: to cut costs, to speed testing, to improve their testing, to save office space, acquire test environment facilities, and release in-house staff to take on development activities.

The Benefits of Outsourcing The use of outsourced test teams has increased dramatically in recent years with the recognition that companies can benefit from a wealth of testing expertise that can add to, and complement, inhouse knowledge. The main advantage is that instead of maintaining a fully resourced test team all year round, when test activity peaks core permanent employees can either be supplemented by outsourced testing personnel, or the excess work is outsourced completely. Companies can now contact service suppliers to avail of end-to-end software quality services, for use at any point during development, to shorten time-to-market cycles without taxing internal resources. Additionally, they can benefit from contracting in testers who will be specialists in the field. Utilising an outside test support services firm can provide the ability to utilise resources in a "justin-time" fashion. Ideally, the outsource vendor can provide all of the services that may be required for the system testing. The energy expended in building the relationship can help the client achieve a level of confidence in the quality of the product, which would historically require a larger internal team. The ability to then call on those resources when they are needed, and then have the ability to give them back when they are no longer needed, provides a significant cost saving. In addition, it allows everybody focus on what they do best, thereby maximising everyone's resources. There are a number of additional benefits that can be achieved, including -



These specialists will seamlessly increase the value of your company and allows for more efficiency because both partners in this outsource relationship will be focused on what they do best - a software developer needs to focus on making great software.



The use of independent test teams can validate that the existing testing and procedures are up to the mark.



As a complementary service, it can improve web-enabled software quality, accelerate development cycles, and reduce cost of building software.



Experienced & skilled testers can be acquired for short periods of time.



The alliance enables the organisation to perform a greater range of testing services than is currently possible on it’s own.



This solution dramatically improves software reliability while accelerating time-to-market and reducing the cost of development.



Compatibility lab testing can provide for testing your application against various combinations of Hardware, Operating Systems & Internet Browsers – particularly important for eBusinesses, but one that might not be achievable by the company alone. [For example, they may have a server farm that you cannot afford or justify for the testing, no matter how important].



Return on Investment - learn how to find costly and embarrassing problems before your customers find them.

The First Steps to Successful Outsourcing I would recommend that you consider the following points when deciding whether to outsource a project or application for testing. 1. The nature of the project Firstly, consider the nature of the project or application that you are considering outsourcing. If the project is an enhancement version of a stable product, with existing test scripts that could be used by the outsourcing partner, it would be ideal to outsource. However, if it is the first release of a totally new application, it may not be ideal to outsource the testing. For example, if there are major problems with the application, testing is not going to solve them, and if your testing partner encounters a large number of serious errors the test coverage will probably be lessened. Lastly, carefully consider what you expect to achieve by outsourcing. In all likelihood, outsourcing will never (and probably should never) completely do away with the necessity to have in-house expertise, so you should consider the loss of knowledge and hands-on experience that your test team will lose. 2. What testing do you want them to do? It is also important to consider what types of testing that you actually require to be performed. For example, if stress testing is important, an outsourced testing firm may be more experienced and suited towards performing this. Or, you may wish to perform a certain aspect of the testing that is very complex and that you believe you would not sufficiently benefit from by outsourcing. 3. The Practicalities Who is going to be fixing the bugs? If your company is going to, the process and procedures by which the error reports are going to be exchanged is critical. If your error management system cannot be remotely updated [i.e. by the outsourced partner reporting the bugs directly] then I would suggest either not outsourcing at all, or else implementing an error management tool which can cater for remote logging, and can be under your direct control. It is important that you retain control (or at least possession) of the bug database for future reference. Secondly, how are you going to hand over new builds of the s/w? If you have to burn CD’s, unless there is a quick way of transferring the new builds over to the outsourced testing partner, valuable testing time may be lost.

Choosing an Outsourced Testing Partner When choosing to utilise outsourced testing services, it is important to check the following: •

Their past experience – specifically in testing the same type of project

•

Their current (not historical) client list

•

Bios of their key personnel that are going to be working on your project (and not just Bio’s of their best people).

•

Detailed evidence of the existence of testing processes and procedures, particularly for preparing test cases/scripts; executing test cases/scripts; and (most importantly) their entire review process.

Risk Planning for Outsourced Testing

What’s the big deal - after all, they will actually do the work, won’t they? I mean, what could go wrong? As the extent of the outsourcing grows, however, so too will the challenge of managing relationships with them. But if companies fail to do this effectively, projects could be jeopardised. The danger is that weak outsourcing partners can stall entire projects. So, how do you provide for this situation? Apart from the contractual elements to formalise the relationship, you must naturally consider how you can actually verify their progress. Additionally, you need also to be sure that the work is being performed to a high level of quality. In my opinion, the only way to ensure this is to hold regular reviews and assessments with the partner. (See next section on reviewing). If possible, send a representative (one of your testers) to be part of their team. In addition to providing you a direct insight into the actual progress and quality, the representative can assist the outsourcer either by having previous experience of that application, or at the very least, an informal channel of communication.

Managing Outsourced Testing Projects One of the key ingredients for success when outsourcing is the relationship. There must be trust between both parties. This trust is necessary in order to communicate the value points and the methods of communication that are critical for supporting the flexibility that using a test support services firm requires.

You should take a collaborative approach, and work together – after all, you are both working towards the same goal – releasing a quality product. I would recommend the following: Firstly, define the channels of communication. Appoint a person from your company, to act as the first level of contact between your company and the outsourced partner, who will deal directly with an agreed specified person on their side (ideally the test manager for the project). These two people are the first-level team for resolving problems as they arise, and additionally your representative also provides and archives all information passed to the outsourcing partner. Secondly, agree escalation procedures, and specify to whom the problem is to be escalated to - on each side. Set out milestones – but beware when setting milestones, that YOU are the ones who state whether the outsourced testing company has reached the milestone or not – for example, if one milestone is the production of the test plan, if you are not happy with the test plan then you must be in a position to state that they have NOT reached the milestone. Otherwise you could end up in a position whereby they gave you a test plan that you considered inadequate, yet they are looking for payment for reaching the milestone. To cater for this, and other concerns, it is vitally important to specify formal review points – stages at which progress can be measured, and where issues can be highlighted. Formal Review Points: Test Plan Insist that you must sign off on the test plan before test preparation can commence. Review Hold a round table review (by phone or videoconferencing as available) and ensure the test plan clearly sets out the approach to be taken, identifying what levels and types of testing are to be applied and the methods, techniques, tools and resources to be used, as well as the review points, schedule, test coverage required, and signoff criteria. [See here for more details about test plan contents].

If you are not satisfied with the test plan, do not signoff. Ensure that one of the criteria for proceeding to the next stage is your signoff. Test Case Ensure that all test cases are reviewed and signed off by your own business Review analysts/product specialists, to ensure that they are of sufficient quality, depth, accuracy and coverage. Insist on modifications to the test cases until you are satisfied with their quality. Remember – the testing will probably only be as good as the test cases – you should assume that anything that is not defined as a test scenario will not be tested, no matter how obvious it may be to you. Execution Readiness Review

An execution readiness review is performed to ensure that the outsourcing company are ready to move into the Test Execution phase. This readiness review checks to ensure that the test entrance criteria, as specified in the Test Plan, have been met.

Execution Progress Reporting

Ensures that regular detailed progress information, including full bug details and test case pass/fail statistics are passed to you throughout the execution phase. During this phase it is important that you can analyse the bugs that are being logged and match the bugs to specific test cases. This can then serve as an independent check of the test execution progress.

In parallel you can perform some testing yourself, to double-check the test cases that are passing. Build Acceptance Tests

If you hand over software that is not of ‘testable’ quality, the outsourced partner is not going to be able make much progress and neither of you are going to benefit at all. Therefore, you should insist that ‘Build Acceptance Tests’ are performed on each build/drop of software - either perform these yourself, or get the outsourcing partner to perform them. These tests are simple checks to ensure that the software is of ‘testable’ quality in that they are going to able to test the various functions and not be stuck by a very obvious showstopper bug. If they cannot proceed with the testing there is no point blaming them.

Bug Turnaround Times

As I am sure you are aware, waiting for bugs to be fixed can be a serious bottleneck for all testing projects, and possibly even more so for an outsourced project. Therefore, you should actively monitor bug turnaround times for bottlenecks. It is probably also more appropriate that somebody from your company be the one escalating/hassling the developers to fix the bugs, rather than an external person trying to.

Exit Report Ensure they prepare an Exit Report, which will summarise the testing and & Final includes the following details: Signoff. • The full bug list •

List of any outstanding bugs

•

List of any uncompleted test cases

•

A copy of all Test spreadsheets

•

A copy of all test results & statistics

Do not sign off on this document until you are satisfied as to the accuracy of the contents, particularly the test coverage metrics. Ask to see log files or any other evidence of test execution having taken place.

Final Deliverables

A final consideration to take into account is that you may have to test this project yourself in the future. However, this could be more difficult because you do not have the benefit of the knowledge and handson experience of actually testing the project in the first place. In order to minimise this risk it is important to specify as a condition of the project that upon completion of the testing, you should be handed back all test documentation. For example, you should include the following as specific deliverables: •

The final test plan document

•

All updated and revised test cases and expected results

•

All test data requirements

•

Testing pre-conditions, including test sequence, test environment setup and initialisation

•

The Bug database

•

All scripts etc.

In this way if you have to test it yourself at least you will not be starting from scratch. Conclusion Once you have picked a good partner, and gone through the first project together, outsourced testing can be an excellent option. But it must be noted that to make a success of it, it requires continuous monitoring and assessment, and careful choice of projects to outsource. There must be trust between both parties. This trust is necessary in order to communicate the value points and the methods of communication that are critical for supporting the flexibility that using a test support services firm requires. You should also take into account that the old adage ‘garbage in, garbage out’ is particularly true in these situations. If you handover a product or information for testing that is not of ‘testable’ quality, it is not going to help either of you. Upon completion of each project you should perform a final acceptance test on all returned software – but be open about this process and give constructive feedback. In addition, you should also assess each other’s performance, particularly with regard to bottlenecks, and be open to accepting criticism of your own company’s performance. Keep in mind that if you do decide to outsource some or all of your testing, a lot of effort will still be required by yourselves in order to support and enable the outsource partner to maximise performance (for example, reviewing test cases etc.) Previously testing often focussed on functional issues and risks as being the main concerns. However, with the advent of eBusiness development, this focus has changed. Non-functional issues such as cross browser compatibility / configuration, usability, performance and security have become more and more important considerations when planning test projects. It is also becoming more prevalent that companies are increasingly making use of outsourced testing to get the expertise quickly without expending all of their capital. Specialisation breeds a continued need for companies to focus on core competencies, and by using strategic outsourcing to fill gaps helps fast-growing companies in the high-tech industry keep focused on their initial goals and keeps resources allocated to the areas that need them most.