Practical Guide To Auditing The Software Development Process

Practical Guide to

Auditing the Software Development Process November 22, 2008

Bruno Collet, MBA Independent Project Manager & Auditor [email protected] www.brunocollet.com Owner, Synapsys Canada www.synapsys.ca

Check the free Agile audit tool Agile Benchmark at www.agilebenchmark.com

TABLE OF CONTENTS Introduction ___________________________________________________________ 3 Definition__________________________________________________________________ 3 Top three principles of auditing _______________________________________________ 3 The auditor ________________________________________________________________ 3 SD audit and transition management ___________________________________________ 4 Phases ____________________________________________________________________ 4

Initiating the audit ______________________________________________________ 4 Defining the mission _________________________________________________________ 4 Identifying the stakeholders __________________________________________________ 5 Planning the audit __________________________________________________________ 5 Kicking-off the audit ________________________________________________________ 5

Analyzing the organization _______________________________________________ 6 General information about the organization _____________________________________ 6 Role of SD in the organization_________________________________________________ 6 Technologies, platforms, tools, and paradigms ___________________________________ 6

Evaluating SD process ___________________________________________________ 7 Sources of information_______________________________________________________ 7 Client perspective ___________________________________________________________ 8 Team perspective ___________________________________________________________ 8 Management perspective ____________________________________________________ 10

Evaluating organizational structure _______________________________________ 10 Evaluating tools and technologies_________________________________________ 11 Testing evaluation results _______________________________________________ 12 Reporting audit results__________________________________________________ 12 Structure of the audit report and presentation __________________________________ 12 How to choose and assess alternatives _________________________________________ 13

Following up _________________________________________________________ 14 Plan of engagements________________________________________________________ 14 Champions _______________________________________________________________ 15 Coaching _________________________________________________________________ 15

Conclusion ___________________________________________________________ 15 References ___________________________________________________________ 16

Practical Guide to Auditing the Software Development Process © Bruno Collet, MBA, Independent Management Consultant

2

Introduction The purpose of this document is to summarize the process for auditing software development (SD) practices. Although business and academic sources extensively cover both auditing and SD processes, there is surprisingly little information about audit applied to SD. This document briefly describes the SD audit process based on publicly available information as well as personal experience in SD auditing and in IT project management.

Definition The purpose of an SD audit is to assess SD practices in the target organization in order to recommend improvements to help the organization achieve its business objectives. Although the scope of an SD audit can vary, it typically encompasses SD processes, organizational structure, and tools and technologies. In essence, the goal of the SD audit is to optimize SD's effectiveness and efficiency. Note that an SD audit is not an IT audit. Although there is some overlap, the IT audit deals primarily with IT operations.

Top three principles of auditing We should keep in mind the top three principles of auditing: 1. Unaudited information lacks sufficient credibility to form a reliable base for decision making. 2. Auditing requires more than common sense. It requires a systematic approach in gathering evidence, and specialized knowledge (SD management in our case). 3. Auditors must be independent from the organization they audit. Their work must be carried out objectively and impartially.

The auditor The SD auditor should have the following characteristics: − − − −

Excellent communication and interpersonal abilities Management consulting skills to formulate recommendation based on clear evidence and flawless analysis techniques Experience in SD management, as project manager for example Experience in IT management, as development or IT manager for example

Practical Guide to Auditing the Software Development Process © Bruno Collet, MBA, Independent Management Consultant

3

SD audit and transition management Although it is sometimes independent, the SD audit is often closely related to transition, change, and crisis management. The sequence is as follows:

SD audit and transition management

1. Crisis: the organization faces SD problems that threaten its business objectives. 2. Audit: the organization performs an SD audit to identify the causes of the problems and to find solutions. 3. Transition and change: the organization implements the recommended solution. The transition can range from non-disruptive changes (a series of minor improvements), to specific changes (a small number of significant changes), to full-fledged transition (changes that deeply affect SD processes, structures, and possibly other business processes).

Phases This document describes the main steps of the SD audit process, which are similar to those of the traditional audit process: 1. 2. 3. 4. 5. 6. 7. 8.

Initiating the audit Analyzing the organization Evaluating SD processes Evaluating organizational structure Evaluating tools and technologies Testing evaluation results Reporting audit results Following up

Initiating the audit Defining the mission The auditor and the sponsor have to define the scope of the audit. The SD audit will always involve SD processes, but may or may not involve organizational structure, tools,

Practical Guide to Auditing the Software Development Process © Bruno Collet, MBA, Independent Management Consultant

4

and technologies. The mission definition must also highlight the objectives in terms of success factors. Additionally, the auditor and the sponsor have to agree on the time and resources needed to complete the audit.

Identifying the stakeholders Stakeholders typically include: − − − −

Management Auditors (internal and external) Employees, also referred as "the team", that consists in analysts, developers, testers, project managers, and so on Sponsors

Planning the audit The auditor has to plan the phases and activities of the audit. The audit can be seen as a project in itself. As such, it has to be planned, and has milestones and objectives. Milestones usually correspond to the release of status and intermediate reports to the sponsor.

Kicking-off the audit It is critical to communicate the audit goals and schedule to all stakeholders. Indeed, poor communication will put some stakeholders on edge and will considerably hinder the audit process. It is the auditor's and sponsor's responsibility to communicate properly to ensure commitment from the start. Kicking-off the audit usually takes the form of a short meeting with all stakeholders. Beside mission specifics, the kick-off should clearly emphasize: − −

−

Achieving the goals also means improving everyone's everyday job and creating opportunities for further personal development. Assessment focuses on processes and structure. The audit does not judge individuals' performance. Although this is not entirely true, it has to be communicated this way. Assessment is 360 degrees (see further).

Practical Guide to Auditing the Software Development Process © Bruno Collet, MBA, Independent Management Consultant

5

Analyzing the organization It is a common pitfall to study SD in isolation. The auditor should always keep in mind the role of SD in the organization and, more specifically, how SD contributes to the organization's success.

General information about the organization General organization's information includes: − − − −

Vision and mission Market and competitive advantage Situation, including financials Structure

Role of SD in the organization The most straightforward method for assessing the role of SD is to determine the value it brings to other business processes. − −

Core business processes: supply, production, marketing, sales Supporting business processes: general administration, human resources, technical support

IT companies selling software solutions constitute a special case; for them SD is a core strategic process at the heart of the production core business process. We also have to stress that SD is not just about software development. It is about the implementation of any information system, which can be made in part or in whole of external software applications, components, or services. Software development does not only refer to programming; it refers to the development of software solutions. Actually, programming might (and often should) only be a small part of the organization's software solutions.

Technologies, platforms, tools, and paradigms The auditor has to determine the technologies used in SD. Technologies can be categorized as base, key, pacing, and emerging. For example, an organization could be relying on Java Enterprise as a base technology, and on XML-based Web Services as an emerging technology in the software solutions. Platforms refer to operating systems, servers, and channels, such as MS Windows, the Web, and IBM Websphere server. Tools refer to the basic tools used for SD in the organization, such as MS Project, IBM Clearcase, and IBM Websphere Developer Studio.

Practical Guide to Auditing the Software Development Process © Bruno Collet, MBA, Independent Management Consultant

6

The auditor should avoid getting bogged down in technical detail, though. On the other hand, the auditor has to gain a good understanding of the paradigms that SD relies on in the organization. Example paradigms are software-oriented architecture (SOA), enterprise application integration (EAI), and software-as-a-service (SaaS). The paradigms heavily influence SD processes.

Evaluating SD process The evaluation is usually the phase that takes up the most time and effort. The evaluation is based on a 360 degrees assessment that takes multiple perspectives into account, as illustrated by the figure below.

360 degrees evaluation

Consolidating these four perspectives gives a reliable description of the reality.

Sources of information At the heart of the audit are the different ways the auditor collects information. The auditor mixes formal/informal, explicit/implicit, and verbal/written sources of information, in order to create a reliable representation of the reality that will be the base for a sound recommendation. We summarize the sources as follows: − −

−

Documentation: written documents or web pages provided by the organization. Surveys: questionnaires filled in by each team member independently, without the auditor being present. The survey provides the auditor raw data on which he can base the subsequent interviews. Interviews: face-to-face or phone interviews with individuals. The purpose of the interviews with team members is to dig deeper into problems that have been discovered through the survey to get a qualitative feedback.

Practical Guide to Auditing the Software Development Process © Bruno Collet, MBA, Independent Management Consultant

7

Client perspective The auditor has to assess client satisfaction, and put that in relationship with the SD process. Ideally, he will be allowed to directly contact some clients. Measuring client satisfaction is the best way to assess the success of SD projects. The auditor should simply ask the clients whether projects met the required quality, were delivered on time and on budget. After having assessed client satisfaction, we have to evaluate the client perception of how the team manages the client touch points: −

− − − − −

Initiating the project o Business case and requirements o Project charter, including scope, schedule, and budget Milestones and releases: schedule and validation Changes Communicating project status Closing the project Maintenance: solving defects

Team perspective "The team" refers to all people directly involved in the production of software in the organization. It usually includes analysts, architects, designers, developers, testers, integrators, and project managers. Assessing the SD process from the team's perspective relies on (1) general information about SD and organizational practices, and (2) comparison of SD processes with the best practices. As mentioned earlier, information can be collected through documentation, surveys, and interviews. Sample general questions − − − − − − − −

−

Is the team using a formal SD process, based either on an existing methodology or on internal practices? What tools are used for project management? Are clients/users satisfied with the results? Are managers satisfied with the team's results? How are time and schedules managed? Are projects delivered on time? How is the budget managed? Are project delivered on budget? How is quality managed? Is the client satisfied with the quality? How is communication managed? Does each member understand what he/she has to do? Is the management aware of projects' situations (status, etc.)? Is the client kept up to date? How are risks managed? Is there a response plan?

Practical Guide to Auditing the Software Development Process © Bruno Collet, MBA, Independent Management Consultant

8

− −

How are changes managed? Is there a change management process? How is the team structured? How are roles distributed among team members?

Using best practices methodology as a reference We use best practices because we need a reference to evaluate SD practices against. First we have to choose what we defined as best practices. Usually we rely on an existing methodology that has been proved successful for SD and that makes sense for the type of organization and project we are dealing with. For example, for a bank with 50 people in SD and 1000+ days projects, we might choose IBM Rational Unified Process. On the other hand, for a small IT consulting company with multiple small teams of 3-6 members working on 20-to-200 days projects, we might choose an Agile methodology such as OpenUP. Note that we do not want to compare SD practices with the chosen best practice methodology in details: the methodology is simply used as a starting point to define best practices, in terms of roles, processes, activities, and deliverables. Below we will describe sample evaluation guidelines based on IBM Rational Unified Process (IBM RUP) and OpenUP. Refer to the documentation of these methodologies for more details. IBM RUP evaluation guidelines When using IBM RUP as best practices, we can for example assess how SD practices compare to IBM RUP's workflows. During the interview, the auditor can examine how the team handles activities, artifacts, and roles associated with these workflows. The IBM RUP workflows are: −

−

Engineering workflows: o Business model o Requirements o Analysis and design o Implementation o Test o Deployment Supporting workflows: o Project management o Configuration and change management o Environment

OpenUP evaluation guidelines When using OpenUP as best practices, we can for example assess how SD practices compare to OpenUP practices. During the interview, the auditor can examine how the team handles work products, tasks, guidance, and roles associated with these practices. Alternatively, the auditor can base his assessment on the more traditional disciplines of OpenUP. Practical Guide to Auditing the Software Development Process © Bruno Collet, MBA, Independent Management Consultant

9

The OpenUP practices are: −

−

Management practices: o Iterative development o Risk-value lifecycle o Two-level project planning o Whole team o Team change management Technical practices: o Concurrent testing o Continuous integration o Evolutionary architecture o Evolutionary design o Shared vision o Test driven development o Use case driven development

Management perspective The auditor has to establish management responsibility in choosing SD processes, tools and technologies, and in structuring the team. Additionally, in the same fashion we did it for other perspectives, we have to look at activities in which managers are involved, such as: − − − −

Project initiating (effort estimation, resource allocation) Project closing Project reporting Billing

We can also collect information about client satisfaction, team structure, and time/budget/quality, in the same way we did for the team perspective.

Evaluating organizational structure Organizational structure is a topic in itself. Therefore this section is an extreme simplification of how to assess and affect organizational structure for SD. In many occurrences the auditor will notice that the organizational structure is not optimal for the organization's SD activity. It often involves one or several of the following: − −

Team does not cover skills, or some skills have no backup Power structure significantly differs from official organizational structure

Practical Guide to Auditing the Software Development Process © Bruno Collet, MBA, Independent Management Consultant

10

− − − −

Some skills are redundant Interpersonal problems Behavior problems Conflicts of interest

These issues tend to affect employee satisfaction, which leads to motivation and productivity problems. Solutions might require adapting the structure, such as: − − − − − −

Transferring Changing roles Promoting Hiring Laying off Training

The question of outsourcing also often surfaces along an SD audit mission. Outsourcing is a topic that we will not delve into.

Evaluating tools and technologies Evaluating tools and technologies strongly depends on the specific context. Therefore, we can only mention a few guidelines. − − −

How are tools and technologies chosen? Are the current technologies and tools efficient? Are current tools used and leveraged appropriately?

In particular, we want to avoid tools and technologies being chosen based on the fashion of the day (how "cool" they are), based on some team members interest to develop personal skills, or based on some managers relationships with suppliers. Other red flags include dependence on a large number of technologies and tools, rogue tools, and compatibility/interoperability issues. In most cases, the auditor does not possess the skills required to assess technologies and tools. He has to delegate investigation to internal auditors (who cannot be stakeholders) or external specialists. However, a simple checklist will allow examining if and how the organization addressed basic tools requirements: − − − −

Project management tools Bug tracking tools Customer service or CRM tools Automated testing tools

Practical Guide to Auditing the Software Development Process © Bruno Collet, MBA, Independent Management Consultant

11

−

Source repository or version control tools

Testing evaluation results In the evaluation phase, the auditor lets others express themselves. Consequently, most information collected so far comes from documents and people's perception. Although the 360 degrees evaluation minimizes the risks for serious misperception, the evaluation is not finished before the auditor can directly assess some elements. Direct assessment consists in collecting facts first-hand, without intermediaries such as people or documents. In essence, testing evaluation results consists in gathering evidence to support the evaluation, with focus on: − −

Gaps: elements for which no significant evidence has been found. Inconsistencies: elements for which contradictory evidence has been found.

Reporting audit results Reporting audit results consists in: 1. Submitting an audit report to management (a document), and presenting the recommendation to management (a PowerPoint presentation for example) 2. Helping the management in making decisions based on the recommendation. The management might also want to adjust the auditor's recommendation. 3. Presenting audit results, recommendation, and implementation plan to all stakeholders (a PowerPoint presentation for example).

Structure of the audit report and presentation The audit report and the presentation to management should have the following structure: − − −

−

Audit goals and link with business objectives (1 slide) Audit process (1 slide) Analysis of the organization: o Organization and market summary (1 slide) o Role of SD in the organization (1 slide) Evaluation results: o Client perspective (1 slide)

Practical Guide to Auditing the Software Development Process © Bruno Collet, MBA, Independent Management Consultant

12

− − − −

o Team perspective (1-2 slides) o Management perspective (1 slide) o Consolidated results (1 slide) Solutions ("alternatives"): See next section. Recommendation (1 slide): summarizes why the recommended alternative is the best one. Plan of action (1-2 slides): brief plan of action, with high-level timeline and tasks. Risk assessment (1 slide): indicate probability of occurrence, impact, and contingency plan for each risk that can hinder the implementation of the solution.

The presentation should be around 20 slides long and last no more than 30 minutes.

How to choose and assess alternatives Selection criteria The auditor chooses 5-to-8 criteria the alternatives will be measured against. Some of these criteria tend to appear in all audits, while some are project-specific. Among the criteria that appear in all audits and recommendations, we find: − − − − −

Return on investment (short term and long term) Fit with resources Fit with skills and culture Fit with strategic vision Risks

Financials No recommendation is complete without strong financial evidence. The auditor has to estimate the costs and benefits of each alternative in dollar value. There are many methods for analyzing ROI. One such method is to calculate the total value of opportunity (TVO). Compared to traditional ROI-based methods, TVO has the advantage to take opportunity costs into account and to rely on net present value (NPV) of cash flows instead of accrual-based value. Although cost and revenue items can vary according to the audit, some items are remarkably invariable: − − − −

Cost savings due to process improvement resulting in increased efficiency Additional revenue due to improved credibility and confidence Cost and forfeited revenue due to learning during implementation of solution Cost of auditing, training, and other consulting fees

Choice of alternatives The auditor presents 3 to 5 alternative solutions. One should be status quo and the others should all be valid solutions that significantly vary when measured to selection criteria. It is important to mention the status quo alternative because we want to measure Practical Guide to Auditing the Software Development Process © Bruno Collet, MBA, Independent Management Consultant

13

alternatives against what we already know, which is the current situation. Note that all alternatives have to be realistic in terms of resources. For each alternative, the auditor has to indicate pros and cons. Recommended solution The auditor summarizes the evaluation of the alternatives against selection criteria. The summary usually takes the form of a decision table with criteria as lines and alternative as columns. Each cell quantifies how well the alternative measures against the criteria. Measures have to be simple, such as a 0-to-5 rating for example. Criteria can also be weighted individually. 1. Status Quo

2. Do This

3. Do That

ROI – 6 months

☺





ROI – 3 years



☺



Fit with skills/culture

☺





Fit with strategy



☺



Low risks



☺



Recommendation Example of simple decision table

Don't be fooled by the simplicity of this decision table. The goal of the audit is precisely to summarize a complex situation in a form that facilitates decision making. Each of these simplistic scores is the result of a thorough analysis. In other words, the auditor must be ready to explain each of these scores in detail. The auditor should not compromise on the results of his analysis. He should have definitive arguments to support his recommendation and should never endorse a solution that he doesn't believe in.

Following up There is little chance that the audit recommendation will be carried on if the auditor does not follow up on the audit. Three things can support the organization in implementing the recommendation and reap the full benefits of the SD audit.

Plan of engagements The auditor has to agree on a plan of engagements with the sponsor and management of the organization. The plan of engagements determines subsequent reviews, for example quarterly reviews during the next year. It focuses on controlling the risks that were identified during the audit. In essence, an SD audit is an iterative project: it is Practical Guide to Auditing the Software Development Process © Bruno Collet, MBA, Independent Management Consultant

14

unlikely that heavy changes can be implemented quickly and completely right after the audit. Therefore, it is imperative to plan these changes along a longer period of time.

Champions The auditor has to raise champions to carry on his work in the organization. Indeed, neither the auditor's distant follow-up nor management's commitment is sufficient to keep up everyday's interest in the changes that have to be implemented after the auditor leaves. Champions are employees in the organization who are convinced that implementing the recommendation is a solution to the problem, and who have power to make the recommendation a reality. This power often comes from influence. Therefore, champions are ideally people who work in or with the team every day. Champions are the auditor's best allies to overcome resistance to change.

Coaching Good auditors will not be afraid of coaching the implementation of their recommendation. They will switch from auditor to coach and work with the stakeholders to implement the plan of actions. Coaching tends to diminish in intensity as the changes are implemented. Alternatively, the auditor can recommend another consultant to do the coaching. Either way, coaching significantly improves the speed and chances of success.

Conclusion SD audit is not a trivial matter. It requires blending several disciplines in a cohesive whole to establish a foundation for change. It borrows from software development, project management, organizational behavior, management, and even managerial finance. Every organization facing potential or actual SD-related problems should seriously consider an SD audit. The return on investment of an SD audit is not easy to evaluate, but guesstimate is usually enough to demonstrate excellent return. Imagine improving efficiency by 10% on all IT projects, and improving client satisfaction. Even small organizations will notice that such relatively humble results will yield several thousands of dollars in savings, as well as increased revenues. ROI is almost certainly positive and payback period is usually in the near future. However, an SD audit can fail. The main causes for failure are (1) a mission scope that does not allow addressing the real problems, and (2) the inability to follow up on the audit to implement the solution. For example, SD problems caused by dysfunctional team structure will not be solved if processes are improved while the team structure remains unchanged. Without doubt, many organizations would benefit from SD audits. The impact of SD on the bottom line is often underestimated. Moreover, many managers still believe that

Practical Guide to Auditing the Software Development Process © Bruno Collet, MBA, Independent Management Consultant

15

general audits based on accounting practices cover the whole business, and fail to understand that these audits do not address SD issues as they should.

References Information technology audit, Wikipedia http://en.wikipedia.org/wiki/Information_technology_audit Software development process, Wikipedia http://en.wikipedia.org/wiki/Software_development_life_cycle Audit, Wikipedia http://en.wikipedia.org/wiki/Audit Business process, Wikipedia http://en.wikipedia.org/wiki/Business_process OpenUP, Eclipse http://epf.eclipse.org/wikis/openup/ IBM Rational Unified Process, Wikipedia http://en.wikipedia.org/wiki/IBM_Rational_Unified_Process Improve Your Audit Interviews, ASQ Audit Quality Division, J. P. Russell (2006), Regulatory Compliance Newsletter http://www.gmplabeling.com/news_letters/spring_news_05_06.pdf Interviewing as an Auditing Tool, Linda M. Leinicke, Joyce A. Ostrosky, W. Max Rexroad, James R. Baker, and Sarah Beckman, (2005), CPA Journal http://www.nysscpa.org/cpajournal/2005/205/essentials/p34.htm How to Write a Survey or Questionnaire, eHow http://www.ehow.com/how_16596_write-survey-questionnaire.html Coaching, Wikipedia http://en.wikipedia.org/wiki/Coaching Putting a Price Tag on IT Projects, Bruno Collet (2008), CIO.com http://advice.cio.com/bruno_collet/putting_a_price_tag_on_it_projects The Total Value of Opportunity (TVO) of an IT Project, Bruno Collet (2008), CIO.com http://advice.cio.com/bruno_collet/the_total_value_of_opportunity_tvo_of_an_it_project Net present value, Wikipedia http://en.wikipedia.org/wiki/Net_present_value Related blog posts written by Bruno Collet:

Practical Guide to Auditing the Software Development Process © Bruno Collet, MBA, Independent Management Consultant

16

Effectiveness and efficiency, http://www.brunocollet.com/blog/index.php?blog=5&title=effectiveness_and_effici ency&more=1&c=1&tb=1&pb=1 How to avoid making unrealistic recommendations, http://www.brunocollet.com/blog/index.php?blog=5&title=how_to_avoid_making_ unrealistic_recommen&more=1&c=1&tb=1&pb=1 The mission behind the mission, http://www.brunocollet.com/blog/index.php?blog=5&title=the_mission_behind_th e_mission&more=1&c=1&tb=1&pb=1 Guidelines to define a consulting mandate: http://www.brunocollet.com/blog/index.php?blog=5&title=guidelines_to_define_a _consulting_mandat&more=1&c=1&tb=1&pb=1

Practical Guide to Auditing the Software Development Process © Bruno Collet, MBA, Independent Management Consultant

17