Project Evaluation

HM

TREASURY

Central Unit on Procurement

No. 43 Project Evaluation This is one of a series of guidances prepared by CUP on the management of works projects. Its use is not mandatory but a statement of good professional practice. 1.

INTRODUCTION

1.1 Major projects are often complex and risky. An enormous number of decisions have to be made and actions taken during a project’s life: often under tight time and budgetary constraints. Project evaluation enables participants to stand back from the detail of the project, review what has occurred and to take a forward look at the remainder of the project. 1.2 Project evaluation is a team effort. It enables individual members to consider aspects of the project from different points of view, identify possible conflicts or poor communication and improve working relationships benefiting the overall project. 1.3 Project sponsors and managers are also responsible for feedback from lessons learned, both good and bad, so where appropriate they can be used to benefit future projects. 1.4 Project evaluation techniques are an essential aid to improving project performance and achieving best value for money (VFM). This guidance sets out the principles of project evaluation explaining how, why and when they should be done. In carrying out such evaluations, project sponsors will need assistance from one or more of the following: - professional adviser; - project manager; and/or - other consultants. The extent to which they will be involved depends on the stage the project has reached and the scope of the evaluation. 1.5 This guidance assumes knowledge of guidance no 33 “Project Sponsorship” which deals with both the responsibilities and the activities of the sponsor and the professional team (see Annex 3 of guidance 33). These activities are also subject to evaluation techniques. Section 3 of guidance No 33 describes a number of key project tasks which form the core topics of project evaluation Core topics are by definition relevant at all stages in the project life cycle.

1.6 Many of the project controls (eg cost, time, quality and change), which must be in place on any well managed project, are subject to evaluation. These are set out at Annex A together with notes on some of the principal issues to be addressed. 1.7 Project evaluation is part of the total quality process. It contributes towards a successful outcome in terms of: - greater assurance of total performance in terms of cost, time and quality; - clearer definitions of responsibilities; - reduced exposure to risk; and - improved VFM and accountability. 2.

SCOPE

2.1 Although this guidance deals with evaluation of majorworksprojects in general terms, experience shows that it also applies to smaller projects. However, given the varied nature of projects, this guidance cannot be comprehensive. The principles of project evaluation can also be applied to projects other than works. 2.2 The guidance is in 9 sections with supporting Annexes. Section 1. Introduction 2. Scope 3. Definition of terms 4. General principles 5. Pre-project evaluation 6. In-project evaluation 7. Post-project evaluation 8. Occupancy review 9. Independent reviews A full list of contents is at Annex E. 3.

DEFINITION OF TERMS

3.1 Project evaluation is a technique to review the current status of a project against plan and to provide practical, comprehensive and forward-looking recommendations for corrective action where necessary. In general terms it should consider: - project objectives in terms of cost, time and quality; - management;

appropriate to influence future projects. The aim is to use data which is of benefit to non-specialists so that it can be used to influence future decision making. This will help to ensure that past mistakes are not repeated, and that “good practice” is properly understood.

- organisation; - systems and procedures; - suitability of contracts; - performance of consultants and contractors; and

4. GENERAL PRINCIPLES

- work to date in terms of cost, time and quality meas-

ured against plan.

4.1 This guidance provides a summary of activities, procedures and controls which should be in place at each of the following stages of a well managed project:

3.2 Evaluation is often carried out by project participants (project manager, design and cost consultants with contributions from contractors and sub-contractors) and should be a routine performed at predetermined stages to establish that objectives are still achievable. Ad hoc evaluations may also be used as a means of focusing on specific problems. The starting point for ad hoc evaluations often arises from specific concerns of the project sponsor.

- pre-project stage covering appraisal and feasibility study; - in-project stage covering outline design to construction completion and commissioning; and - post-project stagecoveringrectificationofdefects, final payments, post-project evaluation and occupancy review.

3.3 A difficulty with evaluation by project participants is that they often find it difficult to be objective about their own decisions. An evaluation or review by an independent consultant overcomes this problem and can bring in new ideas and useful outside experience.

4.2 Evaluations should: - assess the effectiveness of the project organisation; - review the project execution plan;

3.4 Project evaluationshouldnot be confusedwith ‘value engineering’ or ‘economic appraisal’:

- measure progress against planned cost, time, quality and performance objectives; - identify causes of deviation from plan; and

Value engineering is a pre-planned, formal review of the design philosophy and development, at one or - recommend remedial action. more stages of design development, sometimes led by an independent expert. These studies are short 4.3 Project evaluation is used to check that: exercises intended to review the delated design solutions against the objectives and to establish whether - methods for controlling the cost, time, quality and performance characteristics of a project, from the they can be achieved in a more cost effective manner. initial planning to post completion assessment, are Economic appraisal assesses the costs and benefits sound and effectively applied; of expenditure proposals. It is the process of defining - management information is prompt, adequate and objectives, examining options and weighing up the accurate; costs and benefits before a decision is made. The main source of guidance on appraisal is the Treasury’s “Green - assets and interests are properly controlled and Book” (Economic Appraisal in Central Government: safeguarded from all losses; A Technical Guide for Government Departments - the cost of rechargeable works is recovered; HMSO 1991). It contains a checklist to ensure good appraisal techniques and notes some common errors. - there are adequate safeguards against fraud, error and impropriety;

3.5 When an investment decision is made, a project is “born” - though it may not necessarily pass later stages of approval and reach completion. This decision is the first stage of the Treasury’s project approval procedure (see guidance no 38 “Approval of Works Projects”), ie the initial approval of the options and the assessment of the business case. The second stage is identifying the preferred option, establishing a preliminary budget and obtaining approval to take forward planning and design work on this option.

- material losses due to waste or inefficiency are identified and recovered where appropriate; - information on which costs and cash flow projections are based provide a sound basis for the funding of the project; and - internal guidance and other established policies and procedures are appropriate and are followed. 4.4 Project evaluation is independent of any established reporting requirements. It can be used to encourage the project team to stand back and review what it is doing and where it is going in relation to the stated project objectives. It should end with a formal written report presented to senior management by the sponsor based on data and reports from the project manager.

3.6 Project evaluation can be used from the second approval stage (by evaluating the chosen option against the objectives in the appraisal) throughout the project’s life, ending with the post-project evaluation and occupancy review. It is important to note that the term ‘evaluation” in the ‘Green Book’ is concerned with postproject evaluation only.

4.5 A disciplined and systematic approach to evaluation is recommended so that it becomes a routine project activity, heading off problems before they occur.

3.1 Post-project evaluation is a means of recording experience from past projects, so it can be used where 2

The project cycle

Evaluations, project stages and approvals

4.6 Every project can be visualised as a series of steps requiring a variety of technical skills to manage inherent risks. Initially objectives must be defined and options for achieving them generated and evaluated before deciding which option should be implemented. Project evaluation techniques provide the sponsor with an additional means for controlling and monitoring the progress of a project.

4.8 Although projects and their procurement strategies vary, there is a pattern of definite stages common to most projects. Some of these stages overlap depending on the type of project and the method of procurement. Project evaluation is very effective when carried out at the end of the project stages which are immediately prior to the next approval stage. The link between project stages and approval stages is illustrated in the following table.

4 .7 The steps in the project cycle and the stages when a routine project evaluation should happen are at Figure 1.

Figure 1: The Project Cycle

Definition

Generation of options

Evaluation of short-listed options SELECTING CONTRACT

t

Decisions - Choice of options -Whether to implement

Monitoring and control

Monitor and con - Responsiblties Whether to implement

Timing of evaluations Project Stage

1

Approval Stage

there is an identification stage, when the

need is apparent, the benefits are measurable, the business case can be established, initial studies are carried out and options are appraised; the preferred option is taken forward and subjected to further feasibility studies (the preferred option may have established a location and even a site for the building) feasibility studies will examine the ways the building might fit the site, the environmental impact, planning and statutory requirements and estimated cost and funding;

planning and design work is taken forward through outline (concept) design and into scheme design; the project execution plan is further developed;

Business case/option appraisal.

Submission for approval to take forward planning and design work.

Approval Estimate leading to submission for final approval as soon as cost estimates of the required accuracy can be prepared (not linked to any specific design stage but usually around final sketch plan stage).

detail design, procurement and construction which may be sequential

or overlapping depending on the contract strategy;

commissioning and occupation.

Timing

4.9 At intermediate stages, the natural points to consider evaluation are again at the end of one stage and prior to the commencement of the next, eg at the end of scheme design but before detail design or prior to the submission for final approval. 4.10 During construction, evaluations are recommended at the mid point of the construction programme or once a year for longer programmes. More frequent evaluations should be considered for particularly complex projects. Ad hoc evaluations to identify and solve specific problems may be called for at any time throughout the project.

Submission for re-approval if at any time forecast outturn cost is greater than approval and/or conditions established at final approval are likely to be broken. Occupancy review.

4.11 The ability to control the outcome of a project is

greatest in the early stages decreasing rapidly as the project progresses as illustrated in Figure 2 below. However, uncertainties about the outcome of the project are also greater. Project managers should use risk analysis techniques to help evaluations and advise the sponsor to make plans and choose strategies which reduce risk, and where appropriate place remaining risks where they can be managed most effectively. Risks should be reflected in the level of contingency allowance so that outturn cost and time can be contained within forecasts. See guidance no 4 1 ‘Managing Risk and Contingency for Works Projects’.

and systematic manner. Other participants may include members of the sponsor’s organisation, the professional adviser, the design team, the construction team and the intended users of the building.

Figure 2: Degree of Project Control High

4.18 If project sponsors suspect that project managers have not been fulfilling their responsibilities, they should consider either an evaluation by the professional adviser or an independent review. The professional adviser can provide guidance on this. Reporting

Low

4.19 It is important that the evaluation should be documented clearly by all concerned so that if the need for a further review is identified the team can follow the process of planning, work, findings, conclusions and recommendations.

Topics for Evaluation

4.12 Annex A lists some of the topics to be evaluated at various stages of the project and, for each topic, several key issues to which the evaluation team should provide answers. Controls and procedures often have to envolve as the project progresses. The evaluation should check that a control mechanisms are: - established, appropriate and being used;

4.20 Each participant in the evaluation team should keep working files to support individual evaluation reports, recording the details of each stage from evaluation planning through to reporting. 4.2 1 The project manager will provide the sponsor with a written report, based on contributions from all the participants (but checked by the project manager as necessary) highlighting the key areas, This report should address in particular any aspects of control systems and procedures which were found wanting. The implications of these deficiencies should be explained together with recommendations for improvements.

- reviewed and amended to meet changing project needs; and - complied with. 4.13 Where existing controls are considered unsatisfactory, evaluations must provide the sponsor with recommendations for improvement. An independent review may be necessary if the evaluation either cannot establish the problems adequately or reveals a particularly serious problem for which the underlying cause or corrective action cannot be identified quickly. This is particularly applicable if poor performance, fraud or some other impropriety is suspected.

4.22 The project sponsor reports the results of the evaluation to the project director on the basis of data and reports received from the project manager. The sponsor’s evaluation report should comprise: - a short (one to two page) executive summary listing

the reasons for the report, the principal conclusions and the actions to be taken; and

Duration

- sufficient detail on the findings of the evaluation to

4.14 Evaluations should be sufficiently detailed to satisfy

identify any further action required.

the sponsor that the project is on course to meet its planned objectives or, if not, that the cause of any problem has been identified and proposals made for its resolution.

5.

4.15 Typically evaluations should last one to two weeks, depending on the stage the project has reached and its size, importance and complexity. This is particularly important when the evaluation is carried out to deal with a perceived problem, so that corrective action can be taken quickly and the effects of the problem mitigated.

PRE-PROJECT EVALUATION

5.1 At this stage, prior to commencing design and construction, the objectives of the project sponsor’s department and the project itself are established. Project objectives should be defied clearly and prioritised in terms of time, cost, quality and performance. A preproject evaluation should be carried out to confirm that this is so by investigating:

4.16 Evaluation of a specific problem will be achieved best by limiting its scope. For example, an evaluation to identity and reduce the causes of a cost overrun, may be limited to all changes made or planned. The review may be further limited where the problem is more specific, eg whether the cost effects of a particular design change have been fully assessed.

- the clarity of the project objectives; - the viability of achieving those objectives; - whether sufficient options for achieving the objectives have been generated; - that these options have been properly evaluated;

Management

- the degrees of influence in the client organisation;

4.17 The project sponsor is responsible for ensuring that evaluations take place at intervals recommended in this guidance. The project manager, when appointed, is responsible for managing the evaluations in a disciplined

- that the main risks and constraints to the project have been identified and assessed; and - that the project has appropriate funding. 5

5.9 Most crucial decisions are taken prior to commencing design (see paragraph 4.11). The evaluation should identify where action needs to be taken. Changes at this stage are rarely as costly or as disruptive to the project as those implemented later.

The purpose of a pre-project evaluation is to provide a checkthat these matters are being or have been resolved satisfactory. 5.2 If the evaluation reveals serious flaws in the project or its proposed implementation, it may be necessary to delay moving to the design stage until further investigation by a more detailed evaluation or independent review is complete and the implications have been assessed.

5.10 When the pre-project evaluation confirms that the project objectives are achievable and that the necessary controls are in place, the project can proceed to the design stages.

Timing

6.

5.3 The best time for a pre-project evaluation is shortly after appointment of the project manager, and before appointing the permanent design team. It may be necessary to seek specialist advice, for example on risk identification and analysis, during the course of the evaluation. In this event temporary appointments may have to be considered.

IN-PROJECT EVALUATION

6.1 During the design stages of a major project, interaction between the various parties is intense and complex. Oversights and mistakes can easily occur. As the project design progresses the opportunity for introducing significant changes (without incurring costs for abortive design work) reduces rapidly. A preplanned evaluation should be carried out at the pre-tender stage to establish that all is well with the project before entering into the most significant financial commitment (ie construction of the project).

Management 5.4 The project sponsor is responsible for ensuring that the pre-project evaluation takes place. The project manager should manage the evaluation.

6.2 Duringthe constructionphase, the project sponsor’s department incurs the major part of the project expenditure. Changes introduced at this stage are likely to incur disproportionate cost increases and result in delay to the project.

Contents 5.5 Annex A indicates some of the topics to be reviewed. The evaluation should establish that the necessary systems and controls exist to proceed to design stage or identify the means of achieving this. The project objectives, preliminary contract strategy and the project organisation should be identified and procedures should exist for the appointment of consultants, and for design and cost control to the pre-tender stage. Many of the control documents at this stage are the initial versions for the project and will be amended by the professional team as the project progresses. However, they should be appropriate for this stage.

6.3 As projects can be subject to significant risks during both the design and the construction phases, in-project evaluations provide the project team with the opportunity to stand back from the detail of the project and: - evaluate how well they are working together; - assess the status of the project against cost, time and

quality objectives;

- identify existing or potential problems; - make forward-looking, practical recommendations

for improvements which can be implemented immediately;

Reporting 5.6 The project manager’s evaluation report should state clearly whether the project should proceed as currently proposed and if not, the actions necessary for the project to be successfully continued. The report must also state whether the evaluation has been adequate for its intended purpose or whether a more detailed evaluation is recommended for all or part of the project.

- check that the risks to the project are being man-

aged; and

- ensure a commitment of funds to the project.

6.4 It is good practice to carry out in-project evaluations during: - design: the project manager should advise the

5.1 An indication of the issues to be covered in the project sponsors’s evaluation summary is at Annex B. However, the summary will need to be tailored to suit the particular project.

sponsor on the timing, but typically this would be prior to commencing detailed design and/or prior to tender and construction; and

- construction: the project manager should advise the sponsor on the frequency, which might typically be at 6-9 month intervals.

Benefits 5.8 Evaluation plays a valuable role at this stage in ensuring that the department has defined the project objectives clearly and that the chosen preliminary contract strategy is consistent with those objectives prior to entering contractual relationships and commitment of funds.

Management 6.5 The project manager, reporting to the sponsor, is responsible for managing the evaluation, the participants and for providing the report.

6

6.13 The advantage of conducting an evaluation at the outline and/or scheme design stages is that changes are 6.6 Annex A summarises some of the topics whichshould relatively easy and economic to introduce. This is not the be evaluated during the design and construction stages. case once detailed design has taken place. The evaluation process should check that: 6.14 Evaluation at mid-construction provides the final - control procedures for the core topics are appropricheck that the project is on course. Whilst significant ate to meet current and future project needs and are changes to project scope at this stage will be costly to being complied with; and implement, changes in control procedures are not and - new controlmechanisms specific to in-project phases may be necessary if forecast time, cost or quality objecexist, are appropriate and are being complied with. tives are not being met. Contents

POST-PROJECT EVALUATION

Reporting

7.

6.7 Evaluations should be summarised and presented to the sponsor in a report prepared by the project manager based on contributions from project participants. The report should: - state clearly whether the project remains on course to meet its plan - consider risks and how they are being managed;

7.1 This should be carried out soon after completion of construction to: - measure the success of the project in achieving its

planned objectives;

- identify the reasons for any problems that have

arisen:

- determine what remedial actions should be taken:

- provide fully evaluated recommendations to deal with any existing or potential problems; and - improve the effectiveness of project execution.

and

- record the lessons which have been learnt to

improve performance on subsequent projects.

The objective is to learn lessons for application to future projects and to pass them onto other departmentalprojects or departments. The post-project evaluation report for a building project may be supplemented by an occupancy review (see Section 8).

6.8 Design stage evaluations should recommend whether to proceed to detailed design and/or construction They must state whether the evaluation is sufficient or whether a more detailed independent review is required. Duration

7.2 There is a minimum level of post-project evaluation which should be carried out on all major projects. The aim ofpost-project evaluationis to improve project appraisal, design, management and implementation, Its objective is to get maximum benefit from accrued experience, not to apportion blame.

6.9 Outline design stage and scheme design stage evaluations should be completed normally within 2 weeks. If serious problems are revealed, moving to the next design stage should be delayed until a further detailed evaluation has been undertaken, the implications assessed and remedial measures taken.

Timing

6.10 Construction stage evaluations should proceed in parallel with construction and normally be completed within 2 weeks.

7.3 Time is not usually critical at this stage. However, the earlier the post-project evaluation is done, the earlier the lessons are available for subsequent projects. In particular, the evaluation should be completed sufficiently early to ensure that:

6.11 Delaying the construction process is likely to prove very costly and should only be considered if: - the evaluation reveals serious problems which significantly affect the achievement of project objectives; and

- knowledge is still live; - those involvedwiththeproject are still available; and - outstanding obligations are identified and steps taken to ensure they are fulfilled.

- these problems will worsen significantly if construction is allowed to continue before taking remedial action.

Management

In these cases a more detailed evaluation or independent review may be necessary to provide recommendations for corrective action.

7.4 The project manager is usually responsible for directing the post-project evaluation. The evaluation requires the active participation of all parties to the project as part of their final duties and this should be set out in their terms of appointment.

Benefits 6.12 Evaluations during in-project stages should identify the causes of any problems and provide recommendations for remedial action. The sooner this is done, the more likely the project will be completed successfully. It serves to limit the risk to which the project sponsor’s department is exposed.

Contents 7.5 The evaluation should assess whether the objectives of the project were realised and the established controls complied with. The effectiveness of these controls should 7

anticipated to change duringthe course oftheproject, to assess whether those changes were accommodated:

also be reviewed and recommendations should be made for improvement of controls on future projects. Reporting

- confirm that the building functions in use as planned; - ensure that any outstanding work, including defects, have been remedied; and - ensure that key lessons learnt on occupation are put into practice without delay for the benefit of future and continuing major projects,

7.6 Post-project evaluationshouldconcludewithareport prepared by the project manager using contributions fromtheprojectparticipants. It shouldstateclearlywhether the project met its objectives in terms of time, cost, quality and performance. Recommendations for reducing the possibilityofproblemsoccurringonfutureprojectsshould be made. It should also highlight aspects of project management which have worked particularly well or not and state that the evaluation was sufficient for its intended purpose or whether a more detailed evaluation is required.

Management 8.3 The occupying business manager responsible for the property should arrange for the occupancy review to be conducted. Assistance will be required from the senior users and preferably also the project manager.

7.7 The sponsor adds a report on the project manager’s performance. The sponsor submits the final post-project evaluation report to the project director who uses it to check whether the department’s objectives and requirements have been met.

Contents 8.4 An occupancy review should include:

7.8 Typical issues to be addressed in the report are at Annex B but the report should be tailored to particular projects and evaluations.

- an assessment of whether the users’ requirements have been met. If not, consideration should be given to the briefing process and the extent to which users were consulted and involved as the design developed; - costs in use - how the planned operating cost of the building compares with actual cost and how the planned efficiency and effectiveness of the equipment and facilities compare with actuals; and - any further recommendations the users can make to improve VFM performance of future projects. For example, the users may consider that some of the facilities provided were unnecessary.

7.9 Departments who use the Contractor Management Information System (CMIS) and the Consultants Register (CONREG) operated by Construction Industry Sponsorship (CIS), DOE, 2 Marsham Street, London SW1P 3EB, are required to submit performance reports on contractors and consultants as a matter of course. Benefits 7.10 A post-project evaluation provides an opportunity to review the project with the benefit of hindsight. It reviews all project stages and the problems that have arisen thereby providing lessons for the future and identifying how problems arose. It is also valuable to conduct postproject evaluations on major projects that have gone particularly well, in order to identify ingredients for success.

Reporting 8.5 Typical issues to be addressed in the occupancy

review report are at Annex C. The report should be tailored to suit the individual project.

7.11 The evaluation can also provide the sponsor with assurance that the works have been completed in accordance with the contracts, that consultants and contractors have discharged their responsibilities and that all potential liabilities have been recognised and quantified. It will expose any shortfall in these areas and enable the sponsor to recognise where commitments still exist and/ or damages might be recoverable.

a.

Benefits 8.6 The occupancyreviewis the finalfeedbacklinkfrom the users which closes the loop of the project cycle. The review also identifies the existence of any outstanding obligations, with which the project sponsor must deal.

9.

INDEPENDENT REVIEWS

9.1 Project sponsors should commission independent detailed reviews if they are concerned about either the general status and progress of the project or, more usually, about specific issues.

OCCUPANCY REVIEW

8.1 When the project relates to a building, an occupancy review should be performed one to two years after the building has been occupied. The result of this review is an addendum to the post-project evaluation. It records how successfully the completed project meets the requirements of the occupiers and identifies any further work required.

9.2 Its aims should not be limited to identifying the root causes of any problem but should also make practical recommendations for corrective action and aim to generate direct and significant time, cost and quality improvements.

8.2 An occupancy review has the following objectives to: - consider whether the objectives of the users have been met. If those objectives changed, or were

When an independent review is necessary 9.3 The need for an independent review may arise at 8

any time when the project sponsor considers that certain problems are serious enough to require separate investigation However, the project manager should also advise the project sponsor when one is necessary. There are a number of typical warning signs: - cost issues:

- systems evaluation;

- compliance testing; and - tests of detail. The flowchart at Figure 3 (page 10) indicates this process. Systems evaluation

- large increases between successive estimates; - tenders for construction being significantly above or below estimate; - a significant overspend or underspend compared with predicted cash flow; - unpredicted changes resulting in high additional costs;

9.8 This comments on the adequacy of control system and offers recommendations for improvement. Th approach is broadly based rather than detailed and i conducted in two stages. First, the system is recorded in terms of its operation, preferably in flowchart forn Second, the system is evaluated by defining its control objectives and assessing the strengths and weaknesses of the system in meeting those objectives.

- large contractual claims for additional payments or extensions of time; and - a final contract sum which is much higher than the tender sum;

9.9

A typical evaluation might cover the following:

- is the procedure clearly defined? - is someone nominated to carry out the procedure - is this person fully aware of the responsibility fc

-time issues:

carrying out the procedure?

- progress falling significantly behind programme eg failure to meet milestone dates;

- is the person required to demonstrate that th

procedure has been followed?

- delays in providing the contractor with necessary information, which results in delays and/or additional construction costs; and - late completion;

- are there any further procedures providing for a

independent witness or check that the procedure has been carried out? and

- does the procedure provide the intended control

- quality and performance issues: - a pattern of failings in design or workmanship; and - a finished product which fails to meet the users requirements in certain important respects;

Compliance testing 9.10 This is used to check that the system is operating a intended. It focuses on the key controls of the system an the principal management actions which must be carried out if the system is to operate as intended. The tests are i whatever form is appropriate to determine whether the key controls are operating in practice. Examples of these tests include:

- management/organisation issues: - delays, cost increases or reduced quality standards arising from failure to provide the right people with the necessary information to make decisions or take actions at the right time; and

- observation of the procedure in operation; - examination of working papers and formal docu-

- lack of clarity of roles and responsibilities,

mentation; and

Procedures

- interviews with project participants to determine th

actual procedures in use.

9.5 Reviews will be capable of identifying problems and providing recommendations for their resolution. The reviewer’s approach should be objective. Subjective judgements, for example relating to truth or fairness, should be expressed specifically as an ‘opinion’ as distinct from a ‘finding’.

9.11 If systems evaluation has proved that the system is itself inherently weak, compliance testing is usually inappropriate. Tests of detail

9.6 The project sponsor should make reviewers aware of the circumstances which make the project unique. Although reviews may cover the same topics as evaluations (see Annex A) they are often in greater depth and therefore detailedprocedures will need to be established individually for each review.

9.12 These may be necessary when systems evaluatio and/or compliance testing have revealed weaknesse and deficiencies. There are two types of tests of detail: - those concerned with financial values; and - those concerned with the quality of managemer decisions.

Review of project management controls

9.13 Financial tests of detail either start at:

9.1 Project management controls can be audited as follows: 9

Figure 3: Audit Flowchart Select system to be audited

I

Does system exist?

No

)

I

7 I

I

1

Gather data on how system operates

I I

I

Systems evaluation

I

Flowchart the system

I

Define what the system is intended to achieve

I

I

t Evaluate strengths and weakmesses of system

I

Report to management

I

-

-

-

-

-

-

-

-

-

-

-

-

-_-----

1 1 Debekey contr;kthesystem

I I I I I I

Devise sampling procedures and compliance tests

~~

I

Compliance testing

I I I I I /

Reporttort

I

+

-

-

-

-

-

-

-

-

-

-

-

-

- - - -

I

Tests of detail

Report to management

+--_---_____----------

10

I I I I I

I I I I I I I I I I I I I I -I

facts during their investigation, They should have relevant experience and a successful record of conducting construction project reviews. Suitable candidates can be found in construction and property management consultants, project and cost management consultants and consulting engineers. The professional adviser and/or project manager can advise the project sponsor on selection.

- a figure in an account - then trace the data or

evidence validating that figure (eg expenses or costs); or

-

the data or evidence - then check that the correct figure arising from that information is included in the account (eg income or deductions from expenses).

9.14 During the course of a project, some management decisions fall outside the scope of systems and control. These decisions may have to deal with unforeseen circumstances eg poor ground conditions, bad weather or poor interaction between the parties to the project. Objective tests of detail consider the quality of these decisions by: - setting down clearly the relevant history and facts relating to the decision;

9.20 Details of the form of the review, the scope of the work and the final report should be agreed between the project sponsor and the review team before work starts. 9.21 The reviewers will need the co-operation of the project participants in gathering information. However, when the project sponsor has commissioned a review rather than an evaluation of the project, participants may feel threatened and may be reluctant or unwilling to cooperate. This may reduce the effectiveness of the review. The obligation to co-operate must therefore be included in the terms and conditions of engagement of the project participants. When fraud or negligence are being investigated, it may be necessary for the sponsor to remind participants of their contractual obligations on co-operation.

- considering all options available to management;

- analysing how well each option relates to the achievement of the project objectives; and - recording the reasons for selecting a particular

option.

9.15 Good decision makers should have followed the process described above. However, it is stillnecessary to check that the process has been completed correctly and that the associated working papers are on file.

Contents

9.22 Annex A summarises some of the topics which may be included at different stages of the project. The project sponsor should define the scope of the review and prepare a brief for the review team. - 9.23 The level of detail of the review depends on:

Scope of review

9.16 The scope of an independent review will depend on the nature of the problem under consideration, its associated risk and why it is being carried out. If the objective is simply to assess the effectiveness of systems and controls and to provide recommendations for improvement, the review may be limited to systems evaluation and compliance testing. If a judgement on a specific issue is required, compliance and detailed testing will be necessary. If fraud is suspected, all three levels of review are likely to be necessary to reveal the full extent of the problem.

- the project sponsor’s brief to the review team; - the project stage; - the matter under investigation: and - discoveries made during the review process. The reviewers should investigate whether the controls are clearly defined and are being satisfied. November 1993

9.17 However, the review’s size may be limited by sampling techniques. Judgmental sampling relies on the reviewer’s judgement to pick out the major or most contentious items for investigation. Whilst this method is quick and cost-effective, the project sponsor should recognise that oversights may occur. 9.18 Statistical sampling is likely to be more accurate but more time-consuming. The reviewer selects the size and type of sample based on an assessment of risk and importance. The reviewer will advise the sponsor on how and why the sample was chosen and the confidence that can be placed in the results. Management 9.19 The project sponsor is responsible for commissioning independent reviews and appointing the review team to undertake the investigation. The team should be independent, without any conflict of interest, unbiased and have no reason to protect any individual or conceal any

11

ANNEX A OUTLINE CHECK LIST OF TOPICS FOR EVALUATION

This Annex lists some of the topics (with their key issues and problem areas) which may be reviewed during project evaluation. Where appropriate, evaluation teams should both examine and provide answers to each. However, the list is not exhaustive and will need to be reviewed depending on the project’s stage and its contract strategy. In addition, the list may also need amendment to reflect the scope and requirements of the particular project. The project sponsor, on the advice of the project manager and/or professional adviser, should define the scope of partial or specific evaluations.

Force Majeure

Controls andprocedures evolve as theprojectprogresses and evaluations should establish that they are: - established and appropriate to the stage of the project; - being reviewed and amended to suit the changing needs of the project; and - complied with.

Estimating Data

catastrophic events Design development designers design team responsibilities design brief design process design programme design elements the design cost estimate for professional services (resources) works cost estimate other cost estimate project programme Site Conditions the site conditions above and below ground susceptibilities - flooding, bad weather labour availability existing services existing buildings environmental impact assessment

Where procedures and controls are considered unsatisfactory evaluations must provide recommendations for improvement. The suggested topics for evaluation are as follows: Project Definition user requirements project objectives project brief planning, social and environmental impact assessments

Construction and Commissioning contractors site management responsibilities health and safety measures emergency procedures permits to work construction methods long delivery, short supply items commissioning methods handover procedures operating manuals as built drawings

Project organisation and management department’s client organisation project management team project organisation structure risk assessment approved budget cost, time, quality and change control procedures contract strategy project execution plan procurement procedures detail design reporting systems contractual claims Availability of funds Statutory and other External Factors planning approval building regulation approval health and safety regulations site access inflation interest rates currency exchange rates VAT 12

ANNEX A - continued Project Organisation and Management -continued

PROJECT DEFINITION User requirements

Project objectives

Properly defied? Incomplete or unclear? Conflicts between primary/ secondary objectives? Possibility of change?

Project brief

Complete? Inadequate or unclear? Conflicts between different requirements? Possibility of change? Are the following unclear, in conflict or likely to change: - site; - scope: - extent; - quality and performance: - design concept; - cost and time; or - completion date? Sufficiently output orientated to provide design flexibility?

Planning, social and environmental impact assessments

Project management team

Adequately defined? Incomplete or unclear? Conflicts between different user requirements? Likely to change?

continued

Carried out? Issues of concern identified? External authorities consulted? Their requirements met? Outstanding matters?

Project organisation structure

Clearly defined? Clear direction and communication? Project team responsibilities clear? Gaps or overlaps? Arbitrary external interference in project team? Clear communication and identified responsibility with interested organisations not directly involved?

Risk assessment

Carried out at important stages? Used to determine contingency, tolerance and approved budget? Control budget within approved budget? Conditions attached to authorisation to proceed?

Cost control procedures

Effective and working? Control budget regularly reviewed? Costs monitored and exceptions acted upon? Time control effective and working? Project programme regularly reviewed? Progress monitored and exceptions acted upon? Quality control effective and working, exceptions acted upon? Change control effective and working? Sponsor’s approval to changes?

Contract strategy

Takes account of project

Procedures for procurement

Effective and working? Comply with EC/GATT rules? Tender documents adequate? Tender periods adequate? Information reliable?

PROJECT ORGANISATION AND MANAGEMENT Department’s client organisation

Project management team

Clearly defined? Clear direction and communication? Sponsor and client team responsibilities clearly defined? Appropriate? Gaps or overlaps? Arbitrary external interference in client team? Sponsor administers contract efficiently? Notes and decisions issued promptly? Correct balance of resources and expertise? Effective? Responsibilities clear? Appropriate? Project manager exercises adequate authority?

13

Project manager pro-active? Continuity in project management’ Project manager’s links with participants effective? Project manager administers contracts efficiently? Queries dealt with promptly?

objectives? Project execution plan current? Defines the work packaging of design, procurement, construction and commissioning? Defines how activities are coordinated and managed?

STATUTORY AND OTHER EXTERNAL FACTORS

Project Organisation and Management -continued Procedures for procurement

continued

Detailed design

Tender documents protect department from low tenders with potential for claims? Tender evaluation appropriate? Approval and award of contracts effective and working? Contracts clearly define scope of work and responsibility of contracting party? Any nominated sub-contractors? Department’s responsibilities recognised and taken into account?

Possibility of delay in obtaining? Project compliant with building and health and safety regulations? Necessary statutory approvals obtained? Access to site subject to approval of external organisations or authorities? Bureaucratic procedures likely to cause delay? Anticipated changes in regulations or legislation affect project?

Inflation, interest rates, rates of exchange and VAT

Inflation taken into account? Affected by interest rates? - taken into account? Affected by rates of exchange? - taken into account? VAT taken into account?

Complete before construction starts?

Design requirements properly coordinated? Level of design detail appropriate and adequate? Likely to change after start of construction? Design information complete and timely? Queries and conflicts resolved quickly?

Reporting systems

Effective and working? Include narratives on current status of cost, time, change and quality? Shortfalls in meeting objectives highlighted? Corrective action identified and initiated? Identify actual and anticipated critical areas? Essential information provided in a coherent format? Information used to prompt action and take decisions?

Contractual claims

Likely to arise? Performance likely to prompt claims? Adequate records maintained to counter claims? Validity and potential exposure to actual claims independently audited?

FORCE MAJEURE Catastrophic events

Department’s investment protected from extreme weather, flood, landslips, earth quakes, fire, crashing aircraft or other catastrophic events? Adequate plans for dealing with them? Adequate precautions against war, revolution, riot, insurrection, hijack, piracy, pestilence, disease, etc? Protected from claims by consultants and contractors under terms of force majeure clauses?

DESIGN DEVELOPMENT Design team

AVAILABILITY OF FUNDS Availability of funds

Planning approval, building approval, licences and permits

Subject to approval or annuality? Dependent on other departments or agencies? Allocation adequate? Funding policy (or funding policy of other departments or agencies) likely to change? Timing of funds match cashflow forecasts? Timing or scale of funds likely to change?

Adequate vetting before short listing? Selected using weighted quality and performance criteria for bid evaluation? Properly experienced and competent? Financially secure?

Adequately defined? Design team responsibilities Clear?

Clear organisation structure? Effective? Match technical requirements? Correct balance of resources and expertise? Gaps or overlaps?

14

Estimating Data -continued

Design Development -continued Design briefs

Complete? Issued early enough to design team members? Adequate and clear? Conflicts between different requirements? Any omissions? Approved by sponsor?

Design process

Managed and coordinated effectively? Exchange of technical and design information effective? Design optimisation accounts for relevant objectives?

Design programme

Delivery of information ahead of construction? Design resources deployed adequate? Meets all requirements?

Design elements

Selected to meet requirements of contract strategy and execution plan? Costed as developed? Design within constraints of control budget and design prog-ramme?

The design

Appropriate? In accordance with design brief? Design data appropriate? Makes use of new or untried technology? Procedures for designs to be reviewed by sponsor?

ESTIMATING DATA Cost estimate for professional services

Reliable for resources? Based on ad valorem approach? How applied percentages determined? Reflects current market forces? Realistic and reliable? Resources cost estimate based on consultants’ costed proposals? Includes allowances for costs arising from terms and conditions of consultants’ contracts? Allowances realistic? All foreseeable work of external advisers, project managers, architects, design and specialist consultants, quantity surveyors (and in-house staff) included?

Works cost estimate

Reliable? Based on quantity surveyors’ approach? How accurate are quantities? Historical all-in data reliable? Provisional lump sum items reliable? Proportion of work costed using lump sum guestimates? Reliable percentage applied to cover contractors’ preliminaries, staff, plant, accommodation, overheads and profit? Figures reflect current market forces? Realistic and reliable? Based on contractors’ tenders? Includes allowances for costs arising from terms and conditions of contractors’ contracts? Allowances realistic? Work for health and safety, security, site preparation, temporary works and facilities, permanent materials, plant and equipment, construction and installation, connection to services, commissioning and handover included?

Other costs estimates

Reliable? Provide adequately for land purchase? Handover and occupancy costs included and reliable? Department supplied furniture fittings and equipment reliably identified and costed? Works of art included?

Project programme

Reliable? Level of development? Produced using CPM or PERT methodology? Properly reflect objectives of contract strategy and execution plan? Regularly updated?

SITE CONDITIONS The site

I.5

Greenfield or brownfield? Data adequate on archeological remains, underground obstructions, contamination and ground conditions? Adjacent structures or works likely to be affected? Special precautions necessary? Department’s interest protected?

Site Conditions -continued The site

continued

Existing services

Existing installations

Construction and Commissioning -continued

Liable to flooding? Access adequate? Any additional infrastructure? Liable to unusual weather conditions? Seasonal weather pattern affect programme? Remote? Adequate local labour and accommodation? Will local customs, holidays or religious restrictions affect programme?

Health and safety measures

Adequate? Individual and company responsibilities clear? Client’s responsibilities recognised? Emergency procedures established? Updated regularly? Permits to work? Effective and working?

The construction

Methods defined and agreed? Realistic programme drawn and agreed? New, unusual or untried construction processes? Special skills required? Special plant or equipment required? Permanent materials, fixed plant or equipment in short supply? Approval subject to special testing, design or storage? Points when management passes from construction to commissioning defined?

Commissioning

Methods defined and agreed? Scope of work defined? Individual items of equipment, the systems and the whole system identified? Adequate performance specifications? Realistic programme drawn and agreed? Coordinated with achievable dates for construction/ completion? Ongoing construction and partial operation? Points when management responsibility passes from construction to project handover defined?

Handover procedures

Effective and working? Completion of contractual obligations ensured? Operating manuals and as-built drawings complete and of required quality? Scope of required documentation adequately defined? Includes scope of supplier’s documentation?

All identified locations known

accurately? Present a hazard? Require future maintenance? Available for use during construction and installation?

Any locations currently

inaccessible? Locations of asbestos and hazardous material identified? Environmental impact assessment carried out? Any matters arising?

CONSTRUCTION AND COMMISSIONING Contractors

Management

Adequately vetted before short listing? Experienced and competent to undertake contracted works? Quality of management and coordination adequate? Contractors, sub-contractors or specialist suppliers to provide design services? Sub-contractors and specialist suppliers experienced and competent to undertake works? Quality of coordination and supervision works adequate? Financially secure? Responsibilities clear? Representation on site of the various functions? On site organisations have correct balance of resources and expertise? Clear individual management responsibilities? Interfaces between organisations clearly defined and understood?

1

PROJECT EVALUATION SUMMARY

Project name:

Evaluation date:

Project sponsor:

Department:

Project stage: Pre-Project/In-Project/Post-Project * * delete as applicable Project status

Conclusions

Theprojectsponsor’sreportmustbe basedontheproject manager’s evaluation report and record:

The project sponsor should provide information on les-

sons learned during the course of the evaluation so that future projects may benefit from:

- whether the objectives of the evaluation were met;

- what went particularly well and why; and

- whether the systems, procedures and controls for the areas under evaluation:

- where problems were experienced, the reasons for these and recommendations on how they could be avoided.

- have been established; - are or were appropriate; and

Attachment

- are or were being used;

The project sponsor should also attach the brief issued to

the project manager for the evaluation.

- whether the project objectives in terms of time, cost and performance have been defined and are being met at the time of the evaluation; and - an explanation of deviations from those objectives, Recommendations The project sponsor should provide the following recom-

mendations, based on the project manager’s report:

- whether the evaluation was sufficient and, if not: - what particular aspects should be subject to further evaluation or independent review; - whether the project should ‘proceed as currently proposed and, if not: - an explanation of the reasons; - proposed actions to allow the project to proceed; and - implications, timescales and responsibilities of these actions.

17

OCCUPANCY REVIEW SUMMARY Project name:

Review date:

Project sponsor:

Department:

Occupying manager:

Department:

Date of occupation:

Project status

Recommendations

The report must address:

The report should consider from the point of view of the user’s requirements those aspects of the completed building that are particularly:

- whether the user’s original requirements have been met. If not:

- useful;

- what were the shortfalls;

- efficient;

- explanations of these;

- surplus to needs; - uneconomic to operate;

- if the user’s requirements changed significantly

during the course of the project: - what these changes were;

- whether they were accommodated and, if not: - are they still required;

- and under performing. Recommendations and what should be included and avoided on future projects should be provided in the light of this assessment.

- whether the facilities provided are adequate, fall

short of or exceed requirements, appropriate supporting evidence should be provided;

- identification of outstanding defects and remedial

works; and

- an assessment of whether the building is economic

to maintain and operate, with details of any unsatisfactory aspects,

18

OTHER GUIDANCES

Applicable CUP Guidances

No 1

Post Tender Negotiation

No7

Project Sponsorship: Planning and Progress Monitoring

No 11

Debriefing

No 12

Contracts and Contract Management for Construction Works

Produced by Property and Buildings Directorate, Department of the Environment, St Christopher House, Southwark Street, London SE1 OTE Contracting for Works Services

Contractor Management Information System Handbook Consultants Register Guide Guidelines for the Design of Government Buildings

No 13

The Selection and Appointment of Works Consultants

No 15

Estimating for Works Projects

No 17

Quality Assurance in Building and Construction

Fire Precautions Guide

No 19

PTN Update

The Government Property and MOD Fire Standards

No 20

The P&S Function and Works Projects

Standard Fire Precautions for Contractors

No 25

Cost Management for Works Projects

No 26a

Selection of Works Contractors: Prequalification and Tendering Procedures

No 26b

Selection of Works Contractors: Bid Evaluation and Award

No 33.

Project Sponsorship

No 36

Contract Strategy selection for Major Projects

No 38

Approval of Works Projects

No41

Managing Risk and Contingency for Works Projects.

Environmental Action Guide (including series of six notes) Historic Buildings Conservation Guide

Better Briefing means Better Buildings by J J W O’ReilIy published by the Building Research Establishment of the Department of the Environment.

To be published

Procurement of Project Management and other consultancy services (excluding design) including model form of contact.

19

CONTENTS

Commencing paragraph

1. INTRODUCTION

1.1

2. SCOPE

2.2

3. DEFINITION OF TERMS

3.1

4. GENERAL PRINCIPLES

4.1

The project cycle Evaluations, project stages and approvals Timing Topics for evaluation Duration Management Reporting

4.6

5. PRE-PRO JECT EVALUATION Timing Management Contents Reporting Benefits 6. IN-PROJECT EVALUATION Management Contents Reporting Duration Benefits 7. POST-PROJECT EVALUATION Timing Management Contents Reporting Benefits

8. OCCUPANCY REVIEW Management Contents Reporting Benefits 9. INDEPENDENT REVIEWS When an independent review is necessary Procedures Review of project management controls Systems evaluation Compliance testing Tests of detail Scope of review Management Contents

4.8 4.9 4.12 4.14 4.17 4.19

8.1 8.3 8.4 8.5 8.6 9.1 9.3 9.5 9.7 9.8 9.10 9.12 9.16 9.19 9.22

5.1

ANNEX

5.3 5.4 5.5 5.6 5.8

A Topics for evaluation

12

B Project evaluation summary

17

C Occupancy review summary

18

6.1

D Other guidances

19

6.6 6.7 6.8 6.10 6.13

E Contents

20

7.1 7.3 7.4 7.5 7.6 7.10

20

Page