White Paper - Data Warehouse Governance

Data Management & Warehousing

WHITE PAPER

Data Warehouse Governance DAVID M WALKER Version: 1.1 Date: 06/04/2007

Data Management & Warehousing 138 Finchampstead Road, Wokingham, Berkshire, RG41 2NU, United Kingdom http://www.datamgmt.com

White Paper - Data Warehouse Governance

Table of Contents Table of Contents ...................................................................................................................... 2 Synopsis .................................................................................................................................... 3 Intended Audience .................................................................................................................... 3 About Data Management & Warehousing ................................................................................. 3 Introduction................................................................................................................................ 4 What is governance?................................................................................................................. 5 What affects the data warehouse? ............................................................................................ 6 What is affected within the data warehouse? ............................................................................ 7 How often will change occur?.................................................................................................... 9 What organisational structures are needed?........................................................................... 10 Executive Sponsor(s) .......................................................................................................... 10 Steering Committee ............................................................................................................ 11 Programme Management ................................................................................................... 11 Implementation Teams........................................................................................................ 12 Exploitation Teams.............................................................................................................. 12 User Forums ....................................................................................................................... 12 Certification Committee....................................................................................................... 13 Project Development Methodologies....................................................................................... 14 Waterfall .............................................................................................................................. 14 Iterative ............................................................................................................................... 14 Agile .................................................................................................................................... 15 Cowboy Coding................................................................................................................... 16 Best practices .......................................................................................................................... 17 Momentum .......................................................................................................................... 17 Monitor and Measure .......................................................................................................... 17 Prioritise .............................................................................................................................. 17 Light touch........................................................................................................................... 17 Communication ................................................................................................................... 17 Standards............................................................................................................................ 18 Track processes.................................................................................................................. 18 Understand cost and value ................................................................................................. 18 Continuous Learning and Improvement .............................................................................. 18 Pilot & Prototype ................................................................................................................. 18 Authority to act .................................................................................................................... 18 Plan for the long term.......................................................................................................... 19 Governance and Success ....................................................................................................... 19 Summary ................................................................................................................................. 20 Appendices.............................................................................................................................. 21 Appendix 1 - Programme or Project?.................................................................................. 21 Appendix 2 – The "Declaration of Interdependence" .......................................................... 22 Appendix 3 - Team Values and Principles .......................................................................... 23 References .............................................................................................................................. 24 Web resources .................................................................................................................... 24 Acknowledgements ................................................................................................................. 24 Copyright ................................................................................................................................. 24

© 2007 Data Management & Warehousing

Page 2

White Paper - Data Warehouse Governance

Synopsis An organisation that is embarking on a data warehousing project is undertaking a long-term development and maintenance programme of a computer system. This system will be critical to the organisation and cost a significant amount of money, therefore control of the system is vital. Governance defines the model the organisation will use to ensure optimal use and reuse of the data warehouse and enforcement of corporate policies (e.g. business design, technical design and application security) and ultimately derive value for money. This paper has identified five sources of change to the system and the aspects of the system that these sources of change will influence in order to assist the organisation to develop standards and structures to support the development and maintenance of the solution. These standards and structures must then evolve, as the programme develops to meet its changing needs. “Documentation is not understanding, process is not discipline, formality is not skill”

1

The best governance must only be an aid to the development and not an end in itself. Data Warehouses are successful because of good understanding, discipline and the skill of those involved. On the other hand systems built to a template without understanding, discipline and skill will inevitably deliver a system that fails to meet the users’ needs and sooner rather than later will be left on the shelf, or maintained at a very high cost but with little real use.

Intended Audience Reader Executive Business Users IT Management IT Strategy IT Project Management IT Developers

Recommended Reading Synopsis and Summary Entire Document Entire Document Entire Document Entire Document Entire Document

About Data Management & Warehousing Data Management & Warehousing is a specialist consultancy in data warehousing based in Wokingham, Berkshire in the United Kingdom. Founded in 1995 by David M Walker, our consultants have worked for major corporations around the world including the US, Europe, Africa and the Middle East. Our clients are invariably large organisations with a pressing need for business intelligence. We have worked in many industry sectors but have specialists in Telco’s, manufacturing, retail, financial and transport as well as technical expertise in many of the leading technologies. For further information visit our website at: http://www.datamgmt.com

1

Agile Software Development, Cockburn, A., Addison-Wesley, 2002

© 2007 Data Management & Warehousing

Page 3

White Paper - Data Warehouse Governance

Introduction Data warehouse governance defines the model the organisation will use to ensure optimal use and re-use of the data warehouse and enforcement of corporate policies (e.g. business design, technical design and application security). Governance structures must be in place as early as possible, ideally before the project starts and before development begins. They are implemented in parallel with the initial development to ensure that they work together from the outset. A data warehouse is, by definition, a system that interacts with large parts of the organisation. At a technical level this involves the number of systems feeding data to and being fed data from the data warehouse. At a human level it relates to the people who interact with the system. These include those querying the system, those operating the technical environment and those directing the business who may not use the system directly but will mandate that others do so. The published literature on the subject falls into three categories: • • •

Home grown – what a specific project did within a specific environment Vendor specific – what to do when using the vendors tool set System Integrator proprietary – how an SI say they will do it if they are given the contract

The problem is that the data warehouse interacts with and across so much of the organisation. A programme must deploy standards and processes that integrate into the organisations’ existing policies and procedures. This paper looks at the areas that must be covered by governance and discusses some of the standards, options and issues. These ideas should be incorporated into the governance of the data warehouse.

© 2007 Data Management & Warehousing

Page 4

White Paper - Data Warehouse Governance

What is governance? In the introduction governance is defined as ‘the model the organisation will use to ensure optimal use and re-use of the data warehouse and enforcement of corporate policies’, but what does it mean? Data Warehouses are not a single short-term project. They are long-term programmes of work that includes a number of projects and a significant amount of maintenance. The decision to deploy a data warehouse commits the business to a programme that needs to be controlled to ensure that it provides business benefit and value for money. The role of governance is to provide the policies, processes and procedures necessary to ensure that the programme of work is effective. These must be clearly communicated to everyone involved. Governance covers the integrated management of the programme from its initial development, through production running to end-of-life close down. The governance structures must also be reviewed over the lifetime of the programme. This is to ensure that the governance is neither too little to allow effective control or so much such that programme work is stifled. Different organisations also have different requirements for the level of detail and breadth of scope that should be covered by Data Warehouse governance. For example, does governance imply coding standards? For some people this is far too much detail, for others it is essential that the governance of the programme ensures that specific standards are developed and that staff adhered to them. The key to designing a suitable level of governance within an organisation is in understanding the following: •

What events affect the data warehouse environment?

•

What is affected in the data warehouse by the impacting event?

•

What type of organisation is needed to support the data warehouse?

•

What are the policies and procedures needed to control the data warehouse?

From this it is possible to develop the standards, processes and communications that will both drive the programme forward and culturally fit the specific organisation. The processes, used in conjunction with the standards, are the way in which decisions are made. Good processes enabling quick, effective and well-communicated decisions and consequently effective management of the solution. The converse is also true as poor standards and processes lead to delays and unresolved issues that discredit the data warehouse and create significant cost or overheads that eventually destroy the programme.

© 2007 Data Management & Warehousing

Page 5

White Paper - Data Warehouse Governance

What affects the data warehouse? Data Management & Warehousing has identified five types of change that impact a data warehouse: •

Demand Demand is the change needed to the system to give the users what they want. This may include loading new data, possibly from new sources or restructuring existing information. It can also include adjusting system availability or the time at which data is loaded or any other user requirement.

•

External Change A data warehouse has a large number of systems feeding it. The systems operate in a complex and integrated environment. For example, a data warehouse might have three source systems and each source system is only allowed to perform upgrades and patches once a quarter. As a result the data warehouse is faced with twelve changes a year or one a month. In an ideal environment these changes will be developed and regression tested in line with the change in the source system. In practice most data warehouses have many more sources and more frequent changes. When these changes are not communicated they become issues. Managing external change prevents issues and therefore reduces ‘emergency fixes’ and ‘tactical solutions’

•

Maintenance All computer system have to manage routine maintenance issues such as software patches, upgrades to software or hardware, special backups for year end, etc. It is easy to ignore upgrades - the system is currently working so why bother? There comes a point when the software or hardware vendor moves the product out of support. At that point any system failure can have critical consequences. Routine maintenance should be regarded as a critical part of “business as usual” and allowance made for its impact.

•

Risks Risks to the data warehouse can be identified by those in control. These risks need to be addressed before they affect the system. Over recent years typical risks have included changes in regulatory policies, Year 2000 issues, currency changes (e.g. 2 3 revaluations, moving to the Euro or removing trailing zeros ). There are also risks from corporate mergers and acquisitions and from the bankruptcy of other companies that provide data to supplement the data warehouse (e.g. with market share information). These risks will create significant work with tight deadlines.

2

Currently 13 countries are in the Euro zone (http://en.wikipedia.org/wiki/Eurozone#Official_members) 3 49 countries including Turkey, Greece, Israel, Brazil and Argentina (http://www.allaboutturkey.com/ytl.htm)

© 2007 Data Management & Warehousing

Page 6

White Paper - Data Warehouse Governance

•

Issues In an ideal world all of the above would ensure that the data warehouse would never have any issues. The reality is a continuous flow of low-level problems and occasionally critical issues that need to be handled. In some ways this can be a measure of the success of the solution that people care enough to report issues and 4 need support. There are cost and resource implications in handling these issues.

What is affected within the data warehouse?5 Once a data warehouse development has begun these areas can be affected: •

Requirements Catalogue All data warehouse projects should have a comprehensive catalogue of requirements. These will be developed at the start of the system and maintained when new or changed requirements emerge. Requirements are not the same as the functionality of the system. This is because not all requirements may be met, however they provide a catalogue of work that the business needs to be delivered. This also provides a check that new demands have not already been met by another means.

•

Technical Infrastructure The technical infrastructure is the hardware, software and network used to build the system. It can be affected by many different factors including upgrades for better performance, platform re-hosting or a re-architecting exercise (e.g. moving from desktop to web-based clients)

•

Configuration Management Configuration management covers all aspect of managing the developed code and any system configuration files that exist outside the code. Suitable tools must be used to ensure that the people responsible for the maintenance of the system can look at all the different versions of code over time. This enables the deployment of new code. It also can help when there is a need to roll back changes, for example reverting to older code after a failed upgrade or to read an old format source file. It is also important that the code can be related to data models for the data warehouse and to the code versions and data models of the external systems that either supply or use data from the data warehouse.

4

One Data Management & Warehousing client organisation with 2000 users and a 20Tb system raised 10,000 issues in one year post implementation – or about 40 per day 5 This section refers to various documents such as requirements etc. Depending on who has developed the system they will have their own names and templates for these documents. For consistency this document refers to the templates and tools used in the Data Management & Warehousing (http://www.datamgmt.com) Documentation Roadmap which can be found at: http://www.datamgmt.com/index.php?module=article&view=77

© 2007 Data Management & Warehousing

Page 7

White Paper - Data Warehouse Governance

•

Test Management Test management is about asking questions to ensure that the impact of any change is well understood and that preventable errors are avoided. Questions like: o o o o o o o o

•

After a change is made, how is it tested? Is there sufficient data to be considered a representative sample? Is the whole system regression tested, or are changes isolated and therefore separately testable? How can changes or testing be modified so that tests can be isolated? How long does it take to test a change and will this impact emergency fixes to the system? Does the change affect the performance or the data quality of the solution? Can the figures be calculated by an alternative independent method to ensure that the numbers produced are correct? Does the project have sufficient resources (e.g. size of machine, people, time, etc.) to perform the tests?

Data Stewardship Data stewardship is about understanding who is responsible for the data within the system. Again the role is about asking questions: o o o o o o o

•

Who decides which data quality rules should be applied? Where is data cleansed? Who is responsible for cleaning the data? Can systems or processes be changed to improve the quality of the data before it is sent to the data warehouse? Is data cleansed in the source system and if so is this work also done on historical data? What is the impact if historical data that is already in the data warehouse is cleansed in the source? What is the relationship between time and data e.g. is 95% of the data available after one day good enough or must 100% of the data be there before it can be used?

Service Level Agreement The service level agreement defines much about the performance and availability of the system. It will include the times set aside for tasks such as backups and maintenance. It also defines the times that the system will be available to the users. It should also include agreements on how to respond in case of emergencies and how to manage systems failures. The Service Level Agreement also defines how any change that affects the system (either positively or negatively) needs to be communicated to the user community.

•

Support Model The support model describes the processes and escalation paths for any user support request. This model must be updated to reflect any changes to the system or any changes in the organisation. The support model should be modified to improve the responsiveness of support by understanding the frequently asked questions and improving the response time explicitly on these items.

© 2007 Data Management & Warehousing

Page 8

White Paper - Data Warehouse Governance

•

Training Plans Users and operators will need to be trained to use the system when it is first deployed. There is need for on-going training as the system changes, when new staff arrive or when parts of the solution are either commissioned or de-commissioned. Users will also need refresher courses to reduce the number and cost of calls to the support desk.

•

Schedules The system will have an operational schedule that will determine not only when users can be on the system (also covered in part by the service level agreement). The schedule defines which jobs to extract, transform or load data can be run, when they are run and the dependencies between them.

•

Monitoring The system must be monitored and alerts directed to some function that can prioritise them, act and if necessary escalate appropriately. The process by which this is managed and its efficiency is critical to providing a viable service to the users. Monitoring also ensures that service level agreements are met. Analysing the system events allows technical support to improve the system in the same way that analysing support calls assists the support team improve the support. The analysis must look for ways in which to bring about significant improvement to the system by either changing the system or improving the response to frequently occurring events.

How often will change occur?

1: Infrequently

External Change

Maintenance

Risks

Issues

Requirements Catalogue Technical Infrastructure Configuration Management Test Management Data Stewardship Service Level Agreement Support Model Training Plans Schedules Monitoring

Demand

The ten areas affected by change and the five types of change described above result in the table below along with the likely frequency of any given type of change on a specific area of the data warehouse:

3 2 1 2 2 2 1 1 2 1

1 1 3 3 3 2 1 1 3 3

1 2 3 3 1 2 2 1 2 3

2 1 1 1 1 2 1 1 1 1

1 1 3 3 3 1 3 3 2 1

2: Occasionally

3: Frequently

This table can be used throughout the life cycle of the programme to prioritise governance development to ensure that the most frequent changes are under the tightest controls.

© 2007 Data Management & Warehousing

Page 9

White Paper - Data Warehouse Governance

What organisational structures are needed? This paper has identified a large amount of potential change from different sources on different processes on a long-term programme. It is necessary to have sufficient organisational structure in place no control the changes and the communication of that change. The basic model organisation for this can be described as follows:

Figure 1 - Data Warehouse Organisational Structure

Executive Sponsor(s) An executive sponsor is a senior (executive) member of the organisation with an active interest and an understanding of the business uses of the data warehouse. A system of this magnitude and with such cross-functional reach needs to be sponsored at the very highest level. It is likely that the executive sponsor for the solution as a whole will be committed to the idea of developing better strategic information. A number of executive sponsors for individual functional developments will assist this person. The sponsors of individual functional developments are responsible for the delivery of the benefits promised for any given development. The executive sponsor(s) also need to ensure that the steering committee is directing the development in line with the vision, strategic direction and priorities for the organisation.

© 2007 Data Management & Warehousing

Page 10

White Paper - Data Warehouse Governance

Steering Committee The steering committee is the hub of the on-going data warehouse solution from a management perspective. It is here that the development is aligned with the business objectives. Monitoring ensures that the programme is delivering the right projects at the right time and at fair value. By setting the principles and policies the steering committee can control the direction that the development goes in. This maintains an enterprise wide business perspective for the data warehouse. The steering committee is also the centre of communication. It takes input from the user forums and the certification committee as to what is needed. In return the committee manages the expectations of both the business and IT departments as to what is possible. The steering committee is most effective when it is empowered to make quick, effective, cross-functional decisions and communicate to all the stakeholders.

Programme Management Programme management is the co-ordinated management of a portfolio of projects to achieve a set of business objectives. It delivers the co-ordinated support, planning, prioritisation and monitoring of projects to meet changing business needs. To achieve the business objectives the programme manager defines a series of projects with quantifiable benefits that together will meet the long-term objectives of the organisation. These projects may run simultaneously or at least overlap with each other and they may share resources. Such projects might have some resources devoted to the project for a period of time. In addition projects will require a range of specialists available from the programme team whose services are used for shorter periods of time. Projects within the programme can also be linked. Delays within one project will then cause knock on effects in other projects due to logical links between tasks in both of them. Resource and timing conflict resolution is also an integral part of the function of programme management. The programme is usually reflected in the management structure with a programme manager to whom the project managers will report. The programme manager will be concerned with recruiting and maintaining their project management teams, allocation of key, shared resources and on the integration of the deliverables from each project into the overall programme. In this environment every project plays its part towards the organisation’s ultimate aims and objectives. Often, as projects are completed, this translates back into a revised set of corporate objectives.

© 2007 Data Management & Warehousing

Page 11

White Paper - Data Warehouse Governance

Implementation Teams The implementation teams are those that will actually do the work. They will consist of a team of people (either for some or the entire project) that will fulfil the following roles: • • • • • • • • • • •

Project Manager Technical Architect Systems Administrator Network Administrator Database Administrator Metadata Administrator Data Modeller ETL Developer Front End Tool / Report Developer Product Specialist Test Manager 6

Many organisations may have outsourced some or all of this functionality . The resources may have the skills and the time to fulfil more than one role, or for some projects more than one resource may be required to perform a given role. The management and resource levels must be determined by the project scope.

Exploitation Teams Whilst the implementation teams focus on building the system the exploitation team are trying to ensure that the business is extracting the most value from the solution. Exploitation teams work on the current version of the system to help the business use the current system and develop new requirements to exploit the system further. Typical roles for the teams will include: • • • • • • • •

Exploitation Team Leader Business Analyst Business Requirements Specialist Technical Author / Documentation Specialist Trainer End User Support Specialist Communications Specialist Helpdesk Support Specialist

As with the implementation teams the resource level is determined by the project scope.

User Forums The programme should also consider setting up a number of user forums that involve end users, subject matter specialists and staff from the exploitation teams. These forums are useful to allow various teams to express their issues and aspirations for the system and expand on the art of the possible. These forums should also appoint representatives on the steering committee to represent their perspective on the system.

6

Data Warehouses benefit from people with detailed knowledge of the subject and outsourced development to coding shops with no experience is often more costly than the initial perceived savings. When choosing resources Data Management & Warehousing recommend that organisations look at the skills of named individuals on the project

© 2007 Data Management & Warehousing

Page 12

White Paper - Data Warehouse Governance

Certification Committee A number of groups within the organisation will also assess the data warehouse to ensure that it is fit for purpose. These groups can either be consulted individually or brought together as a committee to advise the programme. They include: •

Audit Most organisations will have some internal audit function. The audit function will need to ensure that the system meets the required standards and has sufficient checks and balances especially if the system is used for any form or statutory or fiscal reporting. The internal audit function should also examine how the system is managed. Where there is no (required) internal audit function an individual or team representing those with a stake in the quality of the information should perform the role of auditor.

•

Regulatory Compliance Depending on the organisation and industry there may be a need to comply with requirements from government, local administration or industry regulators over the data that is held and who has access to that data. Current examples 7 include data protection law , financial management laws such as Sarbanes8 9 10 Oxley Act and Mifid and telecoms regulators such as Ofcom .

•

IT Strategy & Architecture Many large organisations have a team responsible for the strategy and architecture of all IT systems. This team ensures inter-operability, re-use and cost reduction of IT systems. The team will need to review and assess any technical infrastructure and changes to the solution that are proposed.

•

Security Security of information is a growing concern for many organisations. Either a security team, if one exists in the organisation, or a nominated individual should review the system periodically to ensure its security.

7

DPA Law website: http://www.dpalaw.info/ Sarbanes Oxley Act: http://en.wikipedia.org/wiki/Sarbanes-Oxley_Act 9 Mifid: http://www.fsa.gov.uk/Pages/About/What/International/EU/fsap/mifid/index.shtml 10 Ofcom website: http://www.ofcom.org.uk/ 8

© 2007 Data Management & Warehousing

Page 13

White Paper - Data Warehouse Governance

Project Development Methodologies There are four main methodologies for the development itself:

Waterfall The waterfall model has the following phases that are followed in order: • • • • • • •

Requirements specification Design Build Integration Testing Deployment Maintenance

To follow the waterfall model, one proceeds from one phase to the next in a purely sequential manner, starting each phase once the previous one has been completed. Phases of development in the waterfall model are thus discrete and there is no jumping back and forth or overlap between them, however there are practical variations on this model.

Iterative11 The basic idea behind iterative development is to develop a software system incrementally. This allows the developer to take advantage of what was being learned during the development of earlier, incremental, deliverable versions of the system. Learning comes from both the development and use of the system where possible. Key steps in the process are to start with a simple implementation of a subset of the software requirements. The system is then iteratively enhanced in a sequence of evolving versions until fully implemented. At each iteration design modifications are made and new functional capabilities are added. The process itself consists of the initialization step, the iteration step and the project control list. The initialization step creates a base version of the system. The goal for this initial implementation is to create a product to which the user can react. It should offer a sample of the key aspects of the problem and provide a solution that is simple enough to understand and implement easily. To guide the iteration process, a project control list is created that contains a record of all tasks that need to be performed. It includes such items as new features to be implemented and areas of redesign of the existing solution. The project control list is constantly being revised because of the analysis phase. The iteration step involves the redesign and implementation of a task from the project control list and the analysis of the current version of the system. The goal for the design and implementation of any iteration is to be simple, straightforward and modular, supporting redesign at that stage or as a task added to the project control list.

11

Description is an edited form of the text from Wikipedia: http://en.wikipedia.org/wiki/Iterative_development

© 2007 Data Management & Warehousing

Page 14

White Paper - Data Warehouse Governance

The code can represent the major source of documentation of the system. The analysis of an iteration is based upon user feedback and the programme analysis facilities available. It involves analysis of the structure, modularity, usability, reliability, efficiency and achievement of goals. The project control list is modified in light of the analysis results.

Agile12 Agile methods are a family of development processes that build on iterative development, not a single approach to software development. Agile evolved in the mid 1990s as part of a reaction against "heavyweight" methods such as the waterfall model of development that often became heavily regulated, regimented and micro-managed. The processes originating from this use of the waterfall model were seen as bureaucratic, slow, demeaning and inconsistent with the ways that software engineers 13 actually perform effective work. In 2001 seventeen prominent figures in the field of agile development (then called "light-weight methodologies") came together to discuss ways of creating software in a lighter, faster, more people-centric way. They created the Agile Manifesto and 14 accompanying agile principles: •

The highest priority is to satisfy the customer through early and continuous delivery of valuable software.

•

Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.

•

Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

•

Business people and developers must work together daily throughout the project.

•

Build projects around motivated individuals. Give them the environment and support they need and trust them to get the job done.

•

The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

•

Working software is the primary measure of progress.

•

Agile processes promote sustainable development. The sponsors, developers and users should be able to maintain a constant pace indefinitely.

•

Continuous attention to technical excellence and good design enhances agility.

•

Simplicity--the art of maximizing the amount of work not done--is essential.

•

The best architectures, requirements and designs emerge from self-organising teams.

•

At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behaviour accordingly

12

Taken from The Agile Manifesto (http://agilemanifesto.org/) Source: Wikipedia - http://en.wikipedia.org/wiki/Agile_software_development 14 Source: Wikipedia - http://en.wikipedia.org/wiki/Agile_software_development 13

© 2007 Data Management & Warehousing

Page 15

White Paper - Data Warehouse Governance

Cowboy Coding15 Cowboy coding is a form of software development method without an actual defined method – team members do whatever they feel is right. Typical cowboy coding will involve no initial definition of the purpose or scope of the project, no formal description 16 of the project and will often involve only one developer . The developer will often be working from his own idea of what the software should do. It is often characterised by a lack of any documentation for either the requirements of the project or the design of the software overall. Within a data warehouse programme it is strongly advised never to allow the cowboy methodology to appear, however any of the other three can be used. There are two deciding factors as to the approach that will be used. The first is the approach that is already being used within the organisation and this may dominate over all other factors. The second is the point in the lifecycle of the warehouse. In the early part of the programme projects using agile methodologies offer fast development and quick delivery for new applications built by expert teams that engage the users. Towards the end of the lifecycle a more waterfall like approach to maintenance and de-commissioning operations run by systems management teams is likely to be more successful for individual projects. This highlights the need for the governance itself to be adaptive to the changing environment. All formal methodologies (lightweight and heavyweight) can still lead to failure as the methodology team members attempt to operate within the social/political environments of the organisation. The probability of such a failure is related to the degree of processes inhibiting the user(s) from deviating from the organisation standard on one hand and the cost in terms of lost efficiency and lost creativity of implementing such processes on the other. Success is therefore reliant on the management choosing appropriate processes that find the correct balance between these two competing aspects.

15

Cowboy coding definition: Wikipedia: http://en.wikipedia.org/wiki/Cowboy_coding Cowboy coding can occur even within projects that are using more appropriate development methods. Warning signals that cowboy coding may be occurring include: • Secrecy about what a developer is working on. • The inability to describe the functionality of current development. • The tendency to do lots of quick fixes. • Working hard and not working clever, working without prioritisation. 16

© 2007 Data Management & Warehousing

Page 16

White Paper - Data Warehouse Governance

Best practices The following items are some of the key best practices that should be implemented:

Momentum Get the project going fast and then keep it moving by keeping the scope and deadlines for delivery very tight. This avoids analysis paralysis and ensures that there is space to take corrective actions if necessary.

Monitor and Measure17 Large long-term projects are difficult but if the management team cannot see what is happening then they do not know when things are going wrong. Ask questions about whether teams deliver on time, to budget and scope. Track the number of issues raised in development, test and production, monitor the service level agreements, design metrics for the quality of the code and the user satisfaction, etc. Much of this is built in to approaches such as agile.

Prioritise Everyone will want their component now, but it cannot all be done at once. Have a clear, transparent process for deciding the priories and then stick to them. One useful technique is to say that (provided the project uses small scopes) once started a phase cannot be stopped but that before the next phase is started all priorities can be reassessed. This forces low value priorities to the back but rapidly brings higher priorities to the fore. The steering committee should be responsible for these priorities and the project methodology should support the prioritisation process.

Light touch This document has covered many aspects of governance however the programme should implement the minimum necessary because otherwise the project will become about governance and not delivery. Review the processes regularly to add process where required and do not be afraid to remove un-necessary process.

Communication The data warehouse is a long-term project for which the perception will shift over time from all parts of the business. Clear communications will help people understand what is available for them, what the priorities are going forward and how to access the information. Publish these widely to encourage involvement and interaction with the system.

17

A useful book on this subject is “Controlling Software Projects: Management, Measurement and Estimates” by Tom DeMarco

© 2007 Data Management & Warehousing

Page 17

White Paper - Data Warehouse Governance

Standards Develop and enforce clear standards for all aspects of the warehouse such as naming conventions for code and documents, data ownership, data cleaning, access to data, 18 service levels, performance etc.

Track processes Ensure that there are clear formal processes for handling change requests, risks and issues. These are the aspects of the project that cause divergence from the baseline and therefore it is critical to know what is happening, when and why. It allows scarce resource to be focused on the correct actions. Use key design decision templates to ensure that design topics are not constantly revisited. Ensure that all change related processes have powerful change management routings to back them up.

Understand cost and value Define methods for determining the cost and value of any action, some apparently low cost (often called quick-win or tactical) solutions deliver little value and their true cost is disproportionately higher as they have to be backed out after a short life. Do not forget to include end of life cost in the calculations and make budget holders responsible for demonstrating the benefit that has been derived because of the effort.

Continuous Learning and Improvement Carry out stage point audits and post-mortems on all aspects of the system, use the lessons learnt from these to improve the processes to ensure that the next development will not make the same mistakes.

Pilot & Prototype Pilot and prototype high risk or complex aspects of the development using methodologies like ‘Agile’ but ensure that the pilots and prototypes do not go into production until the normal quality thresholds have been met.

Authority to act Ensure that the steering committee has the authority to act cross functionally using sufficient information to make good decisions. A steering committee that wants a change to a source system that has a critical effect on the data quality is ineffective if they are told that it will be put in a queue and should be delivered in two years time. In the mean time is it acceptable that the data warehouse will just have to make do with poor data? Conversely a lack of sufficient information might be making the same demand of a system that is about to be de-commissioned or for data for which the business user cannot articulate a use.

18

See the Data Management & Warehousing Documentation Roadmap at http://www.datamgmt.com

© 2007 Data Management & Warehousing

Page 18

White Paper - Data Warehouse Governance

Plan for the long term Ensure that the business and users understand that whilst aspects of the data warehouse will be available quickly the system is a long-term commitment. It will need funding and maintenance across its entire life span. During that lifetime the system will also grow in size and complexity, increasing the financial burden.

Governance and Success Governance is the control aspect of management and should promote a situation where senior management are in control of the situation much like the maestro conductor of a fine orchestra. They need to understand what is going on and where to go for the next action and how to respond to a changing situation. Sometimes the reality is that managers are working in a suppressed panic, not believing what people are telling them or what they themselves are communicating to their management. When managers know that people have to work outside the process to get things done and not knowing what is actually going on then governance has failed. Governance is therefore not the documentation, the processes or the formality, but is about developing a culture where understanding, discipline and skill are regarded as virtues in teams that have leaders with strong technical skills, initiative, communications skills and personal authority. Organisations that ‘buy the book’, be it from a bookshop or as a formal methodology from a vendor and follow it rigidly are guaranteed to achieve the lowest common denominator solution, one that checks all the boxes but underwhelms the management and disappoints the users. Those organisations that use governance and methodologies as enablers and allow systems to fulfil their potential by meeting the constantly changing requirements of the users succeed. Creating effective governance for an organisation requires imagination: “What we need is imagination, but imagination in a terrible strait-jacket. We have to find a new view of the world that has to agree with everything that is known, but disagree in its predictions somewhere .... And in that disagreement it must agree with nature. If you can find any other view of the world which agrees over the entire range where things have already been observed, but disagrees somewhere else, you have made a great discovery. ...A new 19 idea is extremely difficult to think of. It takes a fantastic imagination.”

19

Richard Feynman, The Character of Physical Law, 1965, Chapter 7, "Seeking New Laws.”

© 2007 Data Management & Warehousing

Page 19

White Paper - Data Warehouse Governance

Summary An organisation that is embarking on a data warehousing project is undertaking a long-term development and maintenance programme of a computer system. This system will cost a significant amount of money and should be well controlled. Governance defines the model that the organisation will use to ensure optimal use and re-use of the data warehouse. It will also enforce corporate policies (e.g. business design, technical design and application security) that ultimately derive value for money. This paper has identified five sources of change to the system: • • • • •

Demand External Change Maintenance Risks Issues

It has also looked at the aspects of the system that they will affect: • • • • • • • • • •

Requirements Catalogue Technical Infrastructure Configuration Management Test Management Data Stewardship Service Level Agreement Support Model Training Plans Schedules Monitoring

Using these it has been able to highlight standards and best practices that should be employed by the organisation to deliver effective governance. The paper has also highlighted the fact that because governance is about fitting into the existing organisation. It also highlights the fact that the system grows significantly over time. Consequently there is no single right solution for governance but instead there are ways of working and adapting over time that will ensure success.

© 2007 Data Management & Warehousing

Page 20

White Paper - Data Warehouse Governance

Appendices Appendix 1 - Programme or Project? This document has several times differentiated between programmes and projects. The 20 table below outlines the differing characteristics of the two:

Programmes: Address the entire business change Focus on strategic goals May have imprecise definition May have uncertain timing Evolve over a period of time to derive optimum benefit for the organisation Require much senior management attention, often including strategic and political debate across organisational boundaries Produce an overall improvement in the business that may be multi-faceted and not fully defined at the outset of the programme Require a manager who is highpowered, high-level, visionary, strategic, political, sales-oriented and works with people at the top and across the organisation

20

Projects: Deliver a specific change component Focus on tactical delivery Have a precise objective Are defined with a specific timeline and budget Try to avoid change to the defined scope in order to ensure delivery Require management communication primarily at an operational level concerning operational details Produce specific deliverables

pre-defined

Require a manager who pays attention to detail, has good team leadership, plans in detail, follows a disciplined approach and delivers the goods.

The ePMBook by Simon Wallace (http://www.epmbook.com/)

© 2007 Data Management & Warehousing

Page 21

White Paper - Data Warehouse Governance

Appendix 2 – The "Declaration of Interdependence" Throughout this document reference has been made to the agile development methodology. Because of the development of agile, a set of principles characterised by statements “We accomplish X by doing Y.” was developed for project managers. These principles are of value outside the agile methodology itself and should be used by project managers in any data warehouse programme. The “Declaration of Interdependence” for modern (agile/adaptive) (product/project) 21 management "We ... • • • •

• •

increase return on investment by making continuous flow of value our focus. deliver reliable results by engaging customers in frequent interactions and shared ownership. expect uncertainty and manage for it through iterations, anticipation and adaptation. unleash creativity and innovation by recognizing that individuals are the ultimate source of value and creating an environment where they can make a difference. boost performance through group accountability for results and shared responsibility for team effectiveness. improve effectiveness and reliability through situationally specific strategies, processes and practices."

21

2005 David Anderson, Sanjiv Augustine, Christopher Avery, Alistair Cockburn, Mike Cohn, Doug DeCarlo, Donna Fitzgerald, Jim Highsmith, Ole Jepsen, Lowell Lindstrom, Todd Little, Kent McDonald, Pollyanna Pixton, Preston Smith and Robert Wysocki: http://alistair.cockburn.us/index.php/The_declaration_of_interdependence_for_modern _management

© 2007 Data Management & Warehousing

Page 22

White Paper - Data Warehouse Governance

Appendix 3 - Team Values and Principles The author of this white paper used to work at Sequent Computer Systems. The company had a set of values and principles that the author feels have always proved useful to teams building data warehouses by creating a culture in which governance can work.

Our Values: •

Easy to Do Business With o The Customer ALWAYS Comes First o Do All We Can to Ensure Customer Success - Does Not Always Mean Saying Yes o Three Keys -- Listen, Consider and Act o Go the "Extra Mile" for the Customer

•

Profitability o Required for Our Success o Allows for Growth & Investments for Future o Everyone is Responsible for Profitability o Follow the "Spend Smart and Invest Smart" Concept

•

Teamwork o Strong & Balanced Teams o Put Group Goals & Interests Ahead of Personal Reward o Open & Honest Communication o You're Accountable for Your Commitments o Nobody Wins Unless the Team Wins

•

Quality o o o o o

Exceed the Customer Expectations Strive for Continuous Improvement in All You Do Know Who Your Customers Are Establish Clear Mutually Acceptable Expectations Managed Expectations, Consistently Achieved

•

People o o o o o

Assimilating & Integrating New Members is Critical Open, Honest & Timely Communication Treat Each Other With Respect Take Time to Say "Thank You" for a Good Job Strive for Win-Win Relationships

Our Principles: • • • • • • • • • •

Act With Honesty & Uncompromising Integrity Take Responsibility for Our Customers' Success Strive to be the Best Have a "Can Do" Attitude Respect Each Other Exhibit Team Pride Take Calculated Risks Be Active Community Citizens Urgently Do the Right Thing Make Consultative Decisions

© 2007 Data Management & Warehousing

Page 23

White Paper - Data Warehouse Governance

References The section below represents some useful resources for those considering building a data warehouse solution.

Web resources Organisation Data Management & Warehousing

Website http://www.datamgmt.com

DM Review – Data Warehouse Project Management The Data Warehouse Institute – Data Warehouse Governance Data warehouse governance – Best practices at Blue Cross & Blue Shield of North Carolina Data Warehouse Project Management The Seven Pillars of Data Warehouse Governance The ePM Book Software Testing Resources From Here to Agility: The Physics of Speed Agile Manifesto Agile Alliance e-Programme Data Protection Law Alistair Cockburn

http://www.dmreview.com/

Wikipedia

http://en.wikipedia.com

http://www.tdwi.org/ http://www.amazon.com/ http://www.amazon.com/ http://www.itweb.co.za/ http://www.epmbook.com/ http://www.testingfaqs.org/ http://www.stickyminds.com/ http://agilemanifesto.org/ http://www.agilealliance.org/ http://www.e-programme.com/ http://www.dpalaw.info/ http://alistair.cockburn.us/

Acknowledgements The author and Data Management & Warehousing would like to acknowledge and thank all the clients and colleagues who have taken the time to review and comment on this document prior to publication. Special thanks to Ron Ballard for the most detailed of reviews and proof reading.

Copyright © 2007 Data Management & Warehousing. All rights reserved. Reproduction not permitted without written authorisation. References to other companies and their products use trademarks owned by the respective companies and are for reference purposes only.

© 2007 Data Management & Warehousing

Page 24