Etis09 - Black Swans And White Elephants - Roi In Business Intelligence
This document was uploaded by user and they confirmed that they have the permission to share it. If you are author or own the copyright of this book, please report to us by using this DMCA report form. Report DMCA
Overview
Download & View Etis09 - Black Swans And White Elephants - Roi In Business Intelligence as PDF for free.
More details
- Words: 2,361
- Pages: 25
Black Swans & White Elephants Return On Investment In Business Intelligence
David M Walker Data Management & Warehousing ETIS ‒ Istanbul ‒ Oct 09
Return on Investment A performance measure used to evaluate the efficiency of an investment or to compare the efficiency of a number of different investments. To calculate ROI, the benefit (return) of an investment is divided by the cost of the investment; the result is expressed as a percentage or a ratio. The return on investment formula: ROI = (Gain from Investment – Cost of Investment) Cost of Investment Return on investment is a very popular metric because of its versatility and simplicity. That is, if an investment does not have a positive ROI, or if there are other opportunities with a higher ROI, then the investment should not be undertaken.
So where are the ROI Metrics? Clearly valuing BI is not an exact science and often comes down to mindset. Comparing BI to a college education: It may be expensive and time-‐consuming, but there are many less tangible benefits, like increased earning power and overall improved quality of life, which come years later. It's not easy to persuade someone to go to college based on a purely financial or numbers game, and the same thing goes for BI. You just have to believe that BI is absolutely essential for you as an organization to invest, that this is a fundamental core competency that you have to have. Bill Hostmann, Vice President & Analyst, Gartner, March 2008
What is Good BI? Good BI is the fusion of the right information, the right time, the right format, and the right human and/or system resources. If we wish to improve BI, we ask these questions: Do business users have the information needed, when they need it, to make decisions? Do those people have the expertise, training and mindset to use that information in the best way for the good of the organization? Are they doing their job better because of the information being delivered? How much difference does that information make to them? Dorothy Miller, BI Metrics, Feb 2009
The best approach to evaluating a BI solution is a technical review combined with a business user perception survey
Proposition No organisation has ever delivered the ROI originally promised: The business has moved on in the time it takes to develop the (original) solution meaning that the value has diminished The biggest gains and losses have come from Black Swans and not planned events The biggest costs have come from White Elephants and the failure to recognise where the costs are hidden
Some (mainly those highly committed to BI) have far exceeded the promised ROI but most fail to achieve it Even if the development costs are on budget the on-‐going OPEX costs are hidden and far out-‐weigh the the benefit
Black Swans High-‐impact, hard-‐to-‐predict, and rare events beyond the realm of normal expectations The term Black Swan comes from the 17th century European assumption that 'All swans are white'. In that context, a black swan was a symbol of something that was impossible or could not exist. In the 18th Century, the discovery of black swans in Western Australia metamorphosed the term to connote that a perceived impossibility may actually come to pass.
Black Swans: Product Master Data Data Warehouse requirements delivered Data Warehouse analysis identifies no management of product master data and no product master list Project risk raised and escalated to executive level BI Project postponed and staff re-‐tasked to define and deliver Corporate Product MDM Implements ERP based MDM solution and reduces product catalogue by 90% Directly affects corporate bottom line Restarts BI Project six months later with massively improved data quality and data integration
Black Swans: Financial Reporting Telco in DotCom boom consistently overstates earnings and subscriber growth to boost share price Data Warehouse project identifies issue and produces a set of correct numbers Business refuses to adjust statutory reporting as they have been using the ‘Excel’ numbers, it would be embarrassing to restate the figures and the new ones are just too different Business collapses and is taken over, national newspapers report that ‘internal systems’ had the correct figures The failure to use real data was highlighted by auditors as a significant factor in the poor decisions made by management
Black Swans: The Salesman Fraud Supplier of confectionary to a major supermarket chain First Release of Data Warehouse CEO was un-‐happy with the numbers from the new system He knew that the supermarket was their biggest customer
IT defended the Data Warehouse Every last loading step and transformation had been tested & checked
A Salesman’s commission doubled for the first quarter on new accounts before dropping to standard rate in subsequent quarters The Salesman responsible for the supermarket chain created a new customer each quarter Corporate revenues affected enough for an exchange filing
Black Swans: External Collapse Sudden market collapse Many customers suddenly change profile Looking for value – lower spend
Be Pro-‐active An opportunity to demonstrate the true value of BI (It might save your job too!)
White Elephants A white elephant is a valuable possession of which its owner cannot dispose and whose cost (particularly cost of upkeep) is out of proportion to its usefulness. The term derives from the sacred white elephants kept by Southeast Asian rulers. To possess a white elephant was (and still is) regarded as a sign that the monarch was ruling with justice and power, and that the kingdom was blessed with peace and prosperity. The animals were considered sacred and laws protected them from labour, receiving a gift of a white elephant from a monarch was both a blessing and a curse: a blessing because the animal was sacred and a sign of the monarch's favour, and a curse because the animal had to be kept and could not be put to practical use to offset the cost of maintaining it.
White Elephants Convention over Configuration Bank with existing data warehouse Extension to support Balanced Scorecard Wanted to use a different data modelling technique in the same data warehouse for the new elements Justified as a management decision because “we can’t afford to re-‐ develop the old model but we want the best technique for the new parts” Resultant model could simply not be used ‘On Rails’ Values “DRY -‐ Don’t repeat yourself” & “Convention over configuration” Use each component in a consistent way Aids Understanding and Reduces Maintenance
Use each component for the purpose it was intended
White Elephants: Not evaluating technology Second Generation Data Warehouse Build Existing platform major RDBMS Recommended solution: Componentised architecture with high performance database engine and simple ETL architecture developed by small in house team Chosen solution: Existing RDBMS vendor with tightly coupled ETL tool from a different vendor and major SI doing development on site – “because that’s the way we do it here and management won’t consider anything else” Current issues: Considered too costly and is being delivered late Already having to review technology choices because of performance issues before roll-‐out complete SI has left taking all the knowledge with them
White Elephants: Making an ETL White Elephant Existing data warehouse in personal finance company with a ETL load built from SQL scripts and a shell script and tight control via SVN in production for 3 years New IT Director commissions review by vendor that recommends the vendors ETL tool for US$400K Six Months, 4 Consultants @ US$1.5K per day (a total spend >US$600K) later … All ETL converted Runs 20% slower Other BI developments delayed whilst changes made
Deemed TOO EXPENSIVE to revert back Eventually reverted back 12 months later when vendor quoted for upgrade to system
White Elephants: Breaking an ETL White Elephant Interactive Media corporation moves from traditional RDBMS to commodity appliance technology Data Volumes doubling every six months Internal Review
ETL tools can’t handle load ETL experts too expensive for long term engagements SQL scripts can be developed by more resources SQL scripts allow more work to be done in the appliance SQL scripts allow more agile approach SQL scripts allow tighter change management
New architecture componentised, commodity based with simple script engine reduces costs by 90% and increased productivity by 100%
White Elephants: The OPEX trap BI Projects command headline CAPEX budget figures BI Projects are widely publicised during development Most of the money is spent in OPEX It comes from User Support, Training, Changes and Operational Support It is normally 2x-‐3x and often 5x more than the cost of the build over the lifetime of the system It is hidden – spread over multiple budgets in such a way that it is hard to evaluate and often ignored
Any action (especially tactical actions) in the build/test stage that will result in increased OPEX is deadly to ROI
White Elephants The New Reporting Tool Data Warehouse Second Generation Build New project – new interface concept Current system users were ‘un-‐happy’ with the existing tool The reality was largely issues with the data model and data quality of the current system
Replaced with an equivalent reporting tool
The cost
Retraining of over 2000 staff Failure to de-‐commission old reporting tool 50% too many licences bought for the new tool User dissatisfaction with the (new) reporting tool
White Elephants: Reporting Tool Penetration Business intelligence vendors like to talk up a 20/80 split: only 20 percent of users are actually consuming BI technologies; the remaining 80 percent are disenfranchised. Reasons Security limitations Slow query performance Internal politics and (more precisely) internal power struggles
Nigel Pendse of BI Survey shows just over 8 percent of employees are actually using BI tools. Even in industries that have aggressively adopted BI tools (e.g., wholesale, banking, and retail), usage barely exceeds 11 percent. Hardware cost Data availability Software cost Software was too hard to use User scalability
White Elephants: Not testing the delivery
Any delivery has stages
Every stage can have tests associated with it
Fixing something 1 stage later = 2x more expensive Fixing something 2 stages later = 4x more expensive, etc.
Many more small tests performed early will save huge amounts of time and money
Requirements: Mind Experiment Method Analysis: Cross-‐checking with other sources Design: Algorithmic Checks Build: Unit Tests / Boundary Checking Testing: System Tests / Integration Test
Test early and often
Requirements / Analysis / Design / Build / Test/ Deploy etc.
Have to handle short term slippages If you haven’t got time to test then you are planning to spend OPEX
Massive user perception impact as information is right first time
White Elephants Garbage In :: Garbage Out ‘On two occasions I have been asked "Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?" … I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.’ – Charles Babbage 1791-‐1871
Data Quality is more critical than ever A failure to address Data Quality at every stage will always lead to additional costs Plan for your BI project to spin off dozens of data quality projects and continue to do so throughout its life Data Quality issues drive users away which directly increases cost of ownership and reduces ROI
White Elephants: De-‐commission! Data Warehouse Component Delivered De-‐commission the previous reporting system NO -‐ REALLY -‐ TURN IT OFF !
If it is left on: It costs money to run (and it is all invisible OPEX) Users will compare results and distrust the new system even if it can be proved to be more correct than the old system Users will continue to use the old one because it is familiar and no-‐one likes change Users that do not migrate but have been trained on the new system will have to be re-‐trained when they start to use it
The Solution The proposition suggests that doing BI has not delivered the expected ROI for most organisations. Does this mean that organisations should stop developing BI solutions? Categorically NO – BI should be hugely worthwhile But remember you can only succeed if:
Your organisation fully embraces Business Intelligence You expect and embrace Black Swans You avoid and mitigate White Elephants You make your own luck
How to be lucky Lucky people frequently happen upon chance opportunities
There are positive black swans in every organization, it is just a matter of identifying them as they occur
Lucky people listen to their hunches
Members of your team will have insights beyond their remit based on their experience, identify these people and exploit their insights
Lucky people persevere in the face of failure
Every project will face set-‐backs -‐ plan for them
Lucky people have the ability to turn bad luck into good fortune
Every project will face set-‐backs -‐ embrace them as an opportunity and change your project/remit and reset your goals
Based on work by Prof. Richard Wiseman, University of Hertfordshire, 2003
Improving ROI Black swans (handled positively) massively increase Gain from investment White elephants (eliminated) significantly reduce the Cost of the Investment Whatever the CAPEX investment of the project
the OPEX will be significantly more – design for this
Black Swans & White Elephants Return On Investment In Business Intelligence
THANK YOU
David M Walker Data Management & Warehousing ETIS ‒ Istanbul ‒ Oct 09
Return on Investment A performance measure used to evaluate the efficiency of an investment or to compare the efficiency of a number of different investments. To calculate ROI, the benefit (return) of an investment is divided by the cost of the investment; the result is expressed as a percentage or a ratio. The return on investment formula: ROI = (Gain from Investment – Cost of Investment) Cost of Investment Return on investment is a very popular metric because of its versatility and simplicity. That is, if an investment does not have a positive ROI, or if there are other opportunities with a higher ROI, then the investment should not be undertaken.
So where are the ROI Metrics? Clearly valuing BI is not an exact science and often comes down to mindset. Comparing BI to a college education: It may be expensive and time-‐consuming, but there are many less tangible benefits, like increased earning power and overall improved quality of life, which come years later. It's not easy to persuade someone to go to college based on a purely financial or numbers game, and the same thing goes for BI. You just have to believe that BI is absolutely essential for you as an organization to invest, that this is a fundamental core competency that you have to have. Bill Hostmann, Vice President & Analyst, Gartner, March 2008
What is Good BI? Good BI is the fusion of the right information, the right time, the right format, and the right human and/or system resources. If we wish to improve BI, we ask these questions: Do business users have the information needed, when they need it, to make decisions? Do those people have the expertise, training and mindset to use that information in the best way for the good of the organization? Are they doing their job better because of the information being delivered? How much difference does that information make to them? Dorothy Miller, BI Metrics, Feb 2009
The best approach to evaluating a BI solution is a technical review combined with a business user perception survey
Proposition No organisation has ever delivered the ROI originally promised: The business has moved on in the time it takes to develop the (original) solution meaning that the value has diminished The biggest gains and losses have come from Black Swans and not planned events The biggest costs have come from White Elephants and the failure to recognise where the costs are hidden
Some (mainly those highly committed to BI) have far exceeded the promised ROI but most fail to achieve it Even if the development costs are on budget the on-‐going OPEX costs are hidden and far out-‐weigh the the benefit
Black Swans High-‐impact, hard-‐to-‐predict, and rare events beyond the realm of normal expectations The term Black Swan comes from the 17th century European assumption that 'All swans are white'. In that context, a black swan was a symbol of something that was impossible or could not exist. In the 18th Century, the discovery of black swans in Western Australia metamorphosed the term to connote that a perceived impossibility may actually come to pass.
Black Swans: Product Master Data Data Warehouse requirements delivered Data Warehouse analysis identifies no management of product master data and no product master list Project risk raised and escalated to executive level BI Project postponed and staff re-‐tasked to define and deliver Corporate Product MDM Implements ERP based MDM solution and reduces product catalogue by 90% Directly affects corporate bottom line Restarts BI Project six months later with massively improved data quality and data integration
Black Swans: Financial Reporting Telco in DotCom boom consistently overstates earnings and subscriber growth to boost share price Data Warehouse project identifies issue and produces a set of correct numbers Business refuses to adjust statutory reporting as they have been using the ‘Excel’ numbers, it would be embarrassing to restate the figures and the new ones are just too different Business collapses and is taken over, national newspapers report that ‘internal systems’ had the correct figures The failure to use real data was highlighted by auditors as a significant factor in the poor decisions made by management
Black Swans: The Salesman Fraud Supplier of confectionary to a major supermarket chain First Release of Data Warehouse CEO was un-‐happy with the numbers from the new system He knew that the supermarket was their biggest customer
IT defended the Data Warehouse Every last loading step and transformation had been tested & checked
A Salesman’s commission doubled for the first quarter on new accounts before dropping to standard rate in subsequent quarters The Salesman responsible for the supermarket chain created a new customer each quarter Corporate revenues affected enough for an exchange filing
Black Swans: External Collapse Sudden market collapse Many customers suddenly change profile Looking for value – lower spend
Be Pro-‐active An opportunity to demonstrate the true value of BI (It might save your job too!)
White Elephants A white elephant is a valuable possession of which its owner cannot dispose and whose cost (particularly cost of upkeep) is out of proportion to its usefulness. The term derives from the sacred white elephants kept by Southeast Asian rulers. To possess a white elephant was (and still is) regarded as a sign that the monarch was ruling with justice and power, and that the kingdom was blessed with peace and prosperity. The animals were considered sacred and laws protected them from labour, receiving a gift of a white elephant from a monarch was both a blessing and a curse: a blessing because the animal was sacred and a sign of the monarch's favour, and a curse because the animal had to be kept and could not be put to practical use to offset the cost of maintaining it.
White Elephants Convention over Configuration Bank with existing data warehouse Extension to support Balanced Scorecard Wanted to use a different data modelling technique in the same data warehouse for the new elements Justified as a management decision because “we can’t afford to re-‐ develop the old model but we want the best technique for the new parts” Resultant model could simply not be used ‘On Rails’ Values “DRY -‐ Don’t repeat yourself” & “Convention over configuration” Use each component in a consistent way Aids Understanding and Reduces Maintenance
Use each component for the purpose it was intended
White Elephants: Not evaluating technology Second Generation Data Warehouse Build Existing platform major RDBMS Recommended solution: Componentised architecture with high performance database engine and simple ETL architecture developed by small in house team Chosen solution: Existing RDBMS vendor with tightly coupled ETL tool from a different vendor and major SI doing development on site – “because that’s the way we do it here and management won’t consider anything else” Current issues: Considered too costly and is being delivered late Already having to review technology choices because of performance issues before roll-‐out complete SI has left taking all the knowledge with them
White Elephants: Making an ETL White Elephant Existing data warehouse in personal finance company with a ETL load built from SQL scripts and a shell script and tight control via SVN in production for 3 years New IT Director commissions review by vendor that recommends the vendors ETL tool for US$400K Six Months, 4 Consultants @ US$1.5K per day (a total spend >US$600K) later … All ETL converted Runs 20% slower Other BI developments delayed whilst changes made
Deemed TOO EXPENSIVE to revert back Eventually reverted back 12 months later when vendor quoted for upgrade to system
White Elephants: Breaking an ETL White Elephant Interactive Media corporation moves from traditional RDBMS to commodity appliance technology Data Volumes doubling every six months Internal Review
ETL tools can’t handle load ETL experts too expensive for long term engagements SQL scripts can be developed by more resources SQL scripts allow more work to be done in the appliance SQL scripts allow more agile approach SQL scripts allow tighter change management
New architecture componentised, commodity based with simple script engine reduces costs by 90% and increased productivity by 100%
White Elephants: The OPEX trap BI Projects command headline CAPEX budget figures BI Projects are widely publicised during development Most of the money is spent in OPEX It comes from User Support, Training, Changes and Operational Support It is normally 2x-‐3x and often 5x more than the cost of the build over the lifetime of the system It is hidden – spread over multiple budgets in such a way that it is hard to evaluate and often ignored
Any action (especially tactical actions) in the build/test stage that will result in increased OPEX is deadly to ROI
White Elephants The New Reporting Tool Data Warehouse Second Generation Build New project – new interface concept Current system users were ‘un-‐happy’ with the existing tool The reality was largely issues with the data model and data quality of the current system
Replaced with an equivalent reporting tool
The cost
Retraining of over 2000 staff Failure to de-‐commission old reporting tool 50% too many licences bought for the new tool User dissatisfaction with the (new) reporting tool
White Elephants: Reporting Tool Penetration Business intelligence vendors like to talk up a 20/80 split: only 20 percent of users are actually consuming BI technologies; the remaining 80 percent are disenfranchised. Reasons Security limitations Slow query performance Internal politics and (more precisely) internal power struggles
Nigel Pendse of BI Survey shows just over 8 percent of employees are actually using BI tools. Even in industries that have aggressively adopted BI tools (e.g., wholesale, banking, and retail), usage barely exceeds 11 percent. Hardware cost Data availability Software cost Software was too hard to use User scalability
White Elephants: Not testing the delivery
Any delivery has stages
Every stage can have tests associated with it
Fixing something 1 stage later = 2x more expensive Fixing something 2 stages later = 4x more expensive, etc.
Many more small tests performed early will save huge amounts of time and money
Requirements: Mind Experiment Method Analysis: Cross-‐checking with other sources Design: Algorithmic Checks Build: Unit Tests / Boundary Checking Testing: System Tests / Integration Test
Test early and often
Requirements / Analysis / Design / Build / Test/ Deploy etc.
Have to handle short term slippages If you haven’t got time to test then you are planning to spend OPEX
Massive user perception impact as information is right first time
White Elephants Garbage In :: Garbage Out ‘On two occasions I have been asked "Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?" … I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.’ – Charles Babbage 1791-‐1871
Data Quality is more critical than ever A failure to address Data Quality at every stage will always lead to additional costs Plan for your BI project to spin off dozens of data quality projects and continue to do so throughout its life Data Quality issues drive users away which directly increases cost of ownership and reduces ROI
White Elephants: De-‐commission! Data Warehouse Component Delivered De-‐commission the previous reporting system NO -‐ REALLY -‐ TURN IT OFF !
If it is left on: It costs money to run (and it is all invisible OPEX) Users will compare results and distrust the new system even if it can be proved to be more correct than the old system Users will continue to use the old one because it is familiar and no-‐one likes change Users that do not migrate but have been trained on the new system will have to be re-‐trained when they start to use it
The Solution The proposition suggests that doing BI has not delivered the expected ROI for most organisations. Does this mean that organisations should stop developing BI solutions? Categorically NO – BI should be hugely worthwhile But remember you can only succeed if:
Your organisation fully embraces Business Intelligence You expect and embrace Black Swans You avoid and mitigate White Elephants You make your own luck
How to be lucky Lucky people frequently happen upon chance opportunities
There are positive black swans in every organization, it is just a matter of identifying them as they occur
Lucky people listen to their hunches
Members of your team will have insights beyond their remit based on their experience, identify these people and exploit their insights
Lucky people persevere in the face of failure
Every project will face set-‐backs -‐ plan for them
Lucky people have the ability to turn bad luck into good fortune
Every project will face set-‐backs -‐ embrace them as an opportunity and change your project/remit and reset your goals
Based on work by Prof. Richard Wiseman, University of Hertfordshire, 2003
Improving ROI Black swans (handled positively) massively increase Gain from investment White elephants (eliminated) significantly reduce the Cost of the Investment Whatever the CAPEX investment of the project
the OPEX will be significantly more – design for this
Black Swans & White Elephants Return On Investment In Business Intelligence
THANK YOU
Related Documents
Black White In Colour
November 2019 22
Black White In Colour
November 2019 21
# 72 Black And White
July 2020 15
72 Black And White
November 2019 20
72-black And White
November 2019 23More Documents from ""
