Fallacy Of Iso-cmmi Certifications

Fallacy of Certifications Murali Chemuturi Introduction The definition of quality (Fitness for Use) itself leaves a bit to be desired. It leaves the terms “Fitness” and “Use” open to interpretation. Thus we see a plethora of products claiming to be of “Quality” without even adding adjectives like “good”, or “best”. Still the term “quality” itself implies great quality in the minds of people in general. In manufacturing and other engineering fields, industry associations, Governments, and Armed Forces brought out standards and conforming to those standards ensured “Good Quality” deliverables. When it came to software, the picture of standardization is not that bright. True, IEEE (Institution of Electrical and Electronic Engineers, USA) brought out some standards and termed them as Software Engineering Standards – not “Software (Quality) Standards”. These standards are more in the nature of guidelines rather than standards in the strict sense of the word “standard”. They are open to interpretation and adaptation. Somehow, it came to be believed that adhering to a defined process would ensure software quality and ISO (International Standards Organization) brought out 9000 series of standards. This is followed by SEI (Software Engineering Institute, Carnegie Mellon University, USA) brought out CMM and CMMI (Capabilities Maturity Model Integrated). CMMI itself is twofold – CMMI for Development and CMMI for Acquisition. Many organizations wishing to outsource their development work started insisting on certification, especially of CMMI. A certificate enables opening of doors for bidding and lack of it, closed the doors. The vendors started getting the coveted certificate either by hook or by crook, just to be in the race. And understandably certification organizations sprouted like mushrooms. Now plenty of development organizations got certified – at least on ISO and in many cases both ISO and CMMI. We also have TMM (Testing Maturity Model), People Capability Maturity Model (PCMM), Software Engineering Capability Maturity Model (SE-CMM), IT Service CMM and so on Lee Copeland lists 34 maturity models in his article “The Maturity Maturity Model (M3)” on Stickyminds.com web site. ISO 9000 series of standards started out focusing on Quality with QMS (Quality Management System) as the main document and Quality Policy as the backbone for the organizational processes. But perhaps due to pressure from industry, these process standards diluted into organizational processes, shifting the focus from quality to organizational vision, and goals etc. CMMI goes one step further stating that the process should be to achieve business goals. The quality of deliverables has clearly taken backseat. I had occasion to be associated with some certified organizations, as a consultant or as a member of the audit team or as an employee. A significant number of those do not adhere to their own defined process. I was horrified to find one ISO 9000 certified organization in which the MR (Management Representative) did not read the process documentation. I noticed that the quality head of a CMM level 5 organization, doesn’t

know how to open the URL for the organizational process. I observed a CMM level 4 certified organization that does not collect nor maintain any metrics. In yet in another organization that is certified for ISO 9000 and aiming for CMMI level 3, I heard the CEO stating that he does not want any managers in his organization – I was dumbstruck wondering who would manage their software projects, if everybody were a coder. Organizations unearthed the loopholes in the models, consultants advise how to cook the books to get the certificate and appraisers are available who would certify for a fee. I posed a question for the CMMI appraisers group on Yahoo if they ever refused a certificate – only one or two replied in the affirmative – rest all maintained a dignified silence. The time has come to develop a different paradigm for quality in software, a paradigm that is focused on the quality of the deliverable than on the organization. Let us examine some of the criticisms of these Maturity Models (MMs). One criticism of all these MMs is that they emphasize on the organizational business objectives, but not on Product (or Deliverable if you prefer) Quality! The confidence that a process driven organization delivers Quality is misplaced. One – the process itself may be flawed. Two – each process has loopholes. Three – the people are focused on conforming to process, more than achieving excellence in quality. Four – management of the organization focuses on delivering and selling than on quality. The poor quality head, if there is one, is there to coordinate with certifying agencies and he has no control whatsoever on the product quality. In many organizations, I saw that the person holding the post of quality head (under many other nomenclatures like SEPG Head, Quality Coordinator, Quality Manager, Director Quality and so on) is not really qualified or experienced for holding that post or possess much knowledge about the quality concepts and tools. MMs focused less on development of software and ensuring that quality is built-in but focused more on support processes. CMMI has more (8) specific goals for Project Management (Project Planning, Integrated Project Management, Risk Management, Configuration Management, Project Monitoring and Control, Quantitative Project Management, Supplier Agreement Management, and Requirements Management) where as it has less (3) specific goals for Quality (Process and Product Quality Assurance, Validation and Verification). It has only three specific goals (Product Integration, Requirements Development and Technical Solution) for development of software. It has two specific goals (Organizational Process Focus and Organizational Process Development) for organizational process definition and four specific goals (Causal Analysis and Resolution, Decision Analysis and Resolution, Measurement and Analysis, and organizational Process Performance) for measurement and analysis. The remaining two specific goals are Organizational Innovation and Deployment and Organizational Training. Thus the focus on Quality is too diluted – three out of 22! Even then, the MMs do not insist that their model must be implemented in to-to. They accept “largely implemented” as adequate for giving a certificate. The model itself is not tightly defined and is made so flexible that the practices are open to any interpretation. Some allow “Alternative Practices” in place of the practices defined in the model. This allows the organization to do what they want and still get certified as conforming to the model!

Another criticism of the MMs is that they do not define any quality thresholds for achieving the certification. Conformance to self-defined process is adequate. Say the standard for an electrical equipment would define the insulation resistance in quantitative terms so that human beings do not get an electric shock by handling that equipment. But no software engineering standard defines what should be the defect density for, let us say, a financial application! Another criticism of MMs is that they do not specify the number of years the organization needs to be in operation before they can be mature enough to be ready for certification. That way, even a one-year-old organization can get certified. There are single person organizations that are certified! Another criticism about the MMs is that they do not specify any quality objectives to be achieved for obtaining the certificate. Mere conformance or showing evidence of conformance in just six projects or less gets the organization certified – no need to demonstrate achievement of quality! The owners of the MMs do not maintain the actual performance of the organizations after certification. ISO specifies surveillance audit that is cursory but CMMI does not. Whether the quality has improved, or any reduction in complaints – the owners of MMs do not keep track of. Let us examine some loopholes in these certifications. Pecuniary consideration Certification agencies charge a high fee – ($200 per hour is perhaps an indicator and the appraisal period ranges from 2 days to 3 weeks). Suppose an appraiser rejects certification to one of his clients, do you think that appraiser would get calls for appraising from any other organization? Therefore, the best that an appraiser would do is to cancel the assignment if he is dissatisfied with the preparation of the organization. Who would take the risk of being branded as too strict an appraiser? The organization that offers certification easily is the one most sought after. Another issue is that the certificate to issue certificates is being issued too easily. Pass an exam and you get it or attend a training program and you get it. Go forth and multiply certificates. Besides, these certifying organizations are profit-pursuing business organizations that have expenses to meet, targets to achieve, and growth to be aimed at. That is the reason why we see a plethora of certificates being issued. Method of appraisal The appraisal process itself is under criticism – the appraiser looks at the evidence presented to it. It is more like conformance audit – not an investigative audit. What is the guarantee that the evidence is not cooked up to suit the requirements of the appraiser? If financial accounting books (that are subject to statutory independent audit) are being cooked, why can’t certification evidence be? I have had a call to cook the evidence for a certification audit and the organization head told me, with a straight face, that the

certification agency knows this and is a willing partner. Of course, I told them to go look for somebody else. In many cases, the appraiser-organization itself happens to be both the process consultant as well as the appraiser for the organization. Surprisingly, neither the models nor the appraisers see any conflict of interest in such an engagement. Appraisal invariable goes by sampling. Sample gives a good picture of the universe when – 1. The universe is homogenous 2. The sample is picked up randomly The appraiser doesn’t ensure both the above pre-requisites before beginning the appraisal! In software organizations, both are false. The population is not homogenous. 1. All projects are similar perhaps, but not identical 2. All project managers are not uniformly qualified / trained nor have similar experience There is no guideline when 100% project appraisal becomes mandatory. Sample selection In none of the appraisals that I witnessed, either from close quarters or from a distance did select candidate projects thru any random sampling technique. The appraisers accepted the projects offered by the organizations. Two projects for ISO and six projects for CMMI – that is the norm followed by the organizations. The development organizations term projects as CMMI-Projects and non-CMMI Projects (alternatively ISO Project and non-ISO Projects). The appraiser assumes that all other projects are identical to the projects presented and accords certificate, which these organizations flaunt to induce customers. Redressal Mechanisms A customer or any concerned person has no place to complain if they have something to report on any certified organization. The email ids of officials responsible for looking into complaints about organizations (that are certified and using the certificate to obtain business) are not publicly available. Nor those officials take up suo-moto action on erring organizations. CMM was retired more than two years ago and still, there are many organizations claiming CMM level on their web sites! Post-certification reporting requirements There are none! The certified organizations need not submit compliance reports to any one! ISO mandates surveillance audit twice a year and the NCRs (Non-conformance Reports) raised by the auditors can be cleared by next audit and in the meanwhile the certificate is not suspended. No certificate is ever revoked – at least not to my knowledge or publicized.

CMMI requires re-appraisal once in three years – the certified organization can exploit the certificate for three years fully. Public limited companies (in other words certified by the Registrar of Companies or an equivalent authority who issues a certificate of incorporation) are mandated to publicize their financial achievements audited by an independent auditor. Should the certifying models not mandate such a requirement? There is no requirement in any of the models that the certified organization must make available their quality data like sigma level, defect density or the NCRs raised by auditors or Opportunities for Improvement pointed out by appraisers, on the web site of either the organization or that of the model owner. The organizations do not display their quality performance on their web sites. Any revocations so far? I searched the web sites of SEI and ISO to locate the list of organizations for which they rejected / revoked / cancelled the certificate. I could find none – nor could locate a link on Google search. This can mean that they have not rejected / revoked / cancelled even one certificate or they are keeping such a list confidential. My assumption is that they did not rejected / revoked / canceled a single certificate. Public display of such a list would go a long way in improving the credibility of the certification. Auditing the auditors The way, the certifying agencies are audited, leaves much to be desired. Model owners audit periodically the certifying agencies – not the organizations that are certified by the certifying agency! This again is a conformance audit not an investigative audit. So the surveillance on certifiers is also lax. Final Words The objectives of model definition and certification are not for ensuring quality. Possession of a certificate by an organization does not guarantee quality of deliverable. Satyam Computers has all the certificates from all types of certifying agencies; still those certificates did not prevent the Chairman from committing a massive self-confessed fraud! If the certified processes are working well, how could this happen? World Bank has banned organizations like Satyam and Wipro (both of whom have the highest levels of certification) from carrying our any work for the bank. If these organizations are treating an organization like World Bank in this manner what is the level of quality they are giving to other customers who do not have the same clout as the World Bank? What makes me wonder is that neither the ISO nor the SEI revoked the certificates for these two organizations! Does this tell us something about the credibility of the certificate?

Clearly, the certification failed. I know that I will be ridiculed for this. Some kid has to shout – that the King is naked! ************************************** About the Author – Murali Chemuturi is a Fellow of Industrial Engineering and an MBA and has over 30 years of corporate experience and about 8 years of consulting experience. He can be reached on [email protected] or thru his web site http://www.chemuturi.com . Your feedback is welcome and appreciated - it will be responded to in 24 hours. **************************************