Quantitative Evidence of the Benefits of CMMI Adoption
One of the most frequently asked questions that you tend to encounter as a consultant working with the Capability Maturity Model Integration® (CMMI®), concerns the benefits that client organizations could realistically be expected to realize by investing their time, effort and money in adopting this long established process framework to plan and manage their improvement activities. It is almost an unanswerable question because, as with most endeavours, the more you put into it, the more benefit you are likely to realize. Yet at the same time, it is obviously not unreasonable for a client to want some credible indication as to the possible return on their investment before making any commitments.
ISACA have just released the latest version of their CMMI Performance Report (Full title: “CMMI Technical Report: Performance Results” https://cmmiinstitute.com/resource-files/public/cmmi-performance-report-summary#) which may go some way to help provide an answer to these types of questions.
This document looks at all CMMI appraisals that have been conducted and reported over the period 2019 – 2023 and seeks to identify through the reported appraisal objectives, what benefits organizations are actually realizing through their use of the CMMI. When an organization undertakes a CMMI appraisal, it declares a set of objectives outlining key aspects of organizational performance that they wish to improve. Each improvement objective is allocated to a specific category of improvement (e.g. Quality, Productivity etc.) and a more detailed sub-category (e.g. Defect Detection, Schedule performance etc.), and the full appraisal record then records the situation both ‘before’ and ‘after’ the improvement work. From this, an improvement figure for each objective can be calculated.
The performance report aggregates all this data to identify average improvements against all improvement categories to give a picture of where the most benefit is being realized and what the typical magnitude of the improvement is.
Basis for the Report Conclusions
It is important to stress that the Performance Report is underpinned by a sizeable body of data. In the last year alone (i.e. 2023), over 5000 appraisals were reported and the document is based on nearly 15,000 appraisals in total over the full period (n.b. The COVID pandemic severely impacted the number of appraisals performed at the start of the report period). These appraisals were undertaken by nearly 14,000 organizations and over 50,000 performance objectives were reported. Obviously, every organization is different and faces unique challenges of their own, so there are no guarantees as to exactly how much of a difference an individual improvement program will make, but the sample size used in the creation of this report is sizeable and therefore the results can be seen as indicative of what can and is being achieved within organizations that are using CMMI. Furthermore, all appraisal objectives are independently validated by CMMI Appraisal teams during the appraisal process so the reported figures are backed up by objective evidence and a degree of independent scrutiny.
So what does the Performance Report tell us?
The first thing to note is the success rate of reported objectives. 86% of all stated improvement objectives were identified as having been ‘Met’ with a further 4% having been marked as ‘Partially Met’.
This is compelling evidence that organizations which use CMMI to set improvement goals have been highly successful in achieving those goals. However, objective achievement figures do not indicate the actual magnitude of the improvements that have been realized. For that we have to look at the actual improvement figures.
Where are the biggest improvements being seen?
The Performance Report identifies nearly 100 categories and sub-categories that a performance objective can be assigned to. Of the ones that have been used, the top 5 sub-categories point to impressive average improvements:
· On-Time Delivery of products and projects increased by an average of 33%
· Schedule Variance reduced by an average of 48%
· Development Productivity increased by 17% on average
· Defect Rate decreased by 31%
· Customer Satisfaction increased by 13%
These are significant improvements that are being claimed through objectively validated real-world data. And these are just the headline values for the top 5 improvement areas. The main body of the report contains a lot of detail regarding the achieved improvements in all of the main performance categories. As well as average improvements the report includes upper and lower ranges for all popular improvement areas which helps to put these figures into their true context.
But CMMI Is Just for Large Software Organizations isn’t it?
This is a myth that grew up from the early days of CMMI where most of the early adopters tended to be large government contractors who had been actively involved in the development of the initial capability maturity models, so whilst there may have been some basis in fact for this when CMMI was in its infancy, the Performance Report shows quite clearly that this is certainly not the case today.
In reality, nearly 90% of all appraisals performed over the report period were conducted by organizations with less than 100 FTEs and in fact, nearly 75% of appraisals were performed by organizations with no more than 50 employees. Clearly CMMI is no longer confined purely to large organizations.
Moreover, whilst the initial CMMI sought to unify previously separate Capability Maturity Models for software and systems development, CMMI has now evolved and the latest version defines a total of 8 specialist subject areas. In addition to product development, CMMI now contains best practice information relating to Data Management, People Management, Service Delivery, Supplier Management, Safety, Security and Virtual working. The Performance Report shows data that indicates how the take up of multiple specialist domains is on a steady upward trajectory and the types of organization that are putting CMMI to use continues to expand.
Conclusions
Now in its third main version, CMMI continues to demonstrate that it can give significant benefits to organizations which choose to use it to support their improvement programs. Moreover, these benefits are being reported by organizations of all sizes and across many different industries. The CMMI Performance Report provides evidence of actual benefits achieved through using the CMMI model and shows that the benefits can be substantial and far reaching. With the latest version of CMMI containing 8 specialist domains, the framework can deliver these benefits to a much broader range of industries and organizations.
If you are interested in discussing how CMMI might help your organization, please contact us to arrange an initial consultation.
Comments