Editor’s note: This is part 2 of a blog series by Ron Lear on capability and performance. Read part 1 here.
Trivia Question Topic – Performance vs Compliance: Name this 1979 movie all about performance and name the actor asking the audition questions. Note that the name of the character (not actor’s name) has been changed to add another trivia question and not give away the answer. For extra “Ron Bucks” or points, what is the significance with the alias name used “Brody” vs. the actual character’s name in the movie?
Brody: You were in Traffic Jam?
Hopeful Cast Member: Yes, sir
Brody: Who was the director of Traffic Jam?
Hopeful Cast Member: You were, Mr. Brody
Brody: Oh. How was I?
Hopeful Cast Member: Terrific.
Brody: And who was the choreographer?
Hopeful Cast Member: You were, Mr. Brody.
Brody: How was I?
Hopeful Cast Member: Fantastic.
Brody: That's how you get a job.
Trivia question: Name this movie and the actor asking the audition questions.
Trivia rules: Use your brains, memory and reasoning to figure these out vs. Google, IMDB, etc.
Take a maturity level-based compliance framework as a compliance example. The intention and design of these frameworks is that compliance can be boiled down to a single number, e.g. maturity level 3, which theoretically then equates to certain demonstrable performance expectations. But unless those maturity levels are calibrated through some level of performance benchmarking requirements, specifications, objectives and measured outcomes, they quickly become meaningless and can promote a “race to the bottom” to check the compliance box and then go back to business as usual. This “one and done” approach is also dangerous and risky because of the increasingly rapid changes in the world, technology and business. It also effectively ignores the need for performance improvement and innovation.
Without the corroboration or correlation between abstract maturity levels, no matter how well-defined they may be, maturity levels are meaningless, and that has NOT escaped the notice of US Department of Defense officials who require and use CMMI maturity level ratings in their RFPs. Spitting back what they want to hear, like Mr. Brody, may land them the job initially, but it is NOT a guarantee of better performance. Focusing on continuously improving performance and outcomes does a much better job in terms of performance expectations and real results.
But don’t take my word for it. Here are examples of performance outcomes from nearly 100 organizations that recently adopted CMMI V2.0:
- “The CMMI journey has helped us develop a culture of process transformation with predictable delivery.” (Vice President and Global Head of Quality and Enterprise Risk Management, Wipro)
- “The CMMI model taught us to think in favor of the customer and be very thorough in terms of delivering our development projects. We work to improve performance from task to task, decision to decision and project to project.”
- These results are also consistent across 19 different industries, including finance, aerospace, transportation, telecommunications, and more. Government contracting is only one of those 19 and not even the largest.
- 80% of the targeted objectives showed significant improvement for quality, productivity, cost performance, schedule performance, functionality and process adherence.
- Quality-related objectives represent the biggest targeted area for performance improvement, with significant reductions in delivered defects and increased work product quality and testing effectiveness. Not only did quality performance improve, but the ability to detect and prevent defects also improved consistently for those organizations that targeted this area.
- From a recent March 2021 CMMI Appraisal, excerpts from the appraisal sponsor, who also happens to be the CEO of the company:
- Regarding the CMMI Performance Report, the sponsor mentioned that he thought it was one of the most valuable outputs of the appraisal process because it helped him to validate his organization’s business objectives and alignment to them from the objectives, baselines and models.
- The Performance Report will now become part of the company's process assets and they will be maintaining it on a quarterly basis as their performance and data evolve.
- His primary motivation for being appraised against ML5 was to confirm if his company’s performance had increased compared with the last appraisal (ML3) - not to win contracts. In fact, he told the lead appraiser that he’s already satisfied the barrier of entry for winning contracts by previously being appraised to ML3. In his words: “I’m doing this for me and my company; my customers didn’t ask for it.”
- Any improvement suggestions his staff come up with need to be backed up with statistical analysis. In his words: “anyone can make suggestions, but they need to back it up with the data.”
- He has linked his company’s bonus system to the collection and analysis of quantitative and statistical techniques to encourage habit and persistence (sustained performance).
This need to shift from compliance-based thinking to continuous performance improvement was the lesson the US Food and Drug Administration (FDA) discovered and decided to tackle in 2011. The study the FDA commissioned is very revealing on how a compliance-based audit approach did NOT yield the results they expected.
The following from this 2011 FDA study explain the issues and facts well:
“These inputs uncovered several key facts about marketed medical devices as well as potential catalysts for quality improvement.
- Serious adverse event reports related to medical device use have outpaced industry growth by 8% per annum since 2001.
- Quality risk is not evenly distributed across the industry. This reflects the heterogeneity and complexity of the devices, manufacturers and use environments. Cardiovascular, in vitro diagnostic (IVD) and general hospital/surgical devices account for nearly 60% of adverse events reports. Only 20 of the 1,189 active product codes account for 65% of all serious adverse events reports between 2005 and 2009.
- An analysis of root cause data reveals that failures in product design and manufacturing process control caused more than half of all product recalls. The root causes of quality issues are tied closely to device type. Therapeutic area was not as strong a predictor for recall root cause.”
In short, 20-30 years of FDA regulations and the resulting compliance audits did not prevent medical devices from failing.
Based on this study, the FDA decided to pivot to a performance-based, continuous improvement approach, with four primary objectives and measurements including safety, effectiveness, reliability and availability. The Case for Quality Voluntary Improvement Program (CfQ VIP or VIP) is a collaborative initiative between FDA CDRH, MDIC, ISACA, and the medical device industry. This program was developed with the intention to elevate medical device stakeholders from the established standard of regulatory compliance to advancing medical device quality and safety through sustained and predictive practices that achieve better patient outcomes.
VIP leverages the Medical Device Discovery Appraisal Program (MDDAP), which is a tailored version of the ISACA CMMI framework, as a model and appraisal method by which medical device organizations can measure their capability to manufacture high quality devices. FDA CDRH has adjusted the engagement activities and submission requirements for VIP participants in recognition of completing this independent assessment and committing to continuous improvement in an effort to support their individual improvement journeys and reach industry goals such as:
The tailored version of the ISACA CMMI framework used by the FDA and VIP is based on the CMMI Version 2.0, released in 2018. Given the FDA’s continued success and the results from these first ~100 companies adopting CMMI V2.0 leads to one inescapable conclusion: performance outcomes for organizations adopting CMMI V2.0 are clear, consistent and compelling – so much so, that the US Air Force is currently working on a continuous performance improvement initiative using CMMI V2.0 similar to the MDDAP program. Stay tuned for more on that in future blogs.