There is an oft-quoted business saying (often disputed and typically incorrectly sourced to management guru Peter Drucker) stating that “If you can’t measure it, you can’t manage it.” While we will let management experts debate the validity of this statement, when it comes to the world of qualitative technology risk assessments, we offer our own modification of “If you can measure it, then you can improve it.”
In technology, qualitative risk assessments help enterprises identify, analyze and evaluate weaknesses in their IT processes and security frameworks. They are a tool to help organizations identify potential threats, impacts and mitigation techniques. For example, if you are building a house and there is an impending lumber shortage, this will impact the schedule and you should adjust and react accordingly. Qualitative assessments by definition are highly subjective and reliant on the skill and judgment of subject matter experts to evaluate the likelihood and impact (i.e., relying on your lumber salesman to estimate the realistic due date of when and how much lumber will be delivered.).
Quantification
So why do we want to quantify something that is supposed to be based on subjective judgment? Again, consider the lumber example—the lumber seller can just make a guess using their experience of when and how much lumber will arrive. However, if the seller can use actual volume and delivery data from the past 90 days for similar-sized projects, they can achieve a much more refined and supportable estimate than one based on judgment alone.
The technology field is ripe for quantification support for qualitative judgments because data are typically readily available. A risk assessment normally contains categories such as inherent risk, top/emerging risk, control effectiveness, issue management, metrics and residual risk. These categories form the construct for assessing an organization’s risk posture and journey from inherent risk (the risk present absent of any controls) to residual risk (the risk present after controls are applied). Figure 1 outlines the journey.
Figure 1—The Qualitative Risk Assessment Journey
Inherent and residual risk are assessed by a 3 or 5-point scale (i.e., low, medium or high). The intermediate controls (specific control tests of technology processes or systems), issue management (how well issues are managed, the number of issues carried and their associated severity) and metrics (the state of operational key risk indicators and key performance indicators) are assessed individually and holistically to derive the final residual risk rating.
Aside from using data to better inform subjective conclusions, additional benefits of quantification include:
- Substantive rationale—Highly regulated organizations (banking, insurance, healthcare) are subject to scrutiny by federal or state regulators. Quantification reinforces the validity of the assessment.
- Consistency—Documenting the quantification elements as part of the qualitative risk assessment ensures a stable ongoing process, reducing risk from subject matter expert attrition or organizational changes.
- Better management focus—If one of the intermediate control areas is an outlier or more significantly influencing the overall results, quantification allows management to re-focus on areas to potentially improve the overall rating.
Blending the advantages
Exclusively quantitative and qualitative approaches have their own unique advantages and disadvantages. A purely quantitative approach has challenges such as insufficient data, difficulty in collecting data and more expense to ensure enough coverage of technology areas. Such a risk assessment is also highly subject to gaming the results through manipulation of the data—it can become a political math exercise. In contrast, a legitimate criticism of qualitative is that it is too highly reliant on pure judgment calls. Why not blend the advantages of both approaches and achieve a robust, high-quality assessment?
Editor’s note: For further insights on this topic, read the authors’ recent Journal article, “Quantifying the Qualitative Technology Risk Assessment,” ISACA Journal, volume 5 2022.
ISACA Journal turns 50 this year! Celebrate with us—and do not forget you can still receive the print copy by visiting your MyISACA dashboard and opting in!