The purpose of any risk analysis is to provide the decision-maker with the best possible information about loss probabilities. Consequently, it’s crucial that decision-makers accept the risk analysis methodology being used, and that the information resulting from the analysis is in a form that’s useful to them. In this regard, the limitations of our traditional information security “risk analysis” methods will become clear as we progress through this document.

Limitations of risk analysis

Risk analysis is never perfect. The fact is, all risk analysis models are approximations of reality because reality is far too complex to ever model exactly. Nonetheless, by decomposing a complex subject into clearer, more readily analyzed components, we can understand and make reasoned judgments about the subject.

Any analysis model will be limited by the complexity of the subject, how deeply we understand the subject, and how well the model incorporates that understanding.

Evaluating risk analysis methods

FAIR is just one way of skinning the risk analysis cat. There are many dedicated people working to develop other methods with the same goals in mind. This is terrific news to those of us who are trying to be as effective as possible in managing risk. However, regardless of the methods you consider using, I encourage you to evaluate any risk analysis method on at least three points.

• Is it useful?
• Is it logical?
• Does it track with reality?

A methodology is useful when it meets the needs of the people making the risk decisions. If our analysis provides information regarding the effectiveness of controls in an environment, but decision-makers are looking for information regarding the probability of incidents and losses, then the methodology isn’t useful. Likewise, if we provide “key risk indicator” metrics, but there is no clear linkage between the conditions described by the metrics and the probability of loss, then the metrics become little more than window dressing.

There are “risk analysis” methods in use today whose logic falls apart under close examination. Often, these methods call themselves risk analysis when what they’re really analyzing is a subcomponent of risk (e.g., vulnerability or controls). Keep in mind, however, that these methods may be excellent at what they actually do, so don’t disregard their value. It’s simply a matter of recognizing what they do and don’t provide. A simple way of identifying a bona fide risk analysis method is to determine whether it includes an analysis of threat frequency, vulnerability, and loss magnitude, and whether it treats the problem probabilistically. If one or more of these components is missing, or if the problem isn’t treated as a probability, then logically, it isn’t a risk analysis method.

The last point of consideration – tracking with reality – is especially critical. It’s also a point that may be particularly challenging for our profession to come to terms with. Let’s use, as an example, the controls condition of a typical corporate internal network. My experience has been that most corporate internal networks and systems are not very well protected (at least relative to theoretical “best practices”), yet relatively few organizations have actually experienced a severe loss associated with this fact. Significant loss events do occur, no question of that, but they’re astonishingly infrequent given the prevalence of bad security practices. A realistic risk analysis should reflect this fact. In other words, an effective risk analysis of an average internal corporate information technology environment should, in most instances, identify very few truly high risk issues in spite of the fact that controls may not meet best practice standards.

Our profession has become very good at evaluating the technical controls in an environment – password length/complexity/history, whether appropriate access privileges are in place, how many layers of firewalls there are, how up-to-date the antivirus product is, whether intrusion detection exists, etc. We also know what the key non-technical controls are – whether policies exist, the level of user awareness, executive sponsorship of the security program, etc. But controls are only part of the risk equation. In fact, controls are only part of the vulnerability equation. We’ll cover the risk “equation” as we go along but, as I described in the introduction, vulnerability is always relative to the type and level of force being applied. If we only measure controls, but call the results “vulnerability,” then we’ve made significant assumptions about the level and type of threats involved. In order to effectively evaluate risk, all of the components of risk must be included.

When you’re evaluating any risk analysis method – FAIR included – ask hard questions. Make certain that it meets the needs of the decision-makers, that it’s logical, and that it aligns with reality.

next:  Probabilities versus Possibilities