Issues Magazine

Calculating Risk and Living with the Consequences

By Paul Davis

A risk analyst revisits the assessment and acceptability of risk after losing his house to a forest fire.

For the past several decades I have been performing risk assessments related to environmental issues such as the releases of toxic chemicals to groundwater and the disposal of radioactive waste. These risk assessments were associated with hazards that I will never be exposed to. The closest groundwater contamination problem or radioactive waste site I have dealt with is 70 km from my home. The extreme example of the separation between the risks and me, the risk assessor, is the disposal of radioactive waste. The risks are so far into the future that the people who will bear the risks have yet to be born.

Recent events made risk assessment take on a new dimension for me, as I had to live with the consequences. I lived in the forest, there was a forest fire, and my home of more than 20 years burned down.

Was I aware of the risk? Had I consciously accepted the risk? I was more than aware. As an environmentalist engaged in forest policy, I had calculated the risk of my home burning down in a forest fire before the event. My estimate, presented to the authorities that manage the public forest adjacent to my home, was a less than a one-in-300 chance in any given year. This risk was quite a bit larger than the one-in-10,000 to one-in-a-million risks for toxic chemical spills and radioactive waste disposal.

But, for me, it was a low risk that I knowingly accepted.

Given the fact that my home subsequently burned down in a forest fire, as a risk assessor I was faced with a number of questions. Had this really been an unacceptable risk? Was my risk calculation wrong? And was risk the right metric for decision-making?

Was the Risk of Fire Really Acceptable?
Is there such a thing as an absolute level of acceptable risk? In risk assessment we most often use risk limits set by people who are not exposed to the risk – usually government officials – and the risk limits set by these government officials are not limits that can be applied to other problems. So what is an acceptable level of risk for my home burning down? Answering this fundamental question depends on what could be done to mitigate the risk.

In the case of my home, the risk mitigation options included cutting down all the trees surrounding my home, and/or supporting the local forest manager in their desire to cut down the majority of trees on the adjacent land, or to do nothing. Here is where the devil comes out of the details: there are no risk-free, cost-free decisions.

Neither of the tree-thinning alternatives eliminated the risk of fire. There were arguments that thinning trees might reduce fire risk, but no one was arguing that thinning trees would eliminate the risk. In fact, recent wildfires in the area had burned down a number of homes but those were grassland fires – no trees.

So was the reduction of risk worth the cost? Neither of the thinning options was free, and not just in terms of cost. Thinning trees on the adjacent public land was projected to cost US$17 million, far exceeding the value of all of the taxpayers’ homes burned in the fire.

Removing all of the trees around my home would not have cost much, especially since I own several chain saws. However, the cost associated with removing the trees was not only a monetary cost. That cost was minimal. The cost was the trees.

Let me explain. One giant ponderosa pine tree was just outside my living room window and touched the roof of my home. The cost of cutting down that tree was not monetary. The cost was the loss of many, many hours of watching birds perched in its limbs and squirrels climbing up and down, jumping from branch to branch. All of my life I had wanted to live in the forest to enjoy the trees and now I was being encouraged to cut down those very trees to reduce my fire risk.

When I consider the costs and the slight reduction in risk afforded by tree thinning I believed – and still believe – that the risk associated with keeping the trees was acceptable.

Was My Risk Assessment Correct?
Given that the fire burned down my home, was my risk assessment correct? The fact that a single event happened does not immediately call into question any risk assessment any more than winning the lottery once would increase your chances of winning a second time.

But let’s go further. No risk assessment is “correct”. Risk assessments are based on choices of data and assumptions about the data. Each risk assessor may view certain data as more significant or representative than other data.

Differences in calculated risks are possible even when the same data are used because many equally valid assumptions can be made. For these and other reasons, risk assessment is neither a prediction of future events nor is it even a unique probabilistic assessment of possible future events. It is only a calculation made to inform decisions. Maybe asking whether or not the risk assessment was correct is the wrong question.

The risk assessment I did was credible, to me, because I did it. That sounds kind of arrogant, doesn’t it? It is not meant that way. Quantitative risk assessment appears impersonal and independent of the risk assessor, but it isn’t.

Differences in calculated risks arise from an array of factors including, but not limited to, differences in both the educational backgrounds (not one assessor having a superior education, just different) and the experience of the risk assessors. Such is the problem of “conceptual parallax” – a photographic term meant to convey that differences in answers arise simply because of different points of view.

Then there are biases associated with risk assessment. Institutional bias is a type of conceptual parallax where the point of view of the risk assessor is affected by the institution with whom he or she is associated or works with. Here I use the term “institution” broadly, meaning anything from a corporation or university to a small group of like-minded people.

Institutional bias can range from the explicit to the subliminal, but it is always present in risk analysis. I have always been “impressed” with risk assessors who consistently calculate risks that support their client’s contentions in spite of the evidence. In fact, one local hydrologist argued in a published article that he should produce results that pleased his clients and not results that pleased science.

However, the institutional bias I am speaking of is not that conscious. Unconscious institutional bias runs much deeper and is much harder to detect. Unconscious institutional bias comes from working at an agency, in a field or with a group for so long that the bias becomes part of your genetic code.

In national hearings this year on high-level radioactive waste disposal in the United States, one of my colleagues assured the panel that he had never been asked to change his data or results to conform to his employer’s views or desires. He didn’t have to. He had worked for so long at the same job for an agency with a strong bias in support of nuclear power that his calculations always conformed to his employer’s views. Does he recognise that he is biased? No. Does this mean his risk assessments are wrong? No. The assumptions and data he chooses are valid to use in risk assessment. But there are other equally valid assumptions and data that another risk assessor in another institution may make or use.

In the case of the fire, no other risk assessments were performed. Was my risk assessment biased? There was no way for me to know whether or not my risk assessment was biased at the time I did the calculations – one cannot identify his or her own biases. To know the answer to that question would have required another risk assessment or at least a review of my risk assessment by others with differing backgrounds and points of view.

However, given that the fire occurred I was able to realise one problem in my risk assessment resulting from bias. I had overlooked the possibility that the fire fighters would increase the risk to my home. Here is why.

My risk assessment took into account all the fire data and studies related to the area where I lived. Data indicated that the majority of fires were small and caused by lightning. Lightning fires are generally small in part because the agency that fights forest fires has been efficient at getting to these fires early and keeping them small, but mainly because lightning only ignites the tree it hits.

While lightning started the fire that burned down my home, the agency in charge did not get to the fire for several days, by which time the fire had spread to a large area. But the spreading of the fire itself would not have burned my home.

Unfortunately, while the fire was still over 2 km from my home and moving away, the agency decided to start what they refer to as backfires to reduce fuels. But the wind was high and variable when they lit these backfires and they got out of control and burned down my house.

I had not included this possibility in my risk assessment, but I should have. My bias was assuming that anything done in fighting the fire would reduce the risk of the fire reaching my home. If I had looked harder I would have found that it is not unusual for backfires to increase the size and severity of forest fires. Another person performing the risk assessment with less trust in the agency – a different bias – might have incorporated the data I had left out. Having a more complete risk assessment would not have changed my decision to live in the forest and not cut the trees, but it would have caused me to urge the public lands manager not to use backfires.

Because individual bias is so hard to identify, we should always have multiple risk assessments performed by people with different views. Because of these biases, and because risk assessment plays such a vital role in so many government decisions, I strongly recommend that public policy should be informed by multiple risk assessments, even to the extent of helping the public to do their own risk assessments.

Is Risk the Best Metric to Inform Decision-Making?
Given that I based my decision to live in the forest on risk and a fire burned my house down, was risk the right metric for decision-making? What is the alternative? Hazard.

Hazard can be defined as risk with the implicit assumption of a probability of one. Hazard emphasises the bad things that can happen but leaves out how likely they are to happen. Most people that use the word “risk” when they actually mean “hazard”. We talk of the risk of fire, the risk of war, the risk of a car accident, and in even the risk of terrorism.

But listen carefully – we are really talking about hazards with no mention of probability. For instance, the agency in charge of the public lands adjacent to my home uses the phrase “the risk of fire” but never states the probability of a forest fire. No, they say a forest fire is imminent, the entire forest will burn down, and something must be done to eliminate the “risk”.

Were they right that we should worry about the hazard of a forest fire and not the risk? My home did burn down. However, only a small percentage of the forest and only five homes actually burned, leaving the vast majority of the forest and the homes untouched. So even if they were totally right with respect to my home, they were totally wrong for all the others.

If I had based my decision on hazard I would have never lived in the forest. The threat of an imminent fire would have prevented me from ever considering living in a forested area. Instead, I enjoyed my home and the forest for over 20 years before it burned down. The public lands agency overlooked the fact that people choose to live with risk, and benefit from that choice.

There is a major reason that institutions as well as individuals focus on hazards instead of risk: the propagation of fear to motivate people into actions that otherwise would not seem reasonable. In this case, the agency managing the forest had long wanted to thin the forest for commercial purposes. However, the public was not receptive to commercial thinning. It became much easier to convince the public to thin the trees because of the “risk” of fire.

Which of the following statements would lead to rational decision-making?

There is an imminent threat of the entire forest burning down and we must do something about it.

or

There is a slight to moderate risk of fire to some areas of the forest. Thinning the forest could reduce but not eliminate the risk, and the cost is US$17 million.

Beware when someone is beating the hazard drum with no mention of risk – there are other forces at work.

Overall, my fire risk assessment led to decisions that I could and did live with. But this risk assessment was nothing like the risk assessments I do professionally. In the fire risk, the same person doing the assessment decided on the acceptable level of risk, was willing to live with the risk, and influenced the decision on how to best manage the risk.

This fundamental difference between my fire risk assessment and my professional risk assessments can lead to intractable problems. In the professional case, the people who bear the risk are not the risk assessors and never set the levels of acceptable risk. The people who bear the risk are not necessarily the people who benefit from the decisions but they are the people paying at least part of the cost to manage the risk.

Let’s take one extreme example of what can go wrong when the risk assessment process is completely disconnected from those who bear the risk. The US government proposed Nevada as the location for our high-level radioactive waste facility. The majority of this waste is generated in nuclear power plants, none of which are located in Nevada. The choice of Nevada and the decision that wastes should be disposed of in a geological repository and not stored indefinitely was not made by the citizens of Nevada or the US, but by government officials. And while current Nevada residents may not be exposed to the very long-term risks associated with disposal, they would be exposed to waste transportation risks.

Government officials also decided what level of risk was acceptable. In fact, the federal regulatory agencies gave the Nevada site its own risk level because the site had difficulty complying with the existing standards. These government officials live in or around Washington, DC, not Nevada. And the risk assessors were from everywhere, including a few in Nevada. These risk assessors, working for both the operator and the regulatory agency, calculated risks that met risk standards set for the site.

In a representative government with agencies set up to deal with these types of national problems, it is normal for the government to make such decisions and perform the risk assessments. But is this the best process to follow in defining acceptable risk, risk management options, and calculating risks?

Looking back on my experience, I am not surprised that after spending more than US$12 billion, on 14 April 2011 the project was terminated by President Obama due to local opposition.

In closing let me state that I still support risk assessment as a key component to rational decision-making. But it is not the results I support as much as I support a transparent inclusive risk assessment process that involves those who have to bear the risk in deciding on risk management alternatives, defining acceptable risk, and even in performing risk assessments.

The author kindly thanks Helena Dunn, James Counts, James Syme and Jeremy Weiss for volunteering the time, energy and patience to edit this article.