Model Risk & Validation Risk Modelling Stress Testing

Scenario Analysis in the Age of COVID-19: Do We Need a Different Approach?

2020 has been a challenging year for risk modelers, rife with uncertainty. What have we learned about the effectiveness of existing stress tests and the scenarios that drive them?

After the global financial crisis of 2008/09, macroeconomic scenario analysis became a central activity for credit risk modelers. Bank regulators and accounting standard setters from around the world adopted the technique as a primary plank in the analytical response to the recession. A decade on, considerable modeling muscle is flexed by banks to enumerate the impact of various narrative scenarios on their portfolios.

Tony Hughes headshot
Tony Hughes

The question we ask here, in light of the year we have all just experienced, is whether narrative scenarios provide a sufficient stress-testing solution or whether they should be augmented with other approaches.

The motivation for using scenarios in risk management stems from the highly nonlinear nature of credit losses. During a typical business cycle, defaults will be very low during the expansionary phase and much higher during the subsequent recession. Any credible exploration of bank losses must consider these tail events to give an accurate accounting of underlying portfolio risk.

The problem though, is that such tail events, by definition, are very rare. Recessions occur about once every decade – and many, like the 2001 event, are not especially deep. Any method used to enumerate tail behavior will be based on a vanishingly small amount of data, rendering associated statistical analysis highly speculative in nature.

Narrative vs. Simulation Scenarios: Pros and Cons

Shortly after the GFC, the financial services industry, led by regulators, quickly settled on a methodology involving a small number of narrative macroeconomic scenarios. This was not necessarily the most natural choice available at the time – the more technically minded would probably have plumped for Monte Carlo simulation, which has a long history in the academic literature and which allows the complete distribution of losses to be fleshed out and analyzed.

We can pinpoint a couple of reasons for going down the narrative scenario path. The first, I suspect, was logistical. When the SCAP stress test was implemented in early 2009, speed was of the essence: using two simple scenarios, banks were able to turn a complex set of results around within only a few weeks. This was critical, because markets were in a turmoil induced by counterparty distrust that needed to be quickly assuaged.

Preparing models for a simulation-based methodology is a slow process and the results would have been much harder to communicate to desperate financial markets amid the GFC. Narrative scenarios, in contrast, are simple heuristics allowing complex results to be rapidly compiled and easily digested by the general public.

There are, however, several disadvantages to the heuristic approach. The most obvious, made plain by the onset of the coronavirus, is that only a small number of ultimately irrelevant situations are ever explored in depth. You can re-run the contours of past downturns easily enough but, alas, recessions tend to have unique characteristics.

Next, let’s imagine that a particular bank opts to run, say, 10 scenarios through their modeling processes. The most important iteration would involve a broad, generic recession event, probably of a similar nature to the 2008/09 Great Recession. This scenario would form the backbone for most applications, including capital planning and loss-reserve calculation.

Subsequent scenarios would then need to be orthogonal, in some sense, to those previously considered, so that they add something to the overall analysis. However, since the law of diminishing marginal returns most certainly applies in the field of scenario analytics, it’s very hard to imagine the 10th scenario providing fresh insights that could not be gleaned by interpolation from the previous nine.

When reading public documents from regulators, you would be excused for thinking that scenario analyses were precise mathematical exercises conducted by banks with zero statistical error. But projections from stress-testing models are actually just midpoints in error banks whose widths expand as the scenarios become stranger. Therefore, even in situations involving a small number of scenarios, there must be considerable doubt that the paths generated are statistically distinct from each other.

This is the big danger in applying a simple, heuristic approach to such a complex problem: risk concepts are effectively conveyed to non-experts – but in a form that understates the degree of uncertainty that is manifest in any statistical exploration of tail behavior. The solution is not to abandon scenario analysis altogether but to augment it with other, more statistically-rigorous methodologies.

An approach using Monte Carlo simulation may be an interesting path for regulators to consider here. For one thing, doubts about the model and its associated coefficients could be explicitly incorporated into the results of such a treatment. Moreover, macroeconomic situations are readily available and widely used in the insurance industry, and these could provide a common set of inputs that would allow results to be made comparable across banks.

The use of simulations removes the focus from individual iterations, promoting instead a broader view of the distribution of outcomes.

Parting Thoughts

A simulation-based methodology will not be a panacea. The problems caused by a lack of historical tail observations still apply – i.e., difficult decisions will need to be made about the precise shape of the outer reaches of the target distribution.

A lot of water has flown under the bridge since SCAP, but stress testing, remarkably, has remained basically unchanged. Did the architects of stress testing really design the perfect long-term solution, in the middle of a crisis, at the very first attempt?

Once the COVID-19 clouds lift, hopefully it will spark a new wave of innovation in the industry, better preparing us for the next proper financial crisis, whatever its ultimate cause.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s