18 January 2012

BIS Working Paper: 'Stress-testing macro stress-testing - does it live up to expectations?'

Content:

• One definition and five propositions
• What is a macro stress test?
• Proposition 1: Macro stress testing is a toolbox, not a single tool
• Proposition 2: Beware of macro stress tests as early warning devices
• Proposition 3: Macro stress tests can greatly help in crisis management/resolution
• Proposition 4: …and their additional benefits should not be underestimated
• Proposition 5: …but if you do them, do them right!
• Elements of good practice

Macro stress tests are set to become a core element of the macro-prudential frameworks being put in place across the globe. As offspring of the (micro) stress tests carried out by individual financial institutions, their ascendancy has gone unchallenged. And yet, stress tests failed spectacularly when they were needed most: none of them helped to detect the vulnerabilities in the financial system ahead of the recent financial crisis.

In this paper, the authors have argued that it is important to understand what stress tests can and cannot do. Ironically, macro stress tests are best suited to crisis management and resolution; currently, they are not reliable, in the authors' view, for identifying vulnerabilities in seemingly tranquil times – the purpose for which they were originally designed. They can help, and have helped, discipline and improve the dialogue about financial stability vulnerabilities; but, unless properly interpreted, they risk taking that dialogue astray. They can help, and have helped, spot shortcomings in models of systemic risk and financial crises; but they have so far largely done so because of what they have failed to produce (crisis warnings), rather than for what they have produced (comforting outcomes).

Ways to improve the performance of macro stress tests were discussed. From a technical perspective, it is well recognised by now that generating more realistic non-linearities and feedback effects is a priority. The authors remain sceptical, however, of attempts that see the secret of success in modelling network effects or the iterative bottom-up aggregation of individual responses. Also more global focus, rather than a jurisdiction-by-jurisdiction approach, would be helpful and a focus on common exposures is more promising than one on interlinkages.

Ultimately, however, improvements in the performance of stress tests depend on a change in mindset. No stress test can succeed unless there is a strong will to stress the system hard and to distrust rosy results. And here lies the problem. The importance of the right mindset has been appreciated ever since the inception of stress tests (e.g. CRMPG (1999)). But this proved to be no check on the generalised hubris that prevailed before the recent crisis among market participants and policymakers alike. Will it be any different next time?

Full working paper

© BIS - Bank for International Settlements