10 August 2013

BIS Working Paper: Evaluating early warning indicators of banking crises - Satisfying policy requirements

This paper claims that early warning indicators (EWIs) of banking crises should ideally be evaluated on the basis of their performance relative to the macro-prudential policy-maker's decision problem.

Authors: Mathias Drehmann and Mikael Juselius

The authors translate several practical aspects of this problem – such as difficulties  in assessing the costs and benefits of various policy measures as well as requirements for the timing and stability of EWIs – into statistical evaluation criteria. Applying the criteria to a set of potential EWIs, they find that the credit-to-GDP gap and a new indicator, the debt service ratio (DSR), consistently outperform other measures. The credit-to-GDP gap is the best indicator at longer horizons, whereas the DSR dominates at shorter horizons.

Given the difficulty of estimating the costs and benefits of macroprudential policies, the second best option is to evaluate EWIs over a range of possible utility functions. As the optimal decision under a specific utility function implies a specific trade-off between Type I and Type II errors, one way to achieve this is to consider the full mapping between such trade-offs that a given EWI generates.

The authors find that the credit-to-GDP gap and the DSR are the best performing EWIs  in terms of their evaluation criteria. Their forecasting abilities dominate those of the other EWIs at all policy-relevant horizons. In addition, these two variables satisfy the criteria pertaining to the stability and interpretability of the signals. As the credit-to GDP gap reflects the build-up of leverage of private sector borrowers and the DSR captures incipient liquidity constraints, their timing is somewhat different. While the credit-to-GDP gap performs consistently well, even over horizons of up to five years ahead of crises, the DSR becomes very precise two years ahead of crises. Using and combining the information of both indicators is therefore ideal from a policy perspective. Of the remaining indicators, only the non-core liability ratio fulfils the statistical criteria. But its AUC is always statistically smaller than the AUC of either the credit-to-GDP gap or the DSR. These results are robust with respect to different aspects of the estimation, such as the particular sample or the specific crisis classification used.

The authors argue that assessments of EWIs of banking crises should be based on the underlying decision problem of the policy-maker. Several aspects of this problem have implications for the choice of statistical evaluation procedures. For instance, uncertainty about the costs and benefits of macroprudential policies imply that evaluations need to be robust over a wide range of the policy maker’s preferences. Also, the EWI signals should have the right timing, and their quality should not deteriorate in the run-up to a crisis. The authors embed these requirements in a statistical evaluation procedure for EWIs.

Applying their approach to several EWIs, the authors find that the credit-to-GDP gap, the DSR and the non-core liability ratio satisfy the policy requirements. The first two variables consistently outperform the third, with the credit-to-GDP gap dominating at longer horizons and the DSR at shorter horizons. The authors' results are robust with respect to changes in both sample and crisis classification.

A distinguishing feature of the analysis is that a greater attention is paid to the temporal dimension of the EWI signals. The findings reveal that the signalling quality of different EWIs can fluctuate sharply over time. This suggests that greater reliance should be placed on EWIs that continuously perform well within the policy relevant forecasting interval.

Full paper

© BIS - Bank for International Settlements