IMF Working Paper: Appraising credit ratings - does the CAP fit better than the ROC?

14 May 2012

In this IMF study, the CAP and the ROC methods are compared, using as an illustration the ability of the OECD's country risk ratings to predict whether a country will have a programme with the IMF (an indicator of financial distress).

ROC (Receiver Operating Characteristic) and CAP (Cumulative Accuracy Profile) analysis are alternative methods for evaluating a wide range of diagnostic systems, including assessments of credit risk. ROC analysis is widely used in many fields, but in finance CAP analysis is more common.

ROC and CAP analyses both have the advantage of generating measures of accuracy that are independent of the choice of diagnostic threshold, such as risk rating. ROC analysis has other beneficial features, including theories for fitting models to data and for setting the optimal threshold, that this paper shows could also be incorporated into CAP analysis. But the natural interpretation of the ROC measure of accuracy and the independence of ROC curves from the probability of default are advantages unavailable to CAP analysis.

Full analysis

© International Monetary Fund