The Office of the Comptroller of the Currency (the “OCC”) and the Federal Reserve (the “Fed”) recently issued some guidance regarding Model Risk. OCC Bulletin 2011-12 and Supervision and Regulation Letter 11-7 outline new supervisory guidance describing how financial institutions should manage the risks associated with model use. By model use, we are referring to the analytical models used by management to evaluate the risks and opportunities of their financial institution. Both third party models and user developed models are within the scope of the guidance. This blog post summarizes the guidance and provides some practical examples of how the guidance can be implemented at your financial institution.
The first step in model risk assessment is to create a detailed organization-wide inventory of the models used by your financial institution. I’m sure there are many, but since Twenty Twenty’s Loan Portfolio Analysis Model falls within the scope of the guidance, if there is need for an example that’s the one I’ll use. Another example of a model would be the model you use (if you use one) to calculate your allowance. Your credit score provider uses a model to generate credit scores for your members.
The next step, very broadly, is to evaluate the risk of your models. The OCC Guidance defines a financial institution’s model risk by “the potential that it may experience adverse consequences based on a decision reached by using a model, and it originates due to fundamental model errors or due to incorrect or inappropriate use of an otherwise sound model.” Chopping this definition up into two parts and looking at each part individually seems to me as the easiest way to really explain what the guidance is asking you to assess.
To elaborate on The First Part, my formal training as a financial statement auditor comes into play and I think of the concepts of “Risk of Incorrect Acceptance” and “Risk of Incorrect Rejection.” The Risk of Incorrect Acceptance is when you believe something to be good, or accurate, when it is in fact not. For example, a model telling you that a loan is Low Risk when it is in fact High Risk. You make a loan, or authorize a credit limit increase for a member and six months down the road that loan has defaulted. The Risk of Incorrect Rejection is just the opposite. A model deems a loan High Risk when it is in fact Low Risk. You reject a loan, or credit limit increase to a member that would not have defaulted, foregoing interest revenue that could have been realized from that loan. Whatever the model is evaluating, an effective model decreases the Risk of Incorrect Acceptance or Rejection.
The Second Part of the definition talks about the things that can go wrong leading to Incorrect Acceptance or Rejection. There are two ways a model can lead to these adverse consequences: Fundamental Model Errors and Incorrect or Inappropriate Use of an otherwise sound model.
Fundamental Model Errors could be anything from a calculation error to something more complex like the credit quality indicators used to assess risk not being the credit quality indicators that correlate with future losses. These fundamental errors in modeling have the potential to cause adverse consequences. In the case of Twenty Twenty Analytics, we perform extensive real world case studies (matching losses with credit quality indicators of those borrowers) to determine what attributes lead to future losses. It is important that you know both how a model works and how it was developed to evaluate if there are fundamental errors in the model.
Incorrect or Inappropriate Use of a model, in its simplest form, would be inserting inaccurate data into an effective model. If you’re trying to calculate 2+2 in your calculator and you accidentally type in 2+3, you’re going to get the wrong answer, even though the calculator did its job correctly. Data accuracy is increasingly difficult when data comes from multiple systems or sources. The difficulty of using data from your system in your internally developed model is that there is no independence. You (or your team) compiled and entered that data! Why do we need to test the data? How could it be wrong? It can. I don’t even have to break stride in typing to think of the examples: Maturity dates that have come and gone, duplicate loan entries, incorrect or absent collateral information, credit scores that cannot possibly be credit scores. Unfortunately, credit bureaus also compile inaccurate data. Otherwise, those free credit report websites wouldn’t be so popular.
Mitigating the threat against incorrect or inappropriate model use is one of the ways Twenty Twenty Analytics (and other third party service providers) differentiate ourselves from our software counterparts. Since we receive your data and input it into our database, we have the opportunity to review the data for any potential inaccuracies and discuss them before running them through our model and finalizing your analysis. Since the completed analysis, not only a model for you to utilize internally, is delivered to you, we have the opportunity to ensure that the model is only utilized for its intended use.
The Board of Directors and Senior Management have the ultimate responsibility for implementing model risk management practices. The literature provides a framework for the independent challenging of third party and user developed models but without the channels for an effective challenge, your independent validation team can’t perform its duties. The guidance defines the principle of “effective challenge” as “the critical analysis by objective, informed parties who can identify model limitations and produce appropriate changes.” To do this, your validation team needs to be competent enough to challenge the model, have the ability to influence the decision makers involved in the selection or development of a model and incentivized to make an effective, competent challenge of the model.
Some opponents to the requirement of evaluating your model risk say that they have an adequate capital cushion to withstand any potential losses that would originate from model risk. Having a 15% net worth (or an allowance that is 5% of total loans) doesn’t alleviate the requirement to have sound business practices. A penny saved (or collected) is a penny earned.
– Dan Price, CPA
Twenty Twenty Analytics Blogger