CECL | Model Validation | MRM | MRM Framework
How CECL Drove Model Risk Management into the Spotlight
CECL model development stole the show in 2022, but 2023 will be the first time many institutions will put the new model into production. Congratulations on making it this far on your CECL journey, and best of luck with your Q1 implementations!
Model Risk Management (MRM) is now rising into the spotlight as a byproduct of CECL. For those with an MRM framework in place, onboarding CECL and incorporating it into a broader risk management program should be a straightforward path. However, those without an MRM program are finding the onboarding process more difficult—they are missing crucial documentation and controls, among other broader risk-mitigating processes.
The benefits of implementing an MRM program may not have been apparent before. Still, as the first wave of CECL validations begins, many of the resulting findings could have been addressed had a working MRM program been in place before developing the model.
But do not be discouraged! Now is precisely the right time to forge ahead. You can help ensure a smoother journey and long-term benefits by:
Gathering supporting documentation for key management decisions, model assumptions, and the modeling methodology
Creating formalized and repeatable data and processing controls around the model
Ensuring the CECL model performs as intended and captures the underlying credit risk associated with lending activities through validation
Taking stock of your MRM progress and making it a priority to “right size” it for your organization in 2023
If these action items cause concern that you “aren’t ready for prime time,” don’t panic: it may indicate that you need to add a few CECL and MRM resolutions for 2023 to your list.
Resolution #1: Get Your CECL Model Validated
Model validation is a critical component of the MRM process. During validation, a qualified independent party should review the model and all its parts to confirm that it is working as intended and appropriate for use as an ACL model. For CECL, the scope and extent of the validation will depend on the methodology used, a portfolio's size and complexity, and the availability of information to validators.
During the validation, the model owner should be able to provide the validator with an end-to-end model run, model documentation, controls, and other governance-related material. Validators should review the input data, the processing component, and the model output and assess that it aligns with the model documentation and utilizes an appropriate approach for estimating a reserve. Additionally, model validators should review key modeling assumptions such as prepayment rates and average life estimates, for which the model owner should have supporting evidence and documentation. For example, a prepayment study may support prepayment rate assumptions.
If possible, the model owner should also supply the validator with testing materials such as assumption sensitivity testing, parallel runs with the ALLL model, and back-testing. Complete model validations will include a review of performance monitoring procedures that are in place.
Resolution #2: Create Remediation Plans for Validation Findings
If a CECL model has already undergone rigorous validation, the next step is to ensure that the institution’s team understands any findings and recommendations. The Model Risk Manager should track such findings, capturing management responses from the CECL model owner and formalizing a remediation plan to demonstrate that the institution is actively trying to fix deficiencies in the model and the modeling process. This is the first critical step in smoothing the CECL road ahead.
Based on DCG’s numerous validations of CECL models, common areas of improvement include:
Model risk management
Model governance and controls
Inadequate documentation and assumption support
The OCC[1] and the FDIC[2] have updated their examiner handbooks to include specifics related to model risk management. This includes creating a model inventory, risk rating models, and developing a policy to outline a program for the first, second, and third lines of defense.
Resolution #3: Make Sure CECL Is in Your Model Inventory
A model inventory is one of the first documents to develop for a Model Risk Management program. Each institution uses model inventories differently; the fields tracked will vary by use. Primary fields are collected for institutions that keep an inventory to track what models are in use, under development, or recently retired. For some models, there may be several variations of the same model with different underlying assumptions/data/uses that would merit separate validations and, thus, should be tracked as a separate model in the inventory.
The OCC has published a list of items that examiners may look for in the model inventory:
Source: United States, Congress, Office of the Comptroller of the Currency. Comptroller's Handbook, 1.0 ed., OCC, 2021, pp. 27–27. Model Risk Management. |
The OCC has comprehensive expectations, whereas the FDIC has historically been less prescriptive. If yours is an FDIC-regulated institution, fewer fields may potentially be sufficient. An institution may track additional fields that it finds helpful for overseeing a risk management program, but those noted above may suffice to operate an effective risk management program.
The information collected from each model provides immediate insight into the model's purpose, use, basic construction, risks, and updates on primary risk management aspects such as validations, exceptions to policy, and an indication of the model's overall performance. By reviewing the model inventory, management can quickly identify which models are out of compliance with the risk management program, are developing issues, or may need expanded review to mitigate any emerging risks.
The model inventory is only valuable when it is up to date and contains accurate information from a practical standpoint. All models under development, in use, or recently retired should be tracked. The model inventory should be the responsibility of a sole position or person at the institution rather than a group or committee for accountability reasons. It is often easier to hold a singular person accountable for inventory maintenance than a group. CECL helped bring the importance of model inventories to the forefront.
Resolution #4: Risk Rate All Models (CECL Is a High-Risk Model!)
Once an institution has identified all of its models, it should prioritize them, noting which models pose the most model risk. This is done through a risk assessment process. When constructing a risk assessment process, several aspects of the model that affect its overall risk level must be evaluated. The risk assessment should consider the model's use and its materiality on the organization. The assessment should consider the amount of uncertainty generated by the model's inputs, outputs, or processing components. If the model is part of a suite of models, that may affect its overall risk to the institution. Other aspects to consider are the model's limitations, the frequency of use, the complexity of the model, the resource requirements of the model, and the expandability of the underlying theory/algorithm that drives the model.
There are multiple ways to create a risk assessment process. Some institutions use a definition-based approach by creating definitions for each risk level (typically high, medium, and low), highlighting aspects of the model that categorize it at each level. The institutions that implement this approach tend to use a committee to review each model, in conjunction with the model owner, to arrive at a final risk rating. The process is documented through minutes, so outsiders such as regulators or validators can understand how the rating was determined. Others decided to use a purely quantitative approach for risk rating, utilizing a scorecard or template. This approach takes the various risk categories (model complexity, use, operational impact, etc.) and applies ratings and weightings to arrive at a quantitative risk rating for each model.
Regardless of how an institution risk rates its models, the process should be easily understood and repeatable. The model risk management policy or procedures manual should describe in detail how the risk rating is performed, the criteria, and exception handling. Risk ratings help inform the frequency and depth of model risk management practices, such as the frequency and rigor of model validations, management oversight, and depth of the effective challenge. That said, CECL is a high-risk model, so validation should be on your radar for 2023!
Resolution #5: Create a Model Risk Management Program
Once models and their associated risks are identified, the institution can begin constructing a model risk management program proportional to that risk, the complexity of the institution, and its culture.
The policies (and, if applicable, accompanying procedures documents) should outline the governing structure, roles, responsibilities, model inventory standards, processes, and the risk rating process that has been determined up to this point. The policy and procedures should then describe the criteria and processes related to controls, data management, documentation, performance monitoring, development, implementation, model use, validation, exceptions, and reporting. A model risk management policy should include the definition of the model and model risk.
As mentioned earlier, those organizations that had an MRM program in place had an easier time implementing CECL. The standards in a proper MRM program help ensure proper model development, adequate documentation, testing, controls, and proper governance. These items are necessary to pass a model validation and effectively use the model’s results. MRM, when structured appropriately, can provide significant benefits for your institution.
BONUS Resolution: Don’t Be Afraid to Ask for Help
As CECL continues into its next phase, take a moment to evaluate where you are and what remains to be done. As always, DCG stands ready to help you along the journey, whether it’s helping to finalize CECL implementations, providing validation, or creating an MRM program right-sized for your organization.
For additional perspective on CECL and MRM, please visit DCG’s MRM webinar archive or our CECL resources.
For more information, contact Darling Consulting Group at: DarlingConsulting.com/contact
ABOUT THE AUTHOR
Hannah Glasrud
Risk Management Consultant, Darling Consulting Group
Hannah is a Risk Management Consultant at Darling Consulting Group, working primarily in the realm of model validation and model risk management consulting. Throughout her career, she has worked with various models, including CECL, DFAST, capital, credit, and default rate models. She also performs Model Risk Management process reviews and consulting engagements.
Hannah began her career at DCG in the Data Analytics Group as a business systems analyst, performing prepayment and deposit studies for community banks. From there, she was promoted to Quantitative Analyst, where she assisted in model validations and model risk management engagements.
Using her technical knowledge and coding experience, she has and continues to produce automated reporting tools for various areas within the company and performs code reviews for models written in SAS, R, and Python.
Hannah earned her Bachelor’s degree in economics from Boston University. She has also earned a MicroMasters Certificate from Georgia Tech in analytics: essential tools and methods.
© 2023 Darling Consulting Group, Inc.
