Since the process of credit decision-making encompasses careful evaluation of risk assessment, credit scoring, and scrutiny of financial statements, you might wonder how to filter out the best-performing strategies through comparison of multiple decision-making algorithms in real-time.
Enter Canary Testing and Champion/Challenger models – a dynamic duo in Business Rules Engine (BRE). Let’s dive into how they redefine credit decision-making and how they pave the way for more informed and adaptive credit assessments.
A Deep dive into Canary Testing and Champion/Challenger Features in Business Rules Engine
Let us take a deep dive into the concepts of Canary Testing and Champion/Challenger and their use cases.
What is Canary Testing?
In digital lending, Canary Testing refers to the method of testing new strategies under-real world conditions to determine how they will perform in a live environment. By following this method, risk teams can assess new possibilities, ensuring minimal to no impact on users.
For example, if you’re a lender partnering with a digital lending platform, you might encounter a relatively new user base. You may face several challenges such as:
- Inaccurate historical data that may or may not be representative of the new user base
- Inefficient and time-consuming trial and error practices
- Inapplicable vintage analysis due to the new loan origination channel.
Canary testing helps solve these challenges by allowing you to run a variation of an existing or new workflow in the live production environment without disrupting your daily operations.
Canary Testing Use Cases:
- Performance Monitoring: Lenders can assess the effectiveness of new credit policies by comparing the results of a small (canary) group with the broader borrower population. Using the results, lenders can make data-driven decisions regarding which policies to adopt or remove.
- Incremental Improvements: This is an ongoing approach to policy development. Lenders can adapt to changing market conditions and borrower behaviours by continuously testing and refining strategies based on real-time feedback.
- Risk Mitigation: If a new policy tends to have higher default rates or risks that can impact borrowers, canary testing can help implement new strategies, removing negative outcomes. Lenders can also revert back to the previous policy, thus mitigating risks and protecting their portfolio from potential losses.
Canary Testing Challenges and Considerations
Though canary testing has its benefits in helping refine credit decision-making, it also brings in some challenges that lenders have to deal with for successful policy testing. Some of them include:
- Data Privacy and Compliance
|
The Challenge |
The Solution |
|
Lenders must ensure that sensitive customer data used in canary testing complies with data privacy regulations such as the GDPR and CCPA. Digital lending guidelines issued by the Reserve Bank of India (RBI) are also mandatory to be followed by lenders, which include policies such as: 1] Collecting borrower data on a need-basis with their approval and an audit record. 2] Refraining from accessing additional data stored in their devices such as call logs, contact information, etc. 3] Ensuring biometrics of the borrowers are not stored in lending apps. |
Implementing robust data encryption and anonymization practices and conducting regular audits will help ensure maximum compliance. |
- Implementation Complexities
|
The Challenge |
The Solution |
|
Integrating new models and algorithms into existing systems can make setting up and managing canary tests more complex. |
In this case, automation tools can help streamline the setup and execution of canary tests. |
- Resource Allocation and Cost
|
The Challenge |
The Solution |
|
Performing a successful canary test requires additional resources such as technology infrastructure with data management capabilities and specialized personnel. |
Leveraging cloud-based solutions can be beneficial in scaling canary tests and reducing infrastructure costs. |
|
The Challenge |
The Solution |
|
Scaling canary testing to meet large volumes of transactions and diverse customer segments can be difficult. |
This problem can be addressed by designing modular canary tests and utilizing scalable cloud services to process large datasets and transactions. |
What is Champion/Challenger Testing?
In the realm of digital lending, the Champion typically refers to the tried-and-tested lending policy and the Challenger refers to testing multiple new strategies in a controlled environment to determine if you can enhance the accuracy or approval rates of credit assessments. This approach can be best-described as a litmus test for financial strategies, also popularly known as A/B Testing and can reveal how alterations in credit evaluation tactics impact overall decisioning outcomes.
Champion/Challenger Testing Use Cases:
- Risk Assessment: Champion/Challenger testing can be used to compare different risk assessment models. Financial institutions can test new models against an existing model to evaluate which model predicts creditworthiness more accurately.
- Product Offerings: Champion/Challenger testing can be useful when offering new financial products or credit terms. Financial institutions can use this model to identify which offer resonates better with customers in terms of acceptance rates, default rates, etc.
- Customer Experience: Champion/Challenger testing can be used to enhance customer experiences as well. This is done by testing new application processes or customer-facing policies to improve user experiences.
Champion/Challenger Testing Challenges and Considerations:
Despite being a powerful tool for optimizing credit decision-making, even champion/challenger features come with their own challenges. Some of them include:
|
The Challenge |
The Solution |
|
Credit decision-making models often incorporate multiple variables and data points, making it more complex. Adding to that, comparing challenger models and champion models can be difficult when the models are sophisticated and include machine learning algorithms. |
This problem can be solved by simplifying models and using robust platforms that can facilitate seamless model comparison and highlight key performance metrics effectively. |
|
The Challenge |
The Solution |
|
Measuring the performances of the champion and challenger models and determining their statistical significance can be a challenge, especially with limited sample sizes or inaccurate data. |
Appropriate statistical tools and techniques that can conduct hypothesis testing can be used in this situation. Additionally, the sample sizes can also be enhanced to improve the scope of the tests. |
|
The Challenge |
The Solution |
|
Garnering the support of stakeholders will be difficult, especially from the ones who’ve been accustomed to existing policies and workflows. They may be resistant to change and reject new models, unless it can help bring in profits. |
To better convince stakeholders, pilot results and case studies can be demonstrated to the stakeholders to help them understand the benefits and rationale behind the testing. |
|
The Challenge |
The Solution |
|
Champion/Challenger testing is a resource-intensive process that requires a significant amount of data storage, computational power, and specialized personnel for setup, monitoring, and analysis. |
To solve this issue, cloud-based solutions and automation tools can be leveraged to streamline the testing process and scale technology infrastructure. |
Benefits of Using Canary Testing and Champion/Challenger in Credit Decision-making
Leveraging the power of canary testing, you can build your own Champion/Challenger policy accordingly and introduce it to a subset of your borrowers. The insights derived from the process can be used to enhance your policy and progressively expand its implementation to a larger borrower base. The combined power of canary testing and challenger features will offer rich analytics that can help you mitigate the risks that can have a drastic impact on business performance and customer experience.
Let us look at a few advantages of how canary testing and Champion/Challenger models in business rules engine benefit your credit decision-making processes:
Fraud & Identity Verification
Back in FY23, the number of complaints against digital lending apps in India rose to 1,062 on account of fraudulent apps despite the crackdown by the Government and the Reserve Bank of India (RBI). However, even digital lenders also are victims of online fraud, which can leave them facing severe financial implications. To tackle frauds, lenders perform tough screening processes, which leads to rejection of good customers as well.
This is where BRE features such as canary testing and champion/challenger features come to the rescue to help strike a balance between fraud prevention and borrower satisfaction. You can use canary testing to evaluate the impact of test parameters, measure the relevance of specific data sources, or assess scope of related drivers.
Once you understand what could work, you can randomly test one Champion and five Challenger versions of the workflow in the live environment to determine which version will help you identify the impact of fraud rates and borrower satisfaction.
You can also quantify the trade-offs between customer experience and fraud losses. For instance, if you’re looking to increase loan application completion rates without the chances of frauds impacting borrowers, you can try out different identity verification workflows. This will help gauge the impact on fraud rates and drop offs by analyzing the genuinity of an individual through selfies, biometrics, and government-issued documents.
In the end, having a BRE with canary testing and champion/challenger features is like having a lending autopilot system. You can set monitors and get real-time analytics whenever required, thus giving you a peace of mind when dealing with critical data.
You may also like: Frauds in digital lending apps : https://cloudbankin.com/loan-management/how-to-identify-fraudulent-digital-lending-apps-types-of-lending-frauds-and-their-impact/
Underwriting
One of the significant challenges present in underwriting is that credit risk strategies are unlikely to interest senior management unless it involves profit maximization with limited risks. Even after the shift in focus from risk to profit is made, projected incremental gains can be overshadowed by big ideas and the challenge mainly lies in showing how small improvements can cumulatively lead to big changes.
Thus, canary testing and champion/challenger features become critical as they can accurately assess the future impact of a strategy change. Credit risk teams will be able to gather real-time marginal performance data in a series of controlled experiments that can deliver results on the impact of the strategy when rolled out on scale.
For example, if a bank’s credit risk team is looking to change their credit card approval strategy to increase acceptance rates, they can implement canary testing and champion/challenger models over a small, controlled group. By gathering real-time data on metrics such as default rates and customer retention from both testing modules, risk teams can use the results to analyze if the new strategy will work and how it will impact the larger population.
Improved Risk Management for Borrowers
Both canary testing and champion/challenger features provide a robust framework for minimizing the risks faced by borrowers in credit decision-making. Testing new credit policies and strategies in live environments facilitate data-driven decision-making that offer more accurate assessments of a borrowers’ risk. Lenders can also identify flaws in the policies that put their entire borrower base to risk.
The combined powers of canary testing and champion/challenger features also promote continuous testing and refinement of new models, allowing lenders to adapt to market changes and ensuring their risk management strategies remain effective and unaffected by market changes. Through real-time data from BRE features, lenders can also conduct performance evaluations, leading to timely adjustments in credit policies.
Enhance Your Credit-Decision-making Efficiency With Loan Origination Systems
By reaping the benefits of canary testing and champion/challenger models, you can revolutionize credit decision-making through real-time insights and live testing of new credit policies and strategies. Ultimately, the goal is to minimize risks for both borrowers and yourself as a lender. These methodological approaches to testing and validation will help boost innovation and continuous improvement in credit risk management.
You can also leverage CloudBankin’s Loan Origination System (LOS) as a testing ground for analyzing the creditworthiness of your borrower base and help set up proper workflows to process loans. Using the LOS’ automation and data analytics capabilities, you can run different canary tests and champion/challenger models to obtain relevant performance metrics such as approval rates, default rates, and processing times. To know more about our Loan Origination System’s capabilities, book a demo today.