How Can Credit Unions Evaluate the Accuracy of AI Lending Models?

How Can Credit Unions Evaluate the Accuracy of AI Lending Models?

The short answer: Credit unions evaluate the accuracy of an AI lending model by testing five things, not one. Can it rank good borrowers above risky ones? Are its risk estimates honest when compared with real outcomes? Does it hold up on loans from a later time period it never saw? Does it beat your current process? And is it fair and explainable across member groups? Test on your own historical loans, run the model quietly alongside your lenders before it decides anything, and keep monitoring it after launch.

A vendor puts up a slide in your boardroom. Big blue number in the middle: 94% accurate.

Heads nod. Then one board member, the quiet one who reads every packet, asks a simple question.

"Accurate at what?"

The room goes silent for a second. The vendor starts talking about machine learning. Your CLO looks at the ceiling. Somebody suggests moving on to the next agenda item.

That moment is more common than anyone admits. Credit unions are adopting AI lending tools fast, but most teams were never taught how to judge whether one of these tools is actually any good. And "accuracy," the word on every slide, turns out to be one of the least useful words in the conversation.

So let's fix that. This guide walks through how to evaluate an AI lending model in plain language, what to ask for, what you can test yourself, and how to keep checking after go live. No statistics degree required.

Why Is a Single Accuracy Number Misleading?

Because most loans get repaid.

Imagine that 95 out of every 100 loans in your portfolio are repaid as agreed. Now picture the laziest possible model, one that just approves everybody. It would be right 95 percent of the time. That is a "95% accurate" model, and it is completely useless, because it never catches a single risky borrower.

This is the trap. A headline accuracy figure can look impressive while hiding the thing you care about, which is how well the model handles the small share of loans that go bad, and whether it treats members fairly while doing it. (The 95 percent here is an illustration, not your portfolio's rate.)

Wipfli's 2026 State of the Credit Union Industry report, which surveyed 100 credit union executives, found that 47 percent listed data readiness, quality and governance as a barrier to AI. That is a clue that many teams are still building the muscle to judge these tools well, and it is a good reason to learn the questions before the demo, not after.

What Are the Five Tests of an AI Lending Model?

Think of it like checking out a used car. You do not just ask "does it run?" You look at the engine, the brakes, the history, and how it handles on a road you actually drive.

For a lending model, there are five checks. Each one answers a different question, and a model needs to pass all of them.

Test 1: Can It Rank Borrowers Correctly?

This is called discrimination in the model world, though it has nothing to do with unfair treatment. It simply asks whether the model gives riskier scores to people who go on to have trouble, and safer scores to people who repay.

You will see three common measures:

• AUC (area under the curve). Pick one borrower who repaid and one who defaulted at random. AUC is the chance the model scored the defaulter as riskier. A score of 0.5 is a coin flip. A score of 1.0 is perfect.
• Gini. A rescaled version of AUC, calculated as two times AUC minus one. An AUC of 0.75 equals a Gini of 0.50.
• KS statistic. The biggest gap between the score distributions of good and bad borrowers. A bigger gap means better separation.

The catch is that there is no universal pass mark. A "good" number depends on your products, your members and your data. A model that looks great on prime auto loans may look ordinary on thin file personal loans. So the number that matters is not the one on the vendor's slide. It is how the model compares with your current scorecard, run on your loans.

Test 2: Are Its Risk Estimates Honest?

A model can rank people beautifully and still be wrong about how risky they are.

This is calibration. If a model says a group of applicants has a 5 percent chance of default, then about 5 out of every 100 of them should actually default. If 12 do, the model is overconfident. If 1 does, it is too cautious, and you are turning away good members.

Calibration matters more than people expect, because those percentages feed real decisions: pricing, approval cutoffs and loan loss reserves. A model with great ranking and poor calibration can quietly misprice a whole portfolio.

The check is simple. Group applicants into score bands, and compare what the model predicted with what actually happened in each band.

Test 3: Does It Hold Up on Loans It Has Never Seen?

Any model looks smart on the data it learned from. The real question is whether it works on new loans.

The gold standard is an out of time test. Train the model on loans from one period, say 2021 through 2023, and then test it on loans from a later period it never saw, say 2024. If performance holds, you can trust it more. If it falls off a cliff, the model learned quirks of the past instead of patterns that last.

Federal banking guidance on model risk management, known as SR 11 7, describes the same idea. It calls for backtesting, which compares actual outcomes with model forecasts over a sample period not used in development. It also notes that validation matters most in benign conditions, when risk estimates can become overly optimistic.

One honest complication: loans take time to show their true colors. A 60 month auto loan cannot give you a full verdict next quarter. That is why many teams watch early signals, such as first payment defaults and how loans from each "vintage" (the month they were originated) perform over time.

Test 4: Does It Beat What You Do Today?

A model does not need to be perfect. It needs to be better than the alternative, at a risk level you are comfortable with.

So compare it against what you already do. SR 11 7 calls this benchmarking, comparing a model's inputs and outputs with estimates from alternatives. In practice, it means answering questions like these:

• On the same set of historical applications, would the model have approved more good loans at the same loss rate?
• Where the model and your lenders disagree, who turned out to be right?
• Does it help members your current scorecard struggles with, such as thin file borrowers?

The best way to test this safely is to run the model in the background first. We cover how to do that in the pilot section below, and our guide to champion and challenger testing and supervised AI explains how it fits into a governed rollout.

Test 5: Is It Fair, and Can It Explain Itself?

Accuracy that is not fair is not worth much, and it can get you in legal trouble.

A model can be accurate overall and still produce worse outcomes for certain groups of members. Fair lending testing looks at this directly:

• Compare outcomes across groups. Look at approval rates, denial rates and error rates for different groups. One common screening measure is the adverse impact ratio, where a group's approval rate below about 80 percent of the highest group's rate (the "four fifths rule") flags a closer look. Treat it as a warning light, not a legal safe harbor.
• Look for proxies. A variable that seems neutral, like a ZIP code or a type of device, can stand in for a protected characteristic. Check whether removing a variable changes the gap.
• Search for less discriminatory alternatives. This means asking whether a different version of the model could do the same job with fewer disparities, and writing down what you found.

That last point is not theoretical. A January 2025 CFPB report on examination findings found lenders that did not adequately search for less discriminatory alternatives in their scoring models, did not review thousands of model inputs for fair lending risk, and kept inputs that contributed to disparities without documenting a business justification. Whatever the current federal supervisory climate, those are exactly the questions a state regulator, an examiner or a plaintiff's lawyer will ask.

Be aware of one practical hurdle. For most consumer loans, lenders are restricted from collecting race, ethnicity and similar information from applicants, so fair lending testing often relies on estimation methods. Work with your compliance team or counsel on the right approach for your institution.

Then there is explainability. When a loan is denied, the Equal Credit Opportunity Act and Regulation B (12 CFR 1002.9) require specific principal reasons. A model that cannot produce accurate reasons fails this test no matter how good its numbers are. The same CFPB report found auto lenders that had never validated that their process for selecting adverse action reasons actually produced accurate results. So test the reasons too: pick declined applications at random and check that the reasons given match what the model truly used. We explain how reason codes work in our guide to loan decisioning software.

What Data Do You Need to Test a Model Properly?

Less than you think, but it has to be clean.

• Historical applications and their outcomes. Approved loans, how they performed, and what happened to the bad ones.
• A clear definition of "bad." Is it 60 days late, 90 days late, or a charge off? Pick one and stay consistent, or your results will not mean much.
• Time stamps. You need to know when each loan was originated, so you can run out of time tests.
• Decisions and overrides. What your lenders decided, and when they overrode the system.

Here is a subtle problem worth knowing about. You only see how the loans you approved performed. You never learn what would have happened to the applicants you declined. This is called reject bias, and it means your own history makes any model look slightly better or worse than it really is. Good validation teams account for it, and it is a fair question to put to a vendor.

How Do You Keep Checking After Go Live?

A model is not a one time purchase. Member behavior changes, the economy changes, and the model's view of the world slowly goes stale. This is called drift.

A simple monthly check covers most of what matters:

• Score distribution. Are applicants scoring differently than they used to? A common measure is the population stability index, where a value under 0.1 is generally seen as stable and above 0.25 as a major shift. These cutoffs are a widely used rule of thumb, not a regulation.
• Approval and referral rates. Sudden moves deserve a question.
• Early delinquency by vintage. Are recent loans behaving the way the model predicted?
• Override rates. If lenders keep overriding the model in the same direction, something is off.
• Fairness metrics. Repeat the group comparisons regularly, not just at launch.

Decide in advance what triggers action. For example, a drop in performance beyond an agreed threshold means a review, and a review means a decision to adjust, retrain or pause. Name an owner for every alert.

What Do Regulators Expect?

The honest answer is that expectations are real, but the detailed rulebook for credit unions is thinner than for banks.

NCUA's published position, on an AI resource page last updated in April 2026, is that it supervises AI within its existing framework. Credit unions are expected to identify risks that are unique to AI tools, monitor and measure them regularly, and put controls in place to reduce operational, compliance and security risks. When you use a vendor's AI, you must conduct appropriate due diligence on how the product works, the risks it introduces and the vendor's safeguards, and the board and management must ensure proper oversight.

At the same time, a May 2025 report from the Government Accountability Office found that NCUA's model risk management guidance is limited in scope and detail compared with other banking regulators, and that as of March 2026 NCUA's AI resource page still did not provide detailed model risk management guidance. In plain terms, less detail from the regulator puts more responsibility on your own homework.

A sensible template is the structure of SR 11 7, even though it was written for banks, not credit unions. It rests on three pillars: evaluating whether the model is conceptually sound, monitoring it on an ongoing basis, and analyzing its outcomes against reality. It also states that this applies equally to models built in house and to those bought from vendors, so a vendor's assurance is never a substitute for your own validation.

What Should You Ask an AI Lending Vendor?

Bring these questions to your next demo. A confident vendor, including us, should welcome every one of them.

• How was the model validated, and can we see the validation documentation?
• What do the ranking measures look like on an out of time sample, and on portfolios similar to ours?
• How well calibrated are the risk estimates, and how often is that rechecked?
• What data was it trained on, and over what time period?
• How was fair lending tested, and did you search for less discriminatory alternatives? Can we see the results?
• How are adverse action reasons generated, and how are they validated?
• What monitoring and drift alerts do we get, and who acts on them?
• How are changes to the model controlled, and how are we told about them?
• Can we backtest the model on our own historical loans before we commit?
• Can an independent party validate it, and what will you share under NDA?

Our guide on how to evaluate AI powered lending platforms goes through the wider checklist, including architecture, governance and how the platform fits with your core.

Want to see what a decision with its reasons looks like in a real workflow? You can try the interactive Algebrik demo and click through at your own pace, no sales call needed. It is a useful way to test whether a vendor can actually show its work.

How Can You Run a Safe Pilot on Your Own Loans?

You do not have to take anybody's word for it. Here is a practical five step plan that gives you your own evidence.

• Backtest on your history. Give the vendor a set of past applications with the outcomes hidden, ask for scores, then compare against what really happened. Use a later time period for the test.
• Run in shadow mode for 60 to 90 days. Let the model score live applications in the background without changing a single outcome, and record what it would have done.
• Study the disagreements. Where the model and your lenders split, look at each case. This is where you learn the most. Was the model catching something people missed, or missing context people had?
• Check fairness and reasons. Run the group comparisons on the shadow results, and review a sample of the reasons the model would have given.
• Start narrow. Go live on one product, with a wide referral lane so people see more files at first. Narrow the lane as the evidence builds.

That last step ties in with the rest of your AI plans. Our breakdown of what is hype and what moves the needle in AI for credit unions suggests measuring results over 90 days before expanding, which fits well here.

What Mistakes Do Credit Unions Make When Judging AI Models?

• Accepting one accuracy number. Always ask what it measures, on what data and compared with what.
• Testing on the training data. A model always looks good on loans it learned from.
• Ignoring calibration. Great ranking with bad probabilities leads to bad pricing and reserves.
• Judging only the approved loans. Remember the hidden outcomes of the loans you declined.
• Validating once and forgetting. Models drift, so monitoring has to be a habit.
• Testing fairness only at launch. Run it again whenever the model or the member mix changes.
• Treating the vendor as the validator. You own the risk, so you own the evidence.

No named owner. If everyone is responsible, nobody is.

The Takeaway

The next time someone says an AI lending model is 94 percent accurate, you will know what to ask.

Accurate at ranking, or at honest odds? On old data, or on loans it has never seen? Better than what we do today? Fair across our members, and able to explain itself when it says no?

Test it on your own loans. Run it quietly beside your lenders before it decides anything. Study every disagreement. And keep watching after launch, because a model that was right last year can quietly drift.

Do that, and "accurate" stops being a word on a slide and becomes something you can prove to your board, your examiners and your members.

If you would like to see how a decision, with its reasons, comes together in a real lending workflow, explore the interactive demo at your own pace.