Feature AI Risk
Your Override Rate Is 0.4%. The NAIC Now Wants to Know Why.
NAIC's AI Risk Evaluation Supplement v7.0 tests human-in-the-loop with override rates, reviewer authority, and documentation. Build the register.
Table of Contents
A 0.4% override rate is the kind of number that gets screenshotted into a board deck. Four-tenths of a percent of model decisions changed by a human reads as model quality: the machine is right, the reviewers agree, nothing to see.
The NAIC’s AI Risk Evaluation Supplement, version 7.0, treats that number as the start of a conversation instead of the end of one. The draft, posted as clean for adoption ahead of the Big Data and AI Working Group’s November 16, 2026 meeting, tells examiners to look at whether reviewers can actually override, whether overrides get documented, and whether the review design holds up. Override and escalation rates appear as evidence of oversight capacity, which means a very low rate is now a question you have to answer.
What the supplement is, and how it escalates
Four optional exhibits, applied in sequence. Exhibit A counts AI systems and models. Exhibit B examines the AIS program, as a narrative or a checklist. Exhibits C and D drill into high-risk models and the data feeding them. The document states the intent plainly: regulators will most often use Exhibit A, then escalate from the responses.
It’s supplementary by design. The NAIC Model Bulletin on AI Systems, adopted December 4, 2023 and picked up by roughly two dozen states by March 2025, supplies the expectations. The bulletin’s risk-proportionality factors already include “extent of human involvement in final decision-making.” Version 7.0 is the examiner-side instrument that asks you to demonstrate it.
If you read the pilot coverage from July, that piece dealt with building the inventory and governance exhibits. This one stays on the review evidence layer, which is where programs are thinnest.
The three factors, and what proves each
The general guidance defines human in the loop to cover both human involvement in a decision and human accountability without involvement in every decision — the agentic case. Then it splits the test:
| What regulators evaluate | The evidence that answers it | The failure mode |
|---|---|---|
| Authority — can the reviewer reverse, modify, or escalate, and are overrides documented with reasons? | Delegation memo with decision limits, system entitlements showing edit and hold rights, override register with reason codes, a sample of overrides | Reviewers have read-only screens or a four-eyes approval that fires after the batch posts |
| Qualifications — does the reviewer have the experience to judge the tool’s performance? | Role profile, training curriculum, competency check, model-literacy module, performance review criteria | Reviewer is the lowest-paid analyst in the queue with no exposure to validation findings |
| Capacity and evidence — does the design support oversight, shown through override and escalation rates? | Rate trend by model and by reviewer, escalation log, override sampling, reopened-item count | The rate is reported with no denominator, no trend, and no sample behind it |
Note the pattern: every factor resolves to an artifact. That’s the point of pairing each exhibit question with a documentation request — the examiner isn’t collecting your description of the control, they’re collecting something to compare the description against.
Why the override rate alone misreads
An override rate is a ratio with two moving parts, and only one of them is visible.
The denominator problem shows up first. A rate computed against decisions the reviewer touched is different from a rate computed against all model decisions, and the gap is largest exactly where oversight is thinnest — a queue with sampling skips most outputs before a human sees them. Report the denominator or the number means nothing.
The numerator is gameable in both directions. Require a reason code on every override and watch reviewers start resolving ambiguous cases as “approved” to avoid the write-up. Pair overrides with reopened items — decisions pulled back after the fact — and you get the correction signal the raw rate hides.
Then there’s the direction question. A 0.4% rate distributed evenly across ten models is not the same finding as 0.4% overall with a 3% rate on the claims model and zero everywhere else. Report by model, by reviewer, and against a stated tolerance. If your tolerance is a starting point rather than a benchmark, label it and calibrate against the last six months of your own history before anyone outside your team reads it.
A realistic hypothetical: the decline queue carries a 0.2% override rate, the claims-triage queue sits at 2.8%, and the underwriting referral queue has recorded zero overrides in nine months. Under a blended metric the program looks consistent. Broken out, the underwriting queue is the finding — a reviewer with no recorded override in three quarters either has a perfect model or a queue nobody is really reading.
The register that survives the question
Override documentation is the load-bearing artifact. Minimum fields:
- timestamp and batch or decision ID
- model, version, and the output presented to the reviewer
- reviewer ID and the authority basis for their edit or hold
- evidence the reviewer considered — document, call note, photo, or data element
- the decision: accept, modify, reverse, or escalate
- reason code plus free-text rationale for anything other than accept
- escalation target and resolution, when escalated
- downstream effect — what changed for the consumer, including notice timing
Two design choices determine whether this works. Put reason codes on the record at the moment of decision rather than in a weekly summary, and make escalation a first-class outcome alongside override. A reviewer who can escalate but never does is a different problem from one who can’t. The 45-day human review queue clock some states are now writing into rule puts the same pressure on the same artifact from a different direction: if the register can’t show a decision date, you can’t prove the clock was met.
Authority evidence sits one layer up. A delegation memo that names the decision classes each role may change, system entitlements that actually permit a hold or edit, and a named escalation contact with a response window. If the workflow requires a supervisor to release every hold, say so — and measure how long releases take.
Corroboration is the whole design
Version 7.0’s most consequential shift isn’t a new question; it’s the pairing. Requests for supporting documentation now sit next to the questions about AI use, governance, testing, and human review, so the response package has to include records.
That changes what a good response looks like. Management’s narrative gets an evidence column beside it:
| Assertion | Artifact |
|---|---|
| ”Human review before adverse action” | Two-page sample of adverse decisions with reviewer ID, timestamp, and the evidence considered |
| ”Reviewers can override” | Entitlement export plus three override records with reasons |
| ”Reviewer training is current” | Curriculum, completion dates, and the competency check instrument |
| ”Validation is independent” | Reporting line chart and the last validation report’s sign-off page |
Where the artifact doesn’t exist, the honest response is a gap with a remediation date. Examiners have seen enough programs to know which one of those is real.
The third-party model question is new
Version 7.0 adds a question for third-party models: how the insurer obtains enough evidence to monitor the risk, and how it learns about material changes or incidents. Read that as a contract question. Notification obligations for model changes, retraining, and incidents; the right to validation evidence and drift reporting; and a named contact on the vendor side.
Pair it with the second addition — what material changes trigger approval, revalidation, or enhanced monitoring. Build the trigger list: retraining event, input schema change, new line of business, drift beyond tolerance, vendor release note, regulator inquiry. A trigger list without a vendor notice clause is a list you find out about too late to use. The decommission sequence for retired models is the other end of the same lifecycle control.
What you can stop preparing
Version 7.0 narrowed in places, and over-preparing wastes the window:
- Exhibit A dropped the materiality-threshold field, and an existing AI inventory by business function may satisfy the request if it carries system name, use, program area, and inherent risk level.
- Exhibit B no longer requires a detailed subject-matter-expert roster.
- Exhibit C dropped the standalone agentic AI question and the proxy-relationships testing question.
- Exhibit D is limited to data used in high-risk models.
- Regulators may exclude GLMs, rule-based systems, deterministic algorithms, and fixed formulas where ratemaking oversight already covers them.
One unresolved item worth knowing: the definition of Adverse Consumer Impact still omits a qualifier that the impact must violate a regulatory standard, which trade groups argued could sweep in lawful non-renewals and premium increases. Write your own boundary and cite your reasoning rather than waiting for the definition to settle.
So What?
Human-in-the-loop was a design claim for three years. The supplement turns it into an evidence request with three parts, and the override register is the piece most programs are missing. Start there: pull ninety days of decisions from your highest-volume model, count the overrides, and see whether you can produce reasons for them. If you can’t, the number in the board deck is a claim, not a finding.
The AI Risk Assessment Template has the inventory and risk-tiering structure that Exhibit A and B responses map onto.
FAQ
Do the exhibits apply if we only use vendor models? Yes. Exhibit B covers program governance regardless of who built the model, and v7.0’s third-party question assumes you hold monitoring evidence you don’t generate. Start with the vendor contract.
How long should the override register be retained? The supplement defers to the requesting state’s confidentiality authority and existing handbooks rather than setting a period. Match your model documentation retention, and keep the register contemporaneous — a reconstructed register reads like one.
Is a high override rate evidence of a bad model? No. It’s evidence of an active queue. The paired question is whether overrides cluster on identifiable defects that fed back into retraining. A high rate with a flat defect log means reviewers are compensating for something nobody is fixing.
◆ Need the working template?
Start with the source guide.
These answer-first guides summarize the required fields, evidence, and implementation steps behind the templates practitioners search for.
◆ Related template
AI Risk Assessment Template & Guide
Comprehensive AI model governance and risk assessment templates for financial services teams.
◆ Immaterial Findings · Weekly
Sharp risk & compliance insights. No fluff.
◆ FAQ
Frequently asked questions.
What is the NAIC AI Risk Evaluation Supplement?
How does v7.0 test human-in-the-loop review?
Is a low override rate bad?
What got easier in v7.0?
Does this apply to banks and fintechs?
Author
Rebecca Leung
Rebecca Leung has 8+ years of risk and compliance experience across first and second line roles at commercial banks, asset managers, and fintechs. Former management consultant advising financial institutions on risk strategy. Founder of RiskTemplates.
◆ Related framework
AI Risk Assessment Template & Guide
Comprehensive AI model governance and risk assessment templates for financial services teams.
◆ Keep reading
Related posts.
AI Risk
AI Risk Tiering Just Flipped: OCC 2026-54 Ranks Consequence, Not Sophistication
OCC Bulletin 2026-54 puts a rules-based payment agent in Tier 1 and your frontier model in Tier 4. Build AI risk tiering on agency and reversibility.
Nov 20, 2026
AI Risk
Colorado's Draft ADMT Rules Put a 45-Day Clock on Human Review
Colorado's interim ADMT rules would require acknowledging human-review requests in 10 days and finishing in 45. Here's the review queue to build by Jan. 1.
Nov 19, 2026
AI Risk
Your Agent Is Logged In as a Service Account Nobody Owns
SR 26-2 pushed agentic AI out of model risk scope. AI agent access control now needs an entitlement register, JIT credentials, and a named owner.
Nov 18, 2026