Skip to content
RiskTemplates · The Daily Brief Sunday, November 22, 2026
Wire Backward Hat Capital's $600 Million 'Audited' Forex Fund: What Manager Due Diligence Actually Missed NOV 19

Feature AI Risk

Your Override Rate Is 0.4%. The NAIC Now Wants to Know Why.

NAIC's AI Risk Evaluation Supplement v7.0 tests human-in-the-loop with override rates, reviewer authority, and documentation. Build the register.

By Rebecca Leung · November 22, 2026 ·
Table of Contents

A 0.4% override rate is the kind of number that gets screenshotted into a board deck. Four-tenths of a percent of model decisions changed by a human reads as model quality: the machine is right, the reviewers agree, nothing to see.

The NAIC’s AI Risk Evaluation Supplement, version 7.0, treats that number as the start of a conversation instead of the end of one. The draft, posted as clean for adoption ahead of the Big Data and AI Working Group’s November 16, 2026 meeting, tells examiners to look at whether reviewers can actually override, whether overrides get documented, and whether the review design holds up. Override and escalation rates appear as evidence of oversight capacity, which means a very low rate is now a question you have to answer.

What the supplement is, and how it escalates

Four optional exhibits, applied in sequence. Exhibit A counts AI systems and models. Exhibit B examines the AIS program, as a narrative or a checklist. Exhibits C and D drill into high-risk models and the data feeding them. The document states the intent plainly: regulators will most often use Exhibit A, then escalate from the responses.

It’s supplementary by design. The NAIC Model Bulletin on AI Systems, adopted December 4, 2023 and picked up by roughly two dozen states by March 2025, supplies the expectations. The bulletin’s risk-proportionality factors already include “extent of human involvement in final decision-making.” Version 7.0 is the examiner-side instrument that asks you to demonstrate it.

If you read the pilot coverage from July, that piece dealt with building the inventory and governance exhibits. This one stays on the review evidence layer, which is where programs are thinnest.

The three factors, and what proves each

The general guidance defines human in the loop to cover both human involvement in a decision and human accountability without involvement in every decision — the agentic case. Then it splits the test:

What regulators evaluateThe evidence that answers itThe failure mode
Authority — can the reviewer reverse, modify, or escalate, and are overrides documented with reasons?Delegation memo with decision limits, system entitlements showing edit and hold rights, override register with reason codes, a sample of overridesReviewers have read-only screens or a four-eyes approval that fires after the batch posts
Qualifications — does the reviewer have the experience to judge the tool’s performance?Role profile, training curriculum, competency check, model-literacy module, performance review criteriaReviewer is the lowest-paid analyst in the queue with no exposure to validation findings
Capacity and evidence — does the design support oversight, shown through override and escalation rates?Rate trend by model and by reviewer, escalation log, override sampling, reopened-item countThe rate is reported with no denominator, no trend, and no sample behind it

Note the pattern: every factor resolves to an artifact. That’s the point of pairing each exhibit question with a documentation request — the examiner isn’t collecting your description of the control, they’re collecting something to compare the description against.

Why the override rate alone misreads

An override rate is a ratio with two moving parts, and only one of them is visible.

The denominator problem shows up first. A rate computed against decisions the reviewer touched is different from a rate computed against all model decisions, and the gap is largest exactly where oversight is thinnest — a queue with sampling skips most outputs before a human sees them. Report the denominator or the number means nothing.

The numerator is gameable in both directions. Require a reason code on every override and watch reviewers start resolving ambiguous cases as “approved” to avoid the write-up. Pair overrides with reopened items — decisions pulled back after the fact — and you get the correction signal the raw rate hides.

Then there’s the direction question. A 0.4% rate distributed evenly across ten models is not the same finding as 0.4% overall with a 3% rate on the claims model and zero everywhere else. Report by model, by reviewer, and against a stated tolerance. If your tolerance is a starting point rather than a benchmark, label it and calibrate against the last six months of your own history before anyone outside your team reads it.

A realistic hypothetical: the decline queue carries a 0.2% override rate, the claims-triage queue sits at 2.8%, and the underwriting referral queue has recorded zero overrides in nine months. Under a blended metric the program looks consistent. Broken out, the underwriting queue is the finding — a reviewer with no recorded override in three quarters either has a perfect model or a queue nobody is really reading.

The register that survives the question

Override documentation is the load-bearing artifact. Minimum fields:

  • timestamp and batch or decision ID
  • model, version, and the output presented to the reviewer
  • reviewer ID and the authority basis for their edit or hold
  • evidence the reviewer considered — document, call note, photo, or data element
  • the decision: accept, modify, reverse, or escalate
  • reason code plus free-text rationale for anything other than accept
  • escalation target and resolution, when escalated
  • downstream effect — what changed for the consumer, including notice timing

Two design choices determine whether this works. Put reason codes on the record at the moment of decision rather than in a weekly summary, and make escalation a first-class outcome alongside override. A reviewer who can escalate but never does is a different problem from one who can’t. The 45-day human review queue clock some states are now writing into rule puts the same pressure on the same artifact from a different direction: if the register can’t show a decision date, you can’t prove the clock was met.

Authority evidence sits one layer up. A delegation memo that names the decision classes each role may change, system entitlements that actually permit a hold or edit, and a named escalation contact with a response window. If the workflow requires a supervisor to release every hold, say so — and measure how long releases take.

Corroboration is the whole design

Version 7.0’s most consequential shift isn’t a new question; it’s the pairing. Requests for supporting documentation now sit next to the questions about AI use, governance, testing, and human review, so the response package has to include records.

That changes what a good response looks like. Management’s narrative gets an evidence column beside it:

AssertionArtifact
”Human review before adverse action”Two-page sample of adverse decisions with reviewer ID, timestamp, and the evidence considered
”Reviewers can override”Entitlement export plus three override records with reasons
”Reviewer training is current”Curriculum, completion dates, and the competency check instrument
”Validation is independent”Reporting line chart and the last validation report’s sign-off page

Where the artifact doesn’t exist, the honest response is a gap with a remediation date. Examiners have seen enough programs to know which one of those is real.

The third-party model question is new

Version 7.0 adds a question for third-party models: how the insurer obtains enough evidence to monitor the risk, and how it learns about material changes or incidents. Read that as a contract question. Notification obligations for model changes, retraining, and incidents; the right to validation evidence and drift reporting; and a named contact on the vendor side.

Pair it with the second addition — what material changes trigger approval, revalidation, or enhanced monitoring. Build the trigger list: retraining event, input schema change, new line of business, drift beyond tolerance, vendor release note, regulator inquiry. A trigger list without a vendor notice clause is a list you find out about too late to use. The decommission sequence for retired models is the other end of the same lifecycle control.

What you can stop preparing

Version 7.0 narrowed in places, and over-preparing wastes the window:

  • Exhibit A dropped the materiality-threshold field, and an existing AI inventory by business function may satisfy the request if it carries system name, use, program area, and inherent risk level.
  • Exhibit B no longer requires a detailed subject-matter-expert roster.
  • Exhibit C dropped the standalone agentic AI question and the proxy-relationships testing question.
  • Exhibit D is limited to data used in high-risk models.
  • Regulators may exclude GLMs, rule-based systems, deterministic algorithms, and fixed formulas where ratemaking oversight already covers them.

One unresolved item worth knowing: the definition of Adverse Consumer Impact still omits a qualifier that the impact must violate a regulatory standard, which trade groups argued could sweep in lawful non-renewals and premium increases. Write your own boundary and cite your reasoning rather than waiting for the definition to settle.

So What?

Human-in-the-loop was a design claim for three years. The supplement turns it into an evidence request with three parts, and the override register is the piece most programs are missing. Start there: pull ninety days of decisions from your highest-volume model, count the overrides, and see whether you can produce reasons for them. If you can’t, the number in the board deck is a claim, not a finding.

The AI Risk Assessment Template has the inventory and risk-tiering structure that Exhibit A and B responses map onto.

FAQ

Do the exhibits apply if we only use vendor models? Yes. Exhibit B covers program governance regardless of who built the model, and v7.0’s third-party question assumes you hold monitoring evidence you don’t generate. Start with the vendor contract.

How long should the override register be retained? The supplement defers to the requesting state’s confidentiality authority and existing handbooks rather than setting a period. Match your model documentation retention, and keep the register contemporaneous — a reconstructed register reads like one.

Is a high override rate evidence of a bad model? No. It’s evidence of an active queue. The paired question is whether overrides cluster on identifiable defects that fed back into retraining. A high rate with a flat defect log means reviewers are compensating for something nobody is fixing.

◆ Need the working template?

Start with the source guide.

These answer-first guides summarize the required fields, evidence, and implementation steps behind the templates practitioners search for.

◆ Immaterial Findings · Weekly

Sharp risk & compliance insights. No fluff.

◆ FAQ

Frequently asked questions.

What is the NAIC AI Risk Evaluation Supplement?
It is a supplemental set of examiner exhibits from the NAIC's Big Data and AI Working Group — four optional exhibits (A through D) that escalate from counting AI systems, to reviewing the AI program, to drilling into high-risk models and the data behind them. Version 7.0 was posted as clean for adoption ahead of the working group's November 16, 2026 meeting at the Fall National Meeting. The document says it supplements the Market Regulation Handbook, Financial Condition Examiners Handbook, and Financial Analysis Handbook and does not create substantive legal duties on its own.
How does v7.0 test human-in-the-loop review?
Through three factors in the general guidance. Whether the reviewer has authority to reverse, modify, or escalate, and whether overrides actually happen and are documented with reasons. Whether the reviewer has the experience and qualifications to judge the tool's performance. And whether the review design supports effective oversight, demonstrable through override and escalation rates. Each question is paired with documentation requests so examiners can corroborate management's description.
Is a low override rate bad?
Not necessarily, but it's ambiguous on its own. A strong model and a rubber stamp produce the same number. The fix is to pair the rate with a sample of override records, a reopened-item count, and the distribution of override reasons, so the rate is explained rather than merely reported.
What got easier in v7.0?
Exhibit A dropped the materiality threshold field. Exhibit B no longer needs a detailed subject-matter-expert list, and the inherent-risk question now references compliance with existing insurance law. Exhibit C dropped the standalone agentic AI question and the proxy-relationships testing question. Exhibit D is limited to data used in high-risk AI models, and regulators may exclude GLMs, rule-based systems, and fixed formulas where ratemaking oversight already covers them.
Does this apply to banks and fintechs?
Directly, no — it's insurance market conduct and financial examination material. But the three human-review factors mirror what prudential supervisors already ask about AI controls, and the override register works unchanged in a bank or fintech AI program. Treat it as a preview of the evidence standard rather than a rule that binds you.
Rebecca Leung

Author

Rebecca Leung

Rebecca Leung has 8+ years of risk and compliance experience across first and second line roles at commercial banks, asset managers, and fintechs. Former management consultant advising financial institutions on risk strategy. Founder of RiskTemplates.

◆ Related framework

AI Risk Assessment Template & Guide

Comprehensive AI model governance and risk assessment templates for financial services teams.

Immaterial Findings · Newsletter

The brief, in your inbox.

Enforcement of the week, a framework breakdown, and the prompts that are actually worth running. Delivered to your inbox. Free.