AI Risk Assessment Questionnaire: Split the Questions Between the Business Owner, Technology Team, and Independent Reviewer

TL;DR

• An AI risk assessment questionnaire should not be handed to one person. The business owner explains the use and consequences; Technology proves how the system works; an independent reviewer tests whether the evidence supports deployment.
• Give every question an answer owner, evidence requirement, reviewer conclusion, and unresolved-gap status. “Yes” without an artifact is still an assertion.
• Unknowns should become visible decisions: resolve before launch, restrict the use case, add a compensating control, or issue a time-limited exception with stop conditions.

The fastest way to make an AI assessment unreliable is to ask one person to answer all of it.

A product owner can explain why a customer-service model exists, but may not know whether prompts are retained by a foundation-model provider. An engineer can document the API and evaluation set, but cannot decide whether the output creates a fair-lending or disclosure obligation. Risk can challenge both, but should not invent facts that belong to the system owner.

A usable AI risk assessment questionnaire therefore splits the work. It routes each question to the person closest to the fact, then gives an independent reviewer a separate place to challenge the answer and record the decision. That structure follows the accountability logic in NIST AI RMF 1.0: GOVERN 2.1 calls for documented roles and lines of communication, MAP 1.1 requires the intended purpose and deployment context to be understood and documented, and Measure 1.3 brings in experts who did not serve as front-line developers and/or independent assessors.

The practical payoff is traceability. When a bank partner asks, “Who said this use case was safe to launch?”, the answer is no longer “the questionnaire was completed.” You can show who supplied each fact, what evidence they attached, what the reviewer challenged, and which conditions survived approval.

Start with a routing sheet, not 80 anonymous questions

Before writing question text, define the roles. Titles vary, so assign responsibilities rather than assuming every company has a Model Risk Management team.

Small fintech? One person may wear two titles. Preserve the functional separation anyway. The engineer provides technical evidence; the Head of Compliance challenges it; the CEO or designated risk authority accepts a material residual risk. If the builder also supplies the final approval, document why no independent reviewer was available and narrow the deployment until one is.

NIST’s AI RMF Core is direct on the point: independent review can improve testing effectiveness and mitigate internal bias and conflicts of interest. The Generative AI Profile, NIST AI 600-1, adds that independent evaluations or assessments should be proportionate to identified risk. Independence is not ceremony; it is protection against the team grading its own work.

Questions for the business owner: define the decision before scoring the model

The business section should establish what the system is allowed to do. Do this before technical scoring. A technically strong model attached to an undefined or impermissible use case is still a bad deployment.

The wording matters. “Is there human oversight?” invites “yes.” Ask instead: Which role reviews which outputs before action, what information does that person see, what authority do they have to override, and where is the override logged?

For a lending use case, connect the business answer to the actual decision notice. The CFPB’s September 19, 2023 guidance on credit denials using artificial intelligence said creditors must give accurate, specific adverse-action reasons; selecting a nearby reason from a sample checklist is insufficient when it does not reflect the real basis for the decision. That turns explainability from a model-team talking point into an operational test: can the approved workflow produce the reason the consumer actually receives?

Questions for Technology: require a reproducible evidence package

Technology owns facts about the system. The questionnaire should make those facts reproducible by a reviewer who did not build it.

Architecture and data

Ask for the system name, vendor, model and version where knowable, hosting pattern, upstream and downstream systems, data fields, data sources, retention, access roles, subprocessors, and whether customer data can be used for provider training or service improvement.

Evidence: a dated architecture diagram, data-flow diagram, data dictionary, access-role export, vendor terms, and configuration record. A generic security policy does not prove how this deployment is configured.

Evaluation and limitations

Ask Technology to define:

• the evaluation population and why it represents intended production use;
• performance measures and acceptance criteria chosen before results were reviewed;
• known failure modes and excluded conditions;
• outcome testing across relevant groups where the use affects people;
• robustness, security, misuse, and fallback tests appropriate to the system;
• unresolved findings and their production impact;
• the exact model, prompt, retrieval source, code, and configuration evaluated.

NIST’s MEASURE function says AI systems should be tested before deployment and regularly in operation, with formal reporting and documentation of results. The questionnaire should therefore link to evaluation outputs, not summarize them as “testing passed.”

Monitoring and rollback

Require a monitoring specification with the metric, source, owner, review frequency, decision threshold, escalation route, and rollback action. If no external benchmark exists, label thresholds as internal starter thresholds and calibrate them against pilot or recent production history. Add an anti-gaming check: reconcile sampled production decisions to logs and incident or complaint tickets so a clean dashboard cannot hide missing records.

The independent reviewer gets a separate column

The reviewer should never overwrite the owner’s original answer. Preserve four fields:

1. Owner assertion — what the responsible person says is true.
2. Evidence reference — the artifact and version supporting it.
3. Reviewer conclusion — supported, partially supported, unsupported, or not applicable with rationale.
4. Decision effect — no action, remediation, restriction, launch blocker, exception, or escalation.

A practical challenge scale:

This is where “effective challenge” becomes visible. A reviewer note such as “SOC 2 received” is inventory, not challenge. A useful note says: “Report covers the vendor’s core platform through March, but the new generative feature and retrieval service are outside scope; obtain architecture evidence and restrict customer data until reviewed.”

For a deeper model of documenting disagreement, use the effective challenge guide. For the final approval artifact, connect the questionnaire to the AI governance decision log.

A realistic hypothetical: the customer-support summarizer

A fintech wants to summarize support calls and draft agent notes.

• Business owner answer: Agents will review every draft; the system cannot send messages or change accounts. Evidence is the approved workflow and role permissions.
• Technology answer: Transcripts are sent to a vendor API after direct identifiers are tokenized. Evidence includes the data-flow diagram, tokenization test, API configuration, retention setting, and vendor terms.
• Compliance answer: The approved use excludes complaint disposition, adverse-action explanations, and account decisions. The complaint procedure is updated so agents do not replace the customer’s allegation with an AI summary.
• Independent challenge: The vendor’s documentation does not establish deletion operation for transient logs. The reviewer marks privacy evidence partially supported.
• Decision: Restricted pilot using tokenized transcripts; daily sample comparison to source audio; no training or secondary use; deletion evidence due before expansion; automatic expiry if the condition is missed.

That is a risk assessment. “Vendor uses encryption and has a policy” is not.

So What?

Take one AI questionnaire already marked complete and add four columns: answer owner, evidence reference, independent conclusion, and decision effect. Then find every “yes” with no artifact and every “not applicable” with no rationale.

Do not fix those cells in Risk. Route them back to the people who own the facts. The first useful output may be a list of unknowns. Good. Unknowns exposed before deployment are governance; unknowns hidden inside a completed spreadsheet are future incident work.

For the broader assessment structure, see the pre-deployment AI risk assessment walkthrough and the minimum NIST AI RMF artifact set.

The AI Risk Assessment Template & Guide includes a role-ready use-case inventory, pre-deployment assessment, evidence fields, vendor questionnaire, and worked examples for teams that need to put this workflow into operation.