KRI Data Quality: What to Do When the Metric Is Right but the Source Data Is Trash

TL;DR

• A KRI can be perfectly defined and still be meaningless if the underlying source data is stale, inconsistently extracted, or patched through a manual spreadsheet.
• The four most common failure modes: manual feeds with no validation, source system definition drift, quarterly-refresh metrics presented as current, and no pre-dashboard review gate.
• Before trusting any KRI, confirm: source system, refresh cadence, extraction method, and definition stability — all four, not just the ones you remember.
• Examiners don’t test only the metric. They test whether anyone validated the data before it hit the dashboard.

Your compliance team spent two weeks selecting and calibrating 25 KRIs. The thresholds are documented. The owners are assigned. The dashboard is clean. Every indicator is green.

And then someone from internal audit asks a simple question: where does the SAR filing rate number actually come from?

The answer is a spreadsheet. Finance pulls it from the case management system once a month, manually, by counting rows in an export. The last time anyone verified whether the export includes declined SARs or only filed ones was during initial setup — eighteen months ago.

Your SAR filing rate KRI has been green for six months. But no one can say with confidence what it is actually measuring.

This is the KRI failure mode that shows up least often in framework discussions but most often in practice: not choosing the wrong metric, but trusting the right metric fed by the wrong data. The indicator looks healthy. The underlying information is not.

Why KRI Data Usually Looks Fine Until It Isn’t

KRI design gets significant attention — threshold calibration, ownership assignment, escalation procedures. What gets less attention is the plumbing: where the data originates, who extracts it, and whether the number that lands on the dashboard is measuring what the metric definition says it’s measuring.

Four failure modes account for most KRI data quality problems in practice.

Manual extraction with no validation gate. Someone pulls a number from a source system once a month and pastes it into the tracking sheet. Nobody checks whether the query criteria are consistent with prior periods, whether any system errors affected the export, or whether the pull logic still matches the KRI definition. When the result looks reasonable, it gets posted.

Source system definition drift. A transaction monitoring false positive rate KRI was built against a specific alert disposition code. The vendor upgraded the case management platform in Q3 — and three disposition codes were renamed and consolidated into two. The query still runs without errors. The numbers are no longer comparable to prior periods. Unless someone notices the changelog and traces it to the KRI definition, the dashboard never flags it.

Refresh cadence mismatches. A metric intended to track near-real-time exposure is being refreshed quarterly, not monthly. When it shows green in February, it is reflecting data from November. If a stress event begins in December, the dashboard reports no signal until the quarterly cycle closes — weeks after the problem was visible to anyone watching operations directly.

No pre-dashboard validation. Data moves from source system to dashboard without anyone checking whether the output is plausible. Outliers, null values, and quietly changed definitions all get posted without a review gate. The first indication of a problem is often a question from senior management asking why the number disagrees with what they heard from the business.

The Four Questions to Ask Before You Trust a KRI

Before acting on any metric — especially before treating a threshold breach as an escalation trigger — get answers to these four questions:

1. What system produces this data? “The compliance system” is not a sufficient answer. You need the specific application, module or database table, and query or report used to extract the metric. If the data comes from a report rather than a direct database query, understand whether that report’s criteria match your KRI definition. Reports are built for operations, not for risk monitoring — the defaults frequently include or exclude categories that matter.

2. How frequently is it refreshed, and by what mechanism? Automated extraction (API feed, scheduled query, direct integration) reduces manual error but does not eliminate data quality risk. Scheduled jobs fail silently. Automated processes that depend on upstream feeds inherit their quality problems. Manual extraction introduces additional error and is commonly delayed when the person responsible is unavailable. Both need a documented verification step.

3. Has the definition changed in the last 12 months? This question catches source system upgrades, policy redefinitions, and scope changes that happen outside the KRI governance process. If a metric uses terms like “alert,” “case,” “exception,” or “finding,” verify that those terms mean the same thing in the source system today as they did when the KRI was designed. System upgrades are the most common cause of silent definition drift — the query runs, the number looks plausible, and no one realizes the denominator changed.

4. When was this data last independently validated? “It’s automated” does not answer this question. Automation reduces one class of errors; it does not verify that the data is measuring what the policy says it measures. A validation date means someone reviewed the output against an independent source or performed a spot-check against raw data within the current review period. If that has not happened since the KRI was built, the metric has not been validated — it has been assumed.

Building a KRI Data Quality Register

The practical control is a register — one row per KRI — maintained alongside the KRI library. Minimum fields:

“Active” means the metric has been validated within the current review cycle and is fit for dashboard reporting and escalation decisions. “Under Review” means a data quality question has been flagged — the number should be footnoted in any reporting that includes it. “Suspended” means the metric is paused and should not appear in reporting until the data issue is resolved and independently verified.

This does not require a GRC platform. A consistently maintained spreadsheet demonstrates more defensible data governance than an automated system that nobody validates.

When You Discover Bad Data Mid-Cycle

Data quality problems surface mid-cycle regularly. The response that protects you — operationally and from an audit standpoint — is documentation and communication, not a quiet correction.

When bad data is confirmed in an active KRI, the sequence is:

Suspend the metric. Mark the KRI “Under Review” and remove it from active dashboard reporting. A suspended metric acknowledged and documented is a data governance finding. A metric that continues to report while the team knows it’s unreliable is a governance failure.

Document the exception. Record what was discovered, when, who found it, how long the problematic data may have been in the feed, and what the metric was reporting during that period. This is the starting point for both root cause analysis and the issues management entry.

Assess downstream reliance. If the KRI was reporting green during a period of unreliable data, determine whether any risk assessments, committee reports, or escalation decisions relied on it. If they did, those decisions should be reviewed and footnoted. This is uncomfortable, but it is far less uncomfortable than an auditor discovering it later.

Root cause the failure. Was this a one-time extraction error, a system change that broke the definition, a manual process gap, or something structural in how the source system is accessed? The root cause determines whether the fix is a one-time correction or a process redesign.

Validate before reinstatement. Before the metric returns to active status, verify that it is now being extracted correctly and cross-check at least two prior periods against an independent source. Document the validation.

This process looks like issues management because it is. A KRI data quality failure is a control failure — the information infrastructure supporting your risk program is not working as designed. It belongs in your issues log with a severity rating, owner, and target remediation date.

The instinct to quietly fix the number and move on creates the larger problem: an auditor or examiner who later discovers several quarters of unreliable data and no documentation of when the issue was identified or how it was resolved is looking at a governance deficiency, not just a data problem.

What Examiners Actually Ask About KRI Data

OCC, FDIC, and Federal Reserve examiners reviewing operational risk programs do not stop at “what are your KRIs?” They ask process questions:

• Where does this metric come from?
• How do you know the data is current?
• Has anyone independently validated this number?
• What happens when the metric hits amber or red — what does that actually trigger?

The last question assumes the first three have defensible answers. An escalation process that is well-documented but drawing on unvalidated data is an incomplete program. Examiners are more concerned about hidden gaps than about acknowledged limitations — a KRI data quality register that says “this metric has a known three-week lag due to vendor reporting cycles” is a documented limitation, not a finding. A metric that no one has looked at critically since it was built is a gap that becomes a finding once someone asks.

For practitioners building or auditing KRI programs, the signal that data quality governance is insufficient is usually simple: no one on the team can answer the four questions above without looking it up, or the answers don’t exist in any written form.

Solid KRI design paired with weak data governance produces a dashboard that looks like a risk program but does not function as one. The leading and lagging indicator balance matters — the leading vs. lagging KRI article covers that design challenge — but even the best leading indicator is useless if no one knows whether the data behind it is current.

Similarly, KRI governance focuses on who owns the metric and the escalation chain. Data quality ownership needs to be assigned with the same rigor — the metric owner and the data owner are often different people, and both need to be accountable.

So What?

The KRI data quality problem is pervasive, fixable, and rarely prioritized until it surfaces at the wrong moment.

The fix is not a technology investment or a data engineering project. It is a register, a validation cadence, and accountability for both. For every KRI on your dashboard, you should be able to answer: where does the data come from, how recently was it verified, and who is responsible for catching a problem before it posts? If those answers are not documented, you have metrics but not a functioning risk program.

Build the register before the examiner asks the question. The four questions above are easy to answer prospectively and awkward to answer under scrutiny.

If you’re building or rebuilding a KRI library and want a pre-built structure that includes source system and ownership fields alongside the metrics themselves, the KRI Library (132 Key Risk Indicators) includes the operational infrastructure your team needs to run a defensible program — not just a list of indicators.

Get the KRI Library →