Why Insurance Needs Accountability Architecture Before AI Goes Wrong
Key Takeaways
- Technology has always arrived before the rules governing it, and insurance has always been the industry that builds the framework after the risk materializes. AI is following the same pattern.
- The NAIC’s AI Systems Evaluation Tool pilot launched in March 2026 across 12 states, with nationwide adoption targeted for November. A model law on third-party AI vendors is anticipated later this year.
- McKinsey’s 2026 AI Trust Maturity Survey found that only about one-third of organizations have reached mature levels of AI governance, even as deployment accelerates.
- Accenture and Wharton’s recent research warns that in a poorly designed agentic enterprise, one person can become accountable for an exponential cascade of outcomes they never anticipated.
- Insurers who build accountability architecture now will be positioned as leaders. Those who wait for the failure will be playing catch-up under regulatory scrutiny.
This is the first article in a series on AI governance in insurance. The first article, “The AI Did It” Is Not a Defense, establishes that insurers cannot delegate accountability to the algorithm. The second, Why Vendor AI Doesn’t Transfer Risk (Even If Your Contract Says It Does), demonstrated that vendor contracts don’t move regulatory liability off your books. The third article, When AI Starts Acting on Its Own: The Governance Gap Insurers Aren’t Ready For, addresses what happens when AI stops assisting and starts acting.
Cars were on the road for decades before auto insurance became mandatory. Workers filled factories long before workers’ compensation laws caught up to the injuries. The internet transformed commerce years before cyber liability coverage existed as a product line. Asbestos was woven into buildings for generations before the liability framework materialized.
The insurance industry knows this pattern better than anyone. Risk arrives first. The structure for managing it comes later, usually after a visible failure forces the conversation.
Artificial intelligence is following the same trajectory. AI-driven underwriting models, claims triage systems, and customer-facing chatbots are already embedded in daily operations across the industry. The NAIC’s own surveys show that 92% of health insurers, 88% of auto insurers, and 70% of home insurers are currently using or planning to use AI. Agentic AI entered insurance workflows in early 2025, with pilot deployments reported across claims processing, fraud detection, and underwriting.
The technology is deployed. The accountability architecture, for most organizations, is not.
The Gap Between Deployment and Governance
McKinsey’s 2026 AI Trust Maturity Survey, covering approximately 500 organizations across industries, paints a clear picture of where things stand. The average responsible AI maturity score rose to 2.3 out of 4, up from 2.0 in 2025. Progress, yes. But only about one-third of organizations report maturity levels of three or higher in strategy, governance, and the newly added category of agentic AI governance. Nearly 60% of respondents cite knowledge and training gaps as the primary barrier to implementing responsible AI practices, up from about 50% the year before.
The finding that should concern every insurance executive: organizations that assign clear ownership for responsible AI, particularly through dedicated governance roles or internal audit teams, scored an average maturity of 2.6. Organizations without a clearly accountable function scored just 1.8. The gap isn’t about technology. It’s about naming someone responsible before something goes wrong.
A joint report from Accenture and Wharton’s AI & Analytics Initiative, published in March 2026 and covered in Fortune, sharpened this point with a single sentence worth sitting with: “Intelligence may be scalable, but accountability is not.” Their research found that more than 50% of working hours across the American economy are now subject to reshaping by AI agents. In banking and capital markets alone, the share of hours impacted by digital agents exceeds 45%.
The implication for insurance is direct. As AI agents multiply across underwriting, claims, customer service, and fraud detection, the humans responsible for those outputs don’t multiply with them. In a poorly designed deployment, the Accenture/Wharton report warns, one person could find themselves responsible for an exponential cascade of outcomes they never saw coming. Not because they were negligent, but because the system scaled decisions faster than the organization scaled accountability.
The Air Canada Warning
In 2024, a Canadian tribunal handed down a ruling that should be required reading for every insurer deploying AI. Air Canada’s chatbot gave a grieving customer incorrect information about bereavement fare policies. When the customer relied on that information and later sought a refund, the airline refused, then attempted to argue that the chatbot was essentially a separate legal entity responsible for its own actions.
The tribunal rejected this outright. Air Canada was liable for all information on its website, whether it came from a static page or a chatbot. The company owed its customer a duty of care, and it had failed to ensure its AI provided accurate information. The ruling was modest in dollar terms, roughly $650 CAD, but the principle it established was not: courts and regulators will not accept “the AI did it” as a defense.
For insurers, the parallels are immediate. If your AI-powered claims system denies a legitimate claim, if your underwriting model produces discriminatory outcomes, if your customer-facing chatbot makes a coverage promise the policy doesn’t support, the liability lands on the organization. The NAIC’s Model Bulletin makes this explicit: insurers are responsible for AI-driven decisions the same way they are responsible for decisions made by human adjusters.
The Regulatory Timeline Is No Longer Theoretical
The NAIC has been building toward this moment for years, and the pace is accelerating. The Model Bulletin on the Use of AI Systems by Insurers, adopted in December 2023, has now been adopted in 24 states. It requires insurers to establish written AI governance programs emphasizing transparency, fairness, and risk management. It holds insurers accountable for third-party AI systems, not just tools built in-house.
But the bulletin was principles-based guidance. What’s happening now is operational.
In March 2026, the NAIC launched its AI Systems Evaluation Tool pilot across 12 states: California, Colorado, Connecticut, Florida, Iowa, Louisiana, Maryland, Pennsylvania, Rhode Island, Vermont, Virginia, and Wisconsin. The pilot runs through September, with participating states meeting monthly to share findings. The tool covers four areas: the scope of an insurer’s AI usage, governance and risk assessment frameworks, details on high-risk AI systems, and the data those systems rely on. Regulators have stated they will prioritize AI systems most likely to produce consumer harm, applying less scrutiny to back-office automation.
The tool is expected to be updated based on pilot feedback and re-exposed for public review, with adoption targeted at the NAIC’s fall meeting in November 2026. A model law addressing third-party AI vendors and data providers is also anticipated, potentially including licensing requirements for vendors. Colorado’s AI Act, with specific requirements for high-risk AI systems, takes effect this year as well.
Some insurers have already received participation requests from their domiciliary regulators. Industry trade groups have pushed back, arguing the pilot is “voluntary for regulators while compulsory for companies.” But the direction is clear. As NAIC President Scott White stated at the 2026 Spring National Meeting: regulators want AI used “transparently, fairly and in ways that hold up to scrutiny.”
What Accountability Architecture Looks Like in Practice
“Responsibility architecture” can sound abstract until you translate it into operational terms. For an insurance organization, it means answering a specific set of questions before an AI system goes into production, not after something breaks.
Who is accountable at each decision node? When an AI underwriting model generates a recommendation that gets acted on, was there a defined human authority who approved the model’s deployment for that use case? When a claims triage system routes a claim, who owns the outcome if the routing produces a consumer harm? The NAIC Model Bulletin requires governance structures with stakeholders from actuarial, data science, underwriting, compliance, and legal, each with defined responsibilities, authority, and decision-making powers.
Do you own your vendors’ liability? The Model Bulletin holds insurers responsible for third-party AI systems. If a vendor’s model produces discriminatory pricing or unfair claims decisions, the insurer bears the regulatory consequence. This means vendor contracts need audit rights, documentation of model origins, standards for explainability, and cooperation with regulatory inquiries. The anticipated model law on third-party oversight will likely formalize these expectations further.
Can you explain what your AI did and why? Regulators using the AI Systems Evaluation Tool will ask insurers to document governance frameworks, risk management protocols, and internal controls. If an examiner asks how a specific AI-driven decision was made and your answer is that the algorithm is too complex to explain, you have a problem. Transparency and explainability are not optional features. They are regulatory expectations.
Is your incident response plan AI-aware? Traditional incident response focuses on data breaches and system outages. AI-specific incidents, such as model drift producing biased outcomes, an agent taking unintended actions, or a chatbot making unauthorized commitments, require their own classification and response frameworks.
The Leadership Question
McKinsey partner Rich Isenberg framed it well in a recent discussion of agentic AI governance: “The question shifts from ‘Is the model accurate?’ to ‘Who’s accountable when the system acts?'” His advice to leaders is blunt. If you don’t redesign decision rights, accountability, escalation paths, and controls alongside your AI deployment, you’re not leading a transformation. You’re hoping the system behaves. And that is not a defensible posture when speaking to your board or your regulators.
The Deloitte State of AI in the Enterprise 2026 report reinforced this finding: enterprises where senior leadership actively shapes AI governance achieve significantly greater business value than those delegating governance to technical teams alone. When accountability is delegated downward without clear ownership, decisions get made at scale without anyone senior enough to own the outcomes.
The insurance industry has spent 150 years building frameworks for managing risk after it materializes. AI presents an opportunity to break that pattern, to build the accountability architecture before the failure that would otherwise force it. The organizations that move first won’t just be compliant. They’ll be the ones their regulators, their customers, and their boards trust most when the technology inevitably does something no one anticipated.
The question for every carrier, wholesaler, and agency executive is straightforward: if your AI makes a decision tomorrow that harms a customer, can you trace the chain of accountability from the model’s output back to a named human who authorized it? If the answer takes more than a few seconds, the governance isn’t ready for the deployment already underway.
Sources
- Accenture and The Wharton School, “The Age of Co-Intelligence: How Humans, AI Agents, and Robots Are Redefining Value,” March 2026. Reported in Fortune.
- McKinsey & Company, “State of AI Trust in 2026: Shifting to the Agentic Era,” March 2026. McKinsey.
- McKinsey & Company, “Agentic AI Governance for Autonomous Systems” (podcast transcript with Rich Isenberg), 2026. McKinsey.
- National Association of Insurance Commissioners, “Model Bulletin on the Use of AI Systems by Insurers,” December 2023. NAIC.
- Fenwick & West LLP, “NAIC Expands AI Systems Evaluation Tool Pilot Program to 12 States,” March 2026. Fenwick.
- Foley & Lardner LLP, “What To Do If You Receive an NAIC AI Systems Evaluation Tool Pilot Request,” March 2026. Foley.
- Digital Insurance, “NAIC Begins AI Evaluation Pilot,” March 2026. Digital Insurance.
- Holland & Knight, “The Implications and Scope of the NAIC Model Bulletin on the Use of AI by Insurers,” May 2025. Holland & Knight.
- Plante Moran, “How the NAIC AI Model Bulletin Is Evolving and Why Insurers Should Prepare Now,” March 2026. Plante Moran.
- ISACA, “Responsible AI: From Emerging Technology to Executive Governance Imperative,” February 2026. ISACA.
- American Bar Association, “BC Tribunal Confirms Companies Remain Liable for Information Provided by AI Chatbot,” February 2024. ABA Business Law Today.
- World Economic Forum, “Why Effective AI Governance Is Becoming a Growth Strategy,” January 2026. WEF.
AI Disclaimer: This content was created with assistance from artificial intelligence technology. While content is based on factual information from the source material, readers should verify all details directly with the respective sources before making business decisions.
