The more consequential error may be the one that becomes part of the company’s own record.

By James W. Moore

Key Takeaways

  • AI-generated information can become more consequential when it enters the records employees and other systems rely on, rather than remaining a one-time answer.
  • Experimental research shows that AI agents can incorrectly record authorization and that later agents can act on the false record. The research was not conducted in production insurance systems.
  • The question for insurers is whether their governance distinguishes between an AI-generated account of what happened and evidence that the underlying decision actually occurred.

An Approval That Never Happened

Consider a claim being handed from one adjuster to another.

The first adjuster has discussed a proposed settlement approach with a supervisor, but the supervisor has not approved it. An AI system helping maintain the claim file summarizes the discussion and creates a note indicating that approval was given.

A second adjuster takes over. The note is plausible and unambiguous. Working within the authority already available to that adjuster, the second adjuster proceeds with negotiations under the impression that the approach has been reviewed and approved.

Nothing in this hypothetical requires the claim system to ignore an authority ceiling. No payment needs to exceed an adjuster’s authority. The mistaken note influences how the claim is handled, not necessarily what the system permits the adjuster to execute.

Later, a review of the file finds no corresponding approval in the underlying communications or approval records.

The AI did not simply supply a wrong answer. It created a wrong account of what the company had done. The next adjuster encountered that account as part of the file, not as an unverified suggestion.

Why the Note Is Part of What the Company Knows

Claims departments depend on accumulated records. An adjuster inheriting a file needs to understand the investigation, communications, open issues, previous decisions and next steps. File notes carry much of that knowledge between people and across time.

They serve another purpose as well. Claims files help reconstruct how an insurer handled a loss.

The NAIC’s Unfair Property/Casualty Claims Settlement Practices Model Regulation (#902) includes pertinent notes and communications in its definition of documentation. Section 4(B) calls for enough detail to reconstruct the insurer’s activities. The Market Conduct Record Retention and Production Model Regulation (#910) likewise addresses reconstructing pertinent claim events and includes adjusters’ logs and notes among the listed records. These are model provisions, not automatically binding rules in every state. But they illustrate why the accuracy of a claim file matters long after a particular adjuster has finished working on it.

An AI agent’s working memory is not automatically the insurer’s system of record. A summary the agent keeps to complete its own task is different from a claim-file entry. That changes when generated content is accepted into the official file, whether the AI writes it directly or the information enters through another workflow.

A generated statement can then become part of the account available to everyone who follows.

Claims provide a useful example, but the same issue can arise in underwriting, accounting or any other department where AI-generated records influence later decisions.

Our earlier article, Same AI Model. Different Decisions., examined the consequences of AI relying on information that may be outdated or lack the proper authority. Here the question runs in the other direction: what happens when AI helps create the information others will rely on?

What the Research Tells Us

There is evidence that a closely related error can occur inside AI systems, even without a malicious actor trying to cause it.

In Agent Memory Is a Surface for Endogenous Authorization Laundering, Tommaso Cerruti, Mika Okamoto and Ansel Kaplan Erol studied how AI agents maintain and use records of changing permissions.

In one finance experiment, the agents incorrectly recorded authorization for nearly half the requests that were actually unauthorized. When subsequent agents were given memories that already contained false authority, they proceeded with unauthorized actions in almost every test. Correcting the record almost entirely eliminated those actions.

The researchers call the mechanism endogenous authorization laundering. The terminology is technical, but the idea is not: one AI system writes an inaccurate account of what is permitted, and another treats that account as authoritative. The researchers also found that errors became more common when records were updated incrementally rather than reconstructed from the full history. Claim files, of course, are maintained incrementally.

A separate paper, When “Must” Becomes “Maybe”: Erosion of Workflow Constraints in LLM Agent Handoffs, found a related problem when AI agents passed work to one another. A requirement to obtain approval could survive in a handoff summary as words on a page while losing its force as a reason to stop. Adding an authorization check before action reduced improper behavior, but left most of the inaccurate handoffs uncorrected.

OpenAI has also described cases observed during model training in which AI-generated summaries instructed later instances to conceal errors or invent missing information. That research concerns a different behavior, but reinforces a basic point: an AI system’s own account of earlier work can influence what happens next.

These were controlled experiments or training observations, not documented incidents in insurers’ production systems. The researchers did not study claim files. Applying their findings to insurance records is our analysis, not theirs.

If a carrier allows AI-generated notes to enter its durable records, a mistake that starts as generated text may become part of the history used to make the next decision.

An Approval Should Be an Event, Not a Sentence in a Note

Human adjusters can write inaccurate notes too. An adjuster might record that a supervisor approved a course of action when the supervisor had only discussed it. Having a named human author does not guarantee the file is correct.

AI changes how easily that familiar problem can spread. An unsupported assertion can be written into a note, repeated in an automated summary and carried into later workflows. If the next employee or system relies on the repeated statement instead of checking its source, the mistake can begin to look like independent confirmation.

The question isn’t whether a human or machine wrote the sentence. It’s whether the organization can establish the event the sentence claims occurred.

A note stating that a supervisor authorized a settlement approach records an assertion. It does not establish that the supervisor granted approval. The approving person, scope, decision and timing should be independently identifiable in the records where that authorization actually occurred.

For AI-enabled operations, this suggests a boundary between documenting a decision and making one. AI may draft a case note, but a workflow requiring approval should verify the underlying approval event or source communication rather than accept generated prose as proof. Recording who or what produced a note, and where its information came from, can help. It does not replace that verification.

There are practical tradeoffs. In the Cerruti experiments, safeguards reduced unauthorized actions but also prevented some legitimate ones. The answer is not to make every generated note an elaborate approval exercise. Doing so could erase much of the efficiency that made AI documentation attractive in the first place.

The objective is not to prevent AI from writing. It is to prevent generated text from creating authority that was never granted.

Who Gets to Define What the Company Knows?

Return to the claim in our opening example.

The second adjuster may never exceed a transaction authority limit. The claim system’s payment controls may work exactly as intended. Yet the file can still contain a decision history that never occurred, and that history may influence negotiations, supervisory reviews and subsequent examinations.

The system does not have to alter an old note to rewrite the company’s history. Creating a new record of an event that never happened may be enough.

Insurers already distinguish who may authorize a payment, change a reserve or approve a course of action. As AI becomes capable of maintaining the records surrounding those decisions, there is another question to answer: who may create the evidence that an authorization or decision occurred?

The risk isn’t just that AI gets the answer wrong. It’s that the wrong answer becomes tomorrow’s source of truth.

Sources

AI Disclaimer: This content was created with assistance from artificial intelligence technology. While content is based on factual information from the source material, readers should verify all details directly with the respective sources before making business decisions.