How the information an AI system draws on can change its decisions, even when the underlying model stays the same.
By James W. Moore
Key Takeaways
- An AI model is only one component of an AI decision system. The information retrieved and supplied to it can change its recommendations without any change to the model itself.
- Information retrieval creates distinct failure possibilities: relevant information may never be found, may be found but poorly used, or may be used even though it no longer applies.
- For insurers, the important question is not simply whether the model has been validated. It is whether the organization can establish which information governed a particular decision and reconstruct what the system actually received.
Same Submission. Different Recommendation.
Consider a commercial contractor whose operations include a substantial amount of residential construction.
The carrier’s current underwriting guidelines require referrals to a senior underwriter when residential work exceeds a specified percentage of total operations. An earlier edition of the guidelines permitted those accounts to proceed through the normal quoting process.
An AI underwriting assistant evaluates the contractor’s submission and recommends referral, consistent with the current guideline.
Several months later, the broker resubmits the account for a later effective date. The contractor’s operations, residential-work percentage, and all other underwriting facts remain unchanged. So do the carrier’s underwriting rules, the AI model version, and the assistant’s operating instructions.
This time, however, the assistant recommends proceeding with a quote. The referral requirement was meant to place this account before a senior underwriter. Now the assistant’s recommendation would allow it to bypass that review.
What changed?
Between the two evaluations, the carrier migrated historical underwriting documents into its searchable repository. The migration introduced older editions of its underwriting manual and gave the imported files new timestamps. In this hypothetical system, retrieval ranking uses those timestamps as one signal of recency. The superseded guideline rises above the governing edition and is the only one supplied to the assistant for the relevant rule.
On the second evaluation, the system relies on that superseded guideline instead of the one currently governing the risk.
The model did not need to malfunction. Given the older rule, its recommendation was understandable. It was also inconsistent with the carrier’s current underwriting authority.
This is a hypothetical example, but the mechanism illustrates a real architectural distinction: the AI model and the system supplying its information are not the same thing.
Why the Model Cannot Simply Read Everything
Much of the discussion about AI still focuses on the model itself: which model an insurer selected, how it was trained, how it performs in testing, and whether its outputs can be explained.
Those questions matter. But a model deployed inside an insurance company often needs information that was never part of its training.
It may need the company’s underwriting guidelines, current appetite restrictions, delegated authority limits, submission records, or loss-control reports. Those documents are supplied when the system performs a task, rather than permanently incorporated into the model.
One common approach is retrieval-augmented generation, or RAG. Instead of relying exclusively on what the model learned during training, another component searches an external collection of information and supplies selected material for the model to use.
Why not give the model the entire underwriting manual, along with every related bulletin and historical revision?
Because more information doesn’t necessarily produce a better answer.
AI models have limits on how much information they can process in a single working context. Even within those limits, performance can deteriorate as the amount of material increases.
In its September 2025 discussion of context engineering, Anthropic described the need to manage the information available to an AI agent, including instructions, retrieved documents, tools, and accumulated history.
The challenge is not simply to provide information. It is to provide the information most useful for the task.
Independent research supports that concern. A paper presented at the 2025 International Conference on Learning Representations found that adding more retrieved passages could initially improve a model’s answers and subsequently make them worse. One contributing factor was the presence of passages that appeared relevant but were misleading.
Selection is therefore unavoidable. Sometimes a retrieval system selects material through predefined search and ranking rules. In more autonomous systems, an AI agent may decide what to search for next, making successive choices about which information deserves further attention. In either case, information selection becomes part of the decision process, with its own failure possibilities.
This is more than a conventional document-control problem. Insurers have long managed manuals, bulletins, effective dates, and withdrawn rules. The new dependency lies between that governed repository and the automated decision: the retrieval system decides which portion of the repository becomes evidence for a particular recommendation. A document may be properly marked superseded in the carrier’s records yet still be elevated by a search ranking rule and delivered to the model without the controlling context. The recommendation can remain fluent and plausible even when the selected rule is wrong.
When Relevant Information Isn’t the Right Information
There are at least three ways the information surrounding an AI model can influence an otherwise reasonable recommendation.
The first is straightforward: the system never retrieves information it needs.
Suppose a loss-control report contains a recommendation that materially affects a contractor’s eligibility. The report exists in the carrier’s files but isn’t returned by the search system. Perhaps it was indexed under an unexpected document type, or the assistant never searched the right collection.
The model evaluates the information it received. The missing report plays no part in its recommendation.
The second possibility is subtler: the system retrieves relevant information but doesn’t use it appropriately.
An underwriting assistant might receive guidelines for several closely related contractor classifications. One document addresses the submitted operation; another addresses a neighboring classification with different eligibility rules.
Both documents contain familiar terminology. Both may appear relevant to the search.
The model has the correct material available but gives excessive weight to the rule governing the wrong class.
Retrieval research has identified similar problems when models encounter misleading passages that resemble useful evidence. Finding the correct information is not sufficient if the surrounding material interferes with its application.
The third possibility is central to our opening example: the system retrieves information that was once correct but no longer has authority.
The old underwriting manual was not necessarily inaccurate when written. Its residential-work threshold may have reflected the carrier’s legitimate appetite at the time.
But a superseded guideline is no longer an appropriate basis for the current decision.
A semantic search system generally looks for material relevant to the question being asked. Unless its retrieval and validation processes account for the authority and applicability of documents, an older guideline might appear more relevant than the one that replaced it.
If a retrieval system treats file timestamps as evidence of recency, the migrated manual may appear newer even though its underwriting guidance is obsolete. That is the ranking mistake in our opening example. A publication date, an upload date, an effective date, and a supersession date answer different questions. Systems that account for effective periods and source authority need not make the same mistake.
What the research tells us
A September 2026 preprint, Stale-Document Poisoning: When Outdated Retrieval Overrides Correct Model Answers, examined what happens when AI models encounter previously accurate information that has since become outdated.
The researchers examined changes in public knowledge across medicine, law, software, and platform policies. In some conditions, a single outdated document overturned an answer a model had correctly supplied without retrieval. Importantly, adding document dates alone did little to prevent errors. Explicitly stating when older guidance stopped applying was much more effective for the larger models in the researchers’ tests.
A second finding was counterintuitive: in two smaller open-weight models tested, explicitly instructing the model to follow retrieved documentation made it more likely to rely on outdated material, not less. Those experiments tested whether retrieved evidence could override correct information the models already possessed. They did not test unpublished carrier underwriting rules.
That distinction matters. A general-purpose model ordinarily has no independent knowledge of a carrier’s current, private underwriting appetite. It cannot know a guideline was superseded unless the surrounding information system supplies and enforces the governing authority. Document validity fields, authoritative repositories, access controls, and retrieval filters can all contribute. None should be confused with the model independently knowing which private rule applies.
The study is a September 2026 preprint, not peer-reviewed, and it provides no estimate of error rates in insurance deployments. The researchers also tested recency-aware re-ranking of retrieved material. It reduced the incidence of stale-answer failures by approximately 4.6 to 10 percentage points when date information was accurate. When the researchers supplied incorrect dates however, the same re-ranker performed worse than ordinary retrieval: it failed to select the current document in any of those tested cases. This was a result about document selection, not the accuracy of downstream model answers. Our hypothetical migration creates the corresponding risk when fresh upload timestamps are mistaken for the dates on which underwriting rules became effective.
Underwriting manuals, policy forms, endorsements, regulatory notices, and delegated authority agreements often have effective dates and revision histories. The question is whether those distinctions survive when the information is converted into material an AI system can retrieve and use.
The important technical issue is not whether the AI can find a document that looks relevant.
It is whether the system can establish that the document governs the decision being made.
Can the Carrier Reconstruct What Happened?
Return to the contractor submission.
The first evaluation recommended referral. The second recommended proceeding with a quote. The model version was unchanged, and the carrier hadn’t revised its underwriting rules.
Suppose the difference comes to light six months later.
Can the carrier establish which underwriting guideline the system retrieved during each evaluation?
A log indicating that the assistant consulted the underwriting manual may not be sufficient. If the manual has been updated in place, the document available today may not match what the AI received at the time.
Even a reference to a section number may be inadequate if that section was revised or replaced.
A useful historical record would need to identify the actual version supplied to the system, the relevant instructions in force, and any other material or tools involved in the recommendation.
That isn’t the same as explaining every internal calculation performed by the model. Recording what entered the system does not guarantee that the resulting decision was correct.
But without an adequate record of what the system received, it becomes difficult to establish whether the differing recommendations resulted from changed documents, altered retrieval behavior, modified instructions, or ordinary variation in the model’s output.
Current documentation is not a historical decision record.
There is another implication.
A carrier may have successfully tested an AI model, evaluated its performance, and approved its use. Yet changes to the surrounding document repository can alter the information that model receives without triggering anything that looks like a model update.
The deployed decision system has changed even though the model has not.
The System Is What the Carrier Must Understand
The NAIC’s December 2023 Model Bulletin: Use of Artificial Intelligence Systems by Insurers already recognizes that oversight extends beyond an individual model. States consider adoption individually; the bulletin is not automatically binding nationwide.
Its governance provisions address the AI system lifecycle, ongoing monitoring, updating, data currency, lineage, and suitability. More pointedly, the bulletin’s examination-documentation provisions contemplate evidence that validation, testing, and auditing reflect the underlying AI system’s components, “whether based on Predictive Models or Generative AI.” That raises the reconstruction question: an insurer may have records about the model it tested without preserving the documents and retrieval behavior that shaped the recommendation under examination.
The question, then, is not whether regulators have recognized the importance of broader AI-system governance.
It is how those expectations apply to a system whose working information can change independently of the underlying model.
For a retrieval-based underwriting assistant, information controls have to distinguish documents that are merely available from documents authorized for a particular decision.
Effective dates and supersession status need to be represented in ways the system can actually use. Changes to the available document collection or retrieval rules may warrant renewed evaluation even when nobody has modified the model. And the organization needs records sufficient to establish what information was available to the system when a recommendation was made.
These controls are not guarantees of perfect decisions. They address identifiable dependencies in the technology.
There is a legitimate reason insurers want AI systems to retrieve current information rather than rely on static training knowledge. Updated underwriting appetite, new regulations, and revised operating procedures should influence recommendations.
The flexibility is valuable. It is also why the architecture surrounding the model matters.
In our hypothetical underwriting case, the carrier knew which model it was using. It knew the risk being evaluated and the underwriting rule that should have governed the recommendation.
What it failed to establish was which rule the system actually used.
Same AI model. Different decision. The difference wasn’t in the model itself, but in the information the system was given and the authority it assigned to that information.
Sources
- Anthropic, Effective Context Engineering for AI Agents, September 29, 2025.
- Jin et al., Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG, ICLR 2025 (PDF).
- Ahmed, Galke Poech, and Röttger, Stale-Document Poisoning: When Outdated Retrieval Overrides Correct Model Answers, September 2026 preprint.
- NAIC, NAIC Model Bulletin: Use of Artificial Intelligence Systems by Insurers, adopted December 4, 2023 (PDF).
AI Disclaimer: This content was created with assistance from artificial intelligence technology. While content is based on factual information from the source material, readers should verify all details directly with the respective sources before making business decisions.
