Why You Think AI Doesn’t Work (And Why You’re Probably Wrong)

By James W. Moore | InsuranceIndustry.AI

Key Takeaways

• Free AI tools are entry-level products, not representative of what enterprise-grade AI can actually do.
• Without paid plan features like persistent projects and custom instructions, you’re starting from zero every single conversation.
• Prompting is a learnable skill. Weak inputs produce weak outputs, and most first-time users never learn the difference.
• Capability gaps between AI models are significant. An impression formed two years ago may be completely obsolete today.
• The most common AI “test” executives run — trying to catch it making a mistake — is not a real evaluation.

Introduction: The Skeptic in the Room

You’ve tried it. Maybe a colleague showed you a demo, or you pulled up ChatGPT one evening and typed in a few questions. The answers were okay. Maybe even impressive at first. But then it got something wrong, or gave you something generic, or just didn’t seem all that useful for what you actually do.

So you filed it away. Interesting technology, not ready for prime time.

Here’s an uncomfortable question: Was the test actually fair?

I’m not asking to be provocative. I’ve spent over 40 years in this industry, and I understand the instinct to be skeptical of the next wave of “transformational” technology. That skepticism has served insurance professionals well more than once. But this time, I think a lot of executives are drawing conclusions about a technology they’ve never actually evaluated properly. They’ve evaluated a watered-down, context-free, first-generation version of it — and called it a day.

Before you put AI in the “not ready” column, it’s worth asking whether your experience reflects the technology’s real ceiling, or just the floor.

The Free Plan Problem

Let’s start with the most common scenario: someone signs up for the free tier of ChatGPT, Claude, Gemini, or Grok, spends twenty minutes with it, and forms a lasting opinion.

Free plans exist to give you a taste of the product. They are not designed to give you a representative experience of what the technology can do in a business context. The differences matter.

Free tiers typically run on older, less capable underlying models. They impose strict limits on how much information you can include in a single conversation — what’s called the “context window.” A narrow context window means the AI can’t hold much information in mind at once, which makes it more likely to lose track of your question, contradict itself, or fill in gaps with plausible-sounding but incorrect information. That’s where hallucinations come from. It’s not a random character flaw of AI; it’s a predictable consequence of asking a constrained model to answer a complex question.

Paid plans — typically ranging from $20 to $30 per month for individual users, with enterprise pricing for organizational deployments — run on the most current models with substantially larger context windows. The difference in output quality isn’t incremental. It’s significant.

If you’ve only ever driven the base trim, you haven’t test-driven the car.

Without Projects, You’re Starting from Zero Every Time

This is the feature gap that may frustrate AI users more than any other, and it’s one most people don’t even know exists.

Most free plans treat every conversation as a blank slate. The AI has no memory of your previous sessions, no knowledge of your organization, no standing instructions, and no reference documents. You type a question, it answers, and when you close the window, everything resets.

Paid plans — and particularly enterprise configurations — include what most platforms call “Projects.” This is where AI becomes genuinely useful as a business tool rather than a novelty.

With a properly configured project, you can give the AI persistent instructions that apply to every conversation. You can upload reference documents: your underwriting guidelines, your agency’s standard workflows, policy forms, carrier appetite guides. You can tell it your preferred tone, your audience, your terminology. You can build a customized assistant that understands your business context from the moment you open a conversation.

The difference is enormous. Without projects, using AI for business tasks is like hiring a consultant who forgets everything between meetings. With projects, it’s like working alongside a knowledgeable colleague who has read every document you’ve ever handed them and remembers every conversation you’ve had.

Most executives who are unimpressed with AI have never used a project. They’ve been judging the fully configured version by the out-of-the-box experience.

Prompting Is a Skill, and Most People Don’t Have It Yet

There’s a reasonable assumption many people make when they first use AI: that it works like a search engine. You type something in, it finds the answer.

It doesn’t work that way. AI responds to the quality and specificity of what you give it. A vague question produces a vague answer. A well-constructed prompt produces something genuinely useful.

Here’s the difference in practice.

Weak prompt: “Write me something about cyber liability.”

You’ll get a generic paragraph that wouldn’t pass muster in a high school insurance class, let alone impress a sophisticated commercial client.

Stronger prompt: “Write a two-paragraph explanation of cyber liability coverage for a mid-sized regional manufacturer with 200 employees who is skeptical that they face meaningful cyber risk. The tone should be conversational but authoritative, and it should address the specific exposure created by operational technology and connected manufacturing equipment. Avoid jargon.”

That prompt produces something you could actually use. Same technology. Completely different output.

Prompting is not a technical skill. It doesn’t require any programming knowledge. It requires clarity about what you want, who it’s for, and what constraints apply. Those are skills every experienced insurance professional already has. They just need to apply them differently.

The good news: prompting improves quickly with practice. Most people who invest a few hours in learning the basics see immediate, meaningful improvement in results.

You May Be Using an Outdated Model

AI capabilities have advanced faster in the past two years than in the prior decade combined. The difference between frontier AI models available today and what was publicly accessible in 2022 or 2023 is not a matter of degree. It’s a different category of tool.

If your opinion of AI was formed during an early encounter with a first-generation chatbot, you haven’t evaluated what exists today. It would be like forming your opinion of smartphone technology by using a Blackberry in 2007 and deciding mobile computing wasn’t ready.

Current frontier models — the top-tier offerings from Anthropic, OpenAI, Google, and xAI — demonstrate reasoning capabilities, contextual understanding, and reliability that would have seemed implausible just a few years ago. They are not perfect, and they should always be reviewed before anything they produce goes out the door. But they are genuinely capable business tools in the hands of someone who knows how to use them.

If you haven’t revisited AI in the past twelve months, your mental model of what it can do is probably wrong.

AI Works Best as a Collaborator, Not an Oracle

Here’s a mindset shift that separates executives who get real value from AI from those who don’t: the ones who succeed treat AI as a thinking partner, not a vending machine.

A vending machine interaction looks like this: type question, read answer, accept or reject. If the answer isn’t immediately perfect, the tool gets a failing grade.

A collaborative interaction looks like this: give the AI context, review what it produces, push back on what’s wrong, refine the direction, ask it to try again from a different angle. The output improves through iteration, the same way it would if you were working through a problem with a colleague.

Senior executives are already skilled at this kind of iterative dialogue. They do it in meetings, in negotiations, in strategy sessions. The skill transfers directly to AI. The executives who are most unimpressed with AI are often those who’ve applied the least of what they already know about productive professional collaboration.

The Compliance Wall That Blocked Everything

Many organizations received a blanket prohibition on AI use from legal, IT, or compliance. Sometimes the direction came from the top without much explanation. The concern was data security, client confidentiality, or regulatory risk — all legitimate concerns, none of them wrong to raise.

The problem is that these prohibitions were often issued without distinguishing between consumer tools and enterprise-grade deployments.

There is a meaningful difference between typing a client’s name and policy details into a free public chatbot and deploying an enterprise AI platform with proper data governance, access controls, and contractual data protections in place. Treating these as equivalent is like saying no one in the organization can use email because someone once sent sensitive information to the wrong address.

If your organization’s AI policy was written in 2023 based on early consumer tools, it may be time to revisit it with current enterprise options in mind. The regulatory landscape has also matured. The NAIC’s model AI governance framework and emerging state-level guidance provide a clearer framework for responsible AI use than existed when many of these blanket policies were written.

You Tested It to Break It, Not to Use It

There’s one more pattern worth naming directly.

A common executive response to a new technology is to probe for failure. You ask it a trick question. You give it a test it might fail. You wait for the hallucination. When it comes — and sometimes it does — you log it as evidence and move on.

That’s a reasonable instinct from a risk management perspective. But it’s not an evaluation.

The relevant question isn’t “can I make this fail?” It’s “does this save me meaningful time on work I actually do, while producing output I can stand behind after review?” Those are very different tests, and only one of them tells you whether the technology has a place in your operation.

Before You Conclude AI Isn’t Ready: A Quick Checklist

Run through these questions honestly.

1. Have you used a paid plan (not just the free tier) for at least two weeks?
2. Have you configured a project with your organization’s context, terminology, and standing instructions?
3. Have you spent any time learning basic prompting techniques, or have you approached it like a search engine?
4. Is the AI model you evaluated among the current generation of frontier models, or was your experience with an older tool?
5. Have you tried using AI iteratively — refining and pushing back on outputs — rather than expecting a finished product on the first try?
6. Has your organization distinguished between consumer AI tools and enterprise-grade deployments when setting policy?
7. Have you assigned AI a real task you do regularly and measured whether the output, after your review, was useful?

If you answered no to three or more of these questions, your conclusion about AI may not be based on a fair test.

What to Do Next

You don’t need to commit to an enterprise transformation to find out whether AI is worth your attention. Start with three low-risk steps.

First, sign up for a paid individual plan on one of the major platforms for a single month. The cost is nominal. Use it for something you actually do: drafting correspondence, summarizing a document, preparing talking points for a meeting.

Second, spend one hour learning how to write a useful prompt. Anthropic publishes a free prompting guide that covers the fundamentals clearly and without jargon.

Third, set up a project. Give it your business context. Upload a reference document you use regularly. Then run the same task you ran before and compare the results.

If you do those three things and AI still doesn’t impress you, then your skepticism is informed. Until then, you may be judging a fully loaded tool by the spare-tire experience.

Sources

• National Association of Insurance Commissioners (NAIC). Artificial Intelligence Governance White Paper. December 2023. https://content.naic.org/sites/default/files/inline-files/2023-12-AI-Governance-White-Paper.pdf
• Anthropic. Prompt Engineering Overview. Claude Documentation. https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/overview

AI Disclaimer: This blog post was created with assistance from artificial intelligence technology. While the content is based on factual information from the source material, readers should verify all details, pricing, and features directly with the respective AI tool providers before making business decisions. AI-generated content may not reflect the most current information, and individual results may vary. Always conduct your own research and due diligence before relying on information contained on this site.