What is AI hallucination? A practical guide for businesses

A customer asks a chat assistant about your return policy. The assistant answers fluently, politely and with full confidence. The only problem is that the answer does not match your real policy. The customer relies on it, makes a decision, and later ends up in a dispute with your company.

This is called AI hallucination. Generative AI tools now write text, summarize documents and answer questions in more businesses every day. This article explains, in plain language, what hallucination is, why it happens, what it can cost a business, and the practical steps you can take to reduce the risk.

What is AI hallucination?

In its generative AI risk profile published in July 2024, the US National Institute of Standards and Technology (NIST) calls this behavior "confabulation": a system generating and confidently presenting erroneous or false content. In everyday language, people call it "hallucination".

The NIST definition goes beyond invented facts. Answers that drift away from the instruction or the document you provided, and statements that contradict what the system said earlier in the same conversation, also count. In practice, a model might quote a regulation clause that does not exist, summarize a contract term that is not in the file you uploaded, or quietly change a figure it gave two paragraphs earlier.

Why does it happen?

Hallucination is not a glitch. It is a natural result of how these models work. As NIST explains, large language models produce outputs that approximate the statistical patterns in their training data; roughly speaking, they predict the next word in a sentence. Most of the time this prediction gives accurate and consistent results, but sometimes it gives results that are factually wrong or internally inconsistent.

Instead of saying "I don't know", a model tends to build the answer that sounds most plausible. NIST notes that the risk grows with open-ended, long-form answers and with topics that require deep domain expertise. In a factory, that means technical specifications, regulations, warranty terms and price lists: exactly the areas where mistakes are expensive.

Another trap is persuasive reasoning. According to NIST, models can also produce invented citations or logical steps that seem to justify a wrong answer. A cited source does not make an answer correct.

What it can cost: the Air Canada case

A concrete example of the legal side comes from Canada. In the Moffatt v. Air Canada decision of 14 February 2024, the British Columbia Civil Resolution Tribunal looked at a case where the chat assistant on the airline's website told a customer he could apply for a bereavement fare after buying his ticket. The airline's own web page, however, said the policy did not apply to requests made after travel was completed.

The airline argued that it could not be held liable for information provided by the chat assistant. The tribunal rejected that argument, stating that the chat assistant was part of the company's website and that the company was responsible for all the information on it. The airline was ordered to pay a total of 812.02 Canadian dollars, including damages, interest and fees.

The amount was small, but the lesson is big: what a customer-facing AI tool says can be treated as what your company says. The real cost of such a case is not the money but the loss of reputation and trust.

Risky areas for factories and small businesses

Hallucination is not equally dangerous everywhere. A small error in the first draft of a marketing text is easy to fix. These are the areas that need the most attention:

• Answers to customers: prices, delivery times, warranty and return terms.
• Technical information: material properties, tolerances, maintenance instructions and safety steps.
• Regulations and contracts: regulation clauses, tender specifications and summaries of supplier contracts.
• Numerical reports: interpreting production, stock or cost data.
• Automated actions: flows where the AI output goes straight into another system as a record, with no human check.

NIST points to one more risk: automation bias. Over time, people start to trust automated systems more than they should and stop questioning the output. That makes it easier for a hallucination to slip into daily work unnoticed.

Practical ways to reduce the risk

You cannot eliminate hallucination completely today, but you can limit its impact a great deal. Among the actions NIST suggests, these are the most practical for a business:

1. Define the use case and its risk level. Write down where AI will be used. Producing a first draft is low risk; answering customers automatically is high risk. Set the level of control accordingly.

2. Ground the model in your own documents. Use methods that make the model answer from your current product catalog, price list and policies rather than from its general knowledge. The most common approach is to find the relevant document first and give it to the model before it answers. NIST also recommends verifying that this grounding data is accurate.

3. Make source checks mandatory. NIST recommends reviewing and verifying the sources and citations in system outputs, both before deployment and during ongoing use. Give your team a simple rule: no figure is used until someone has opened and checked its source.

4. Keep human approval for critical outputs. Quotes, technical instructions and contract summaries that go to customers should pass through an expert. AI saves time; a person makes the final call.

5. Test with real data. Before you roll out a tool, try it with questions whose answers you already know. NIST advises comparing outputs with known correct data and warns against drawing broad conclusions from narrow, anecdotal tests.

6. Open a way to report errors. Give employees and customers an easy channel to flag a wrong answer. Log every report and review the log regularly.

7. Write down the limits. Define the topics the assistant should not answer, and make sure it says something like "let me forward this to the right team" when it is unsure.

Frequently asked questions

Will a newer or more expensive model end hallucination? No. NIST describes hallucination as a natural result of how these models are designed. Model choice matters, but it does not replace good process design and control steps.

Is it safer not to use AI at all? Not always. Used in the right places with the right controls, AI saves a lot of time. The key is to keep human checks on high-risk work and to ground the tool in your own data.

How can I spot a hallucination? Be skeptical of exact figures, names, dates and citations. Open the source and check it. If you get a different answer when you ask the same question in a different way, treat that as a warning sign.

Conclusion

AI hallucination is an unavoidable feature of generative AI and a real business risk. The Air Canada decision showed that a wrong answer from a customer-facing tool can become the company's responsibility. The NIST framework shows that this risk can be managed by defining the use case, grounding the model in the right data, checking sources and keeping people in the loop for critical decisions.

If you want to use AI safely in your business, with a workflow grounded in your own documents and clear control steps, AtomAxis can help you take that first step.