If you’re a regular reader or work with me daily, you won’t be surprised to see another AI article from me. The space moves so fast that every week brings new headlines about how this technology is reshaping our world. Today, though, we’re tackling an increasingly problematic phenomenon: generative AI’s tendency to make confident assertions about things that are completely untrue.
When I say it’s getting worse, I am not referring to the technology (quite the opposite, we are making vast improvements in this space). Rather, the root of the issue is that more people are using generative AI technologies to acquire information that they use for important decision-making, and this information can be unreliable.
This isn’t a niche technical problem for developers to solve in a lab. It’s a real, present, and growing issue affecting real users, from those who use these technologies in a professional or educational environment, to users at home. The confident, articulate, and utterly false statements generated by Large Language Models (LLMs) like ChatGPT, Gemini, and others are what we call “hallucinations”. They are the fever-dreams of machines, and while they might sound benign, they are starting to have very real-world consequences.
Perhaps the most infamous cautionary tale comes from the legal profession. In a 2023 case, Mata v. Avianca, Inc., a law firm representing a plaintiff in a personal injury lawsuit submitted a legal brief to a New York federal court. The brief was compelling, citing over half a dozen previous court cases to support its arguments. The problem? Those cases were entirely fake.
The lawyers had used ChatGPT for legal research, and the chatbot, instead of admitting it couldn’t find relevant precedents, simply invented them—complete with bogus quotes and fictitious internal citations. The judge, P. Kevin Castel, described it as an “unprecedented circumstance”, and the fallout was severe. The client’s case was dismissed, and the lawyers were fined $5,000 for acting in bad faith and abandoning their professional responsibilities to a piece of technology that, as one of the lawyers later admitted, he was “unaware…could be false”.
This wasn’t an isolated incident. In a separate matter, Michael Cohen, the former lawyer for Donald Trump, admitted to using Google Bard (a precursor to Gemini), which similarly provided him with non-existent legal citations that were then submitted in a court filing. And it’s not just the legal world. In February 2024, Canadian airline Air Canada was ordered by a tribunal to compensate a customer based on a “bereavement fare” policy its own support chatbot had completely fabricated. The chatbot confidently told a customer they could retroactively apply for a discount, and the court ruled that Air Canada was liable for its chatbot’s hallucinated claims.
These examples are not edge cases; they are canaries in the coal mine. They illustrate a fundamental flaw in how these systems operate and underscore the urgent need for users—from students to CEOs—to understand what hallucinations are, where they come from, and how to build a defence against them. In this deep dive, we’ll unpack the machinery behind these confident charlatans, explore the root causes of their fictions, and, most importantly, lay out a roadmap for how you can use these powerful tools reliably and safely.
The Heart of the Machine: How LLMs Work and Why They Create Fiction
To truly grasp why a multi-billion dollar AI model can invent legal cases out of thin air, we need to pop the hood and look at its inner workings. The first and most critical thing to understand is that an LLM is not a thinking entity. It’s not a database of facts or a repository of knowledge in the human sense. At its core, an LLM is a spectacularly complex pattern-matching engine whose sole objective is to predict the next most plausible word in a sentence.
Imagine it this way: if you were to read the phrase “I’ll have my tea with…”, your brain, drawing on a lifetime of experience and understanding of the world, might suggest “milk” (even if we all know only monsters ruin a perfectly good tea with milk!), “sugar” (still contraverrcial, but I’ll allow it), and that options like “dimethylamidophenyldimethylpyrazolone” or “gunea pigs” are less likley. An LLM does something similar, but on an astronomical scale and without any real-world understanding. Its “experience” is its training data—a colossal dataset of text and code scraped from the internet, books, and articles, containing trillions of words.
This process begins with a foundation inspired by the human brain: an artificial neural network. This network is composed of layers of interconnected digital “neurons,” and each connection has a “weight,” a numerical value that determines the strength of the signal passing through it. During training, the model processes its vast dataset and constantly fine-tunes these weights. It learns that the token “fox” is statistically very likely to appear after the sequence “the quick brown”, so it strengthens the connections that produce that outcome.
It doesn’t learn what a fox is, what brown is, or why a fox might be quick. It only learns the statistical relationship between the tokens. As an aside, while I do know what brown foxes are, I have never seen one jump over a lazy dog, and only know this through repeated exposure to this text, so LLMs and I have that in common. All this to say the entirety of an LLM’s “knowledge” is a massive matrix of these statistical probabilities.
The technological leap that made today’s LLMs possible was the transformer architecture, introduced in 2017. Before transformers, models struggled to keep track of context over long sentences. The transformer architecture introduced a revolutionary concept called the attention mechanism. Attention allows the model to dynamically weigh the importance of different words in the input when it’s formulating a response.
For instance, in the text “the chicken did not cross the road because it was busy…”, the attention mechanism helps the model figure out (based on context) if it is the chicken that is busy, or the road. If the text that preceded this statement was “there was a lot of traffic in the city that day…”, the attention mechanism would modify its “understanding” that it was the road that was busy, based on available context. Now, we can see how these intricate systems, designed for linguistic fluency, become a breeding ground for hallucinations. The very process is built on probability, not truth.
The model is a master wordsmith, not a fact-checker. When it constructs a sentence, it’s stringing together words that are statistically likely to follow one another based on the patterns in its training data, and these patterns are not static. A factually incorrect statement can be generated with high confidence if its sentence structure is common and plausible, or if it has made an error when it was parsing the text (you might have a very different sentence if it understood the chicken to be busy, in our example above).
This is made trickier by how abstract the model’s “understanding” really is. It doesn’t see words—it sees tokens (pieces of words), each represented as a vector in a high-dimensional mathematical space. That might sound a bit sci-fi, but it’s actually not too hard to grasp. Think of a vector like a set of coordinates, and the high-dimensional space like a weird, invisible version of the Earth. Each “point” in this space has a meaning based on where it sits.
Take GPS coordinates: Hobart is about 42.8° south, 147.3° east. If you mirrored that latitude to the north (42.8°N) and kept the longitude the same, you’d land just east of Vladivostok, Russia. Despite being thousands of kilometres apart, both Hobart and Vladivostok share the same time zone in winter (UTC+10). Now compare Hobart with Chicago, which also sits around 42.8° latitude, but far to the west at 87.6°. Its climate has more in common with Hobart than, say, Houston, Texas, despite being in the same time zone and being in the same country (USA).
Much like places with similar geographic attributes have similar relationships in the two-dimensional GPS vector space, words, too, share similar spaces in the vector space in which they live. Binary opposites like “happy” and “sad” are analogous to similar coordinates on earth (such as Vladivostok and Hobart: opposite hemispheres but similar timezone and climate). While words with similar meanings are located near each other in this space, the subtle nuances of truth, falsehood, and context can be easily lost in this mathematical abstraction. Even the powerful attention mechanism can fail. In a very long, convoluted, or vague prompt, the model might fail to “attend” to a crucial negative or a key constraint you provided, leading it to generate an answer that completely misses the point.
Fundamentally, an LLM is an autocomplete system on steroids. It is an incredibly sophisticated mimic, capable of reproducing the patterns of human language with breathtaking accuracy. But it possesses no internal model of the world, no common sense, and no mechanism for distinguishing fact from fiction. It’s all just maths and probability, which is why it can so confidently and eloquently lead you astray.
The B’reishit (Genesis) of Error: Deconstructing the Causes of Hallucinations
While the core architecture of an LLM provides the fertile ground for hallucinations to sprout, several specific factors act as the seeds and fertiliser for these errors. When I advise organisations on AI implementation, I always stress that understanding these root causes is the first and most vital step toward building a strategy to mitigate them.
The most significant factor, without question, is the model’s training data. The adage “garbage in, garbage out” has never been more relevant. LLMs are a direct reflection of the data they consume, and the internet is a chaotic, unfiltered, and often unreliable source. It’s a soup of human brilliance and human folly. During its training, an LLM can ingest a diverse range of content, often with limited segmentation: peer-reviewed science, historical archives, and Pulitzer-winning journalism sit right alongside conspiracy theories, ancient forum posts with incorrect information, and biased blog rants. The model learns that a falsehood is a high-probability pattern if it appears frequently enough, internalising it just as deeply as any established fact.
Furthermore, this vast dataset is inevitably riddled with data voids. An LLM cannot know what it has not been shown. If you ask about a highly specialised engineering topic, a local community issue, or a news event that occurred after its last training update, it may have no relevant information. For instance, for months after Donald Trump was confirmed and inaugurated for his second term as president, LLMs (including those with web search capabilities were referring to him as “former US President, Donald Trump” (though I have found that many have caught up now).
But the model is not programmed to simply say, “I don’t know”. Its entire objective is to predict the next word, and given that most content it was trained on uses an active and authoritative voice, regardless of how legitimate that source is, the LLM will undertake this prediction with the tone of voice of an expert or authority. It will “fill in the gaps” by generating what it predicts is a plausible-sounding answer, weaving together tangentially related information into a coherent but entirely fabricated response. This is often the genesis of the fake legal cases or academic studies—the model “knows” what a legal citation or a study looks like structurally and generates text to fit that pattern, regardless of factual basis.
Another technical issue that contributes to this is a machine learning concept called overfitting. This occurs when a model learns its training data too well. It becomes brilliant at regurgitating specific examples and phrases it has memorised, but loses the ability to generalise its knowledge to new and unseen situations. In the context of an LLM, this can lead to it spitting out memorised chunks of text that are contextually inappropriate or nonsensical, simply because a part of the user’s prompt triggered a strong association with that specific piece of data.
Finally, the very act of generating a response involves a degree of controlled chaos through what is known as a decoding strategy. When an LLM writes text, it doesn’t just pick the single most probable next token every time, as that would lead to deterministic and robotic prose. Instead, a parameter often called “temperature” is used to inject a bit of randomness.
A low temperature makes the model conservative and focused, sticking to the most likely word choices. A high temperature increases the randomness, encouraging the model to be more creative and pick less obvious words. While this can lead to more novel and interesting outputs, it dramatically increases the risk of the model going off the rails into pure fiction. There is a constant tension between factual grounding and creative freedom, and a slight miscalibration can easily open the floodgates to hallucination.
The Bias in the Machine: How Source Selection and Algorithmic Prejudice Shape “Truth”
The problem of hallucinations is deeply intertwined with the equally critical issue of bias. This isn’t just about the model having a particular “opinion”; it’s a systemic, often invisible skew in its output that can reinforce harmful stereotypes, misrepresent entire communities, and present a single perspective as objective truth. When we use these tools to understand the world, this algorithmic prejudice is profoundly dangerous.
The original sin, once again, lies in the training data. At the time of writing, the vast majority of the web content used to train the major models is in English and originates from Western, developed nations. This means the model’s foundational understanding of history, culture, and social norms is inherently skewed. It will have a rich and nuanced grasp of Shakespeare but a shallow and likely stereotypical understanding of non-Western oral traditions.
Even within Western datasets you might assume to be authoritative (like mainstream news outlets, encyclopedias, and academic literature), the bias doesn’t disappear. Take Word2Vec, for example, trained on a massive corpus of Google News. If you ask it to complete “The man works as a _____” and you’ll get doctor, architect, engineer. Try “The woman works as a _____“, and you’re more likely to see nurse, receptionist, or housekeeper—and not uncommonly, prostitute. In one widely cited test (Bolukbasi et al., 2016), it solved “man is to computer programmer as woman is to _____” with homemaker.
These aren’t random glitches. They’re statistical truths, pulled directly from the training data—and that’s the problem. The model isn’t dreaming this up; it’s faithfully replicating biases it saw over and over again in the material it was fed. When that material is skewed, the model’s “understanding” becomes skewed too.
One might think that Retrieval-Augmented Generation (RAG)—a technique we’ll discuss more later, which grounds the LLM by having it retrieve information from an external source like a live web search before answering—would solve the bias problem. In theory, it grounds the model in current, factual data. In practice, however, RAG can sometimes inherit and even amplify bias.
This happens in several ways.
First, the retrieval system itself has biases. Search engine algorithms are not neutral arbiters of truth; they have their own complex ranking signals that can favour established, mainstream sources and marginalise alternative or minority viewpoints, even when those viewpoints are valid and relevant. The LLM’s answer is therefore built upon a pre-filtered, potentially biased selection of sources.
Second, the LLM can amplify the bias it finds in the retrieved content. If it pulls information from ten sources, and one of them contains subtly biased or stereotypical language, the model might latch onto that language and make it a central theme of its generated summary, giving it undue weight.
Finally, the problem of source scarcity is critical. When you ask about a topic where reliable, neutral sources are few and far between—perhaps a controversial political event in a country with limited press freedom—the RAG system is forced to pull from whatever is available. This could be state-controlled news agencies, highly opinionated blogs, or anonymous forums. The LLM, which lacks the human capacity to critically evaluate a source’s credibility or agenda, will treat this low-quality information as factual. It will then synthesise these biased sources into a confident, coherent, and utterly misleading response, effectively laundering propaganda into what appears to be an objective overview (with citations!).
The Quest for Ground Truth: Building a Framework for Reliable AI
Given these inherent flaws, it’s easy to feel pessimistic. However, as the initial shock of these new technologies wears off, we are developing a much more sophisticated understanding of how to work with and around their limitations. Achieving reliability isn’t about finding a “perfect” model; it’s about implementing a combination of better technology and smarter user practices.
Part A: The Power of Real-Time Information and Verifiable Sources
One of the most powerful developments in this area is the rise of search-augmented LLMs, a category that includes platforms like Perplexity AI and is a core feature of major chatbots from Google and Microsoft. These systems, sometimes employing what are known as FreshLLM patterns, fundamentally change the operating model from that of a closed-book exam to an open-book one.
Instead of relying solely on its static, pre-trained knowledge, a search-augmented LLM treats your prompt as a research assignment. The process is a game-changer for reliability:
- Deconstruct and Query: The model first analyses your prompt to understand your core intent and breaks it down into a series of optimised search queries.
- Live Web Search: It then executes these queries on a real-time web index, just like Google Search or Bing, pulling back a list of the most relevant pages.
- Synthesise and Augment: The model “reads” the content of these retrieved pages and uses this fresh, specific information as the primary context for generating its answer.
- Cite Everything: This is the most crucial step. The final answer is presented with clear, inline citations and footnotes that link directly to the source web pages it used.
This approach offers profound advantages for any user concerned with accuracy. It almost completely solves the problem of knowledge cutoffs, as the model’s information is as current as the internet itself. More importantly, it provides verifiability. The inclusion of sources is a fundamental shift in the user’s role. You are no longer a passive recipient of information from a black box; you are an active evaluator.
You can, and absolutely should, click on the sources to read the original context, judge the credibility of the publication for yourself, and verify that the LLM’s summary is accurate. This single feature transforms the LLM from an untrustworthy oracle into a transparent and auditable research assistant. While it doesn’t eliminate bias—as it’s still subject to the bias of search engine rankings—it exposes the sources of that bias for you to see.
Part B: Context Design and Engineering – The Art of the Masterful Prompt
Beyond the technology of the model itself, the single greatest instrument of control you have is the way you ask your question. The prompt is your steering wheel. I like to call the practice of crafting sophisticated prompts Context Design and Engineering. It’s about moving beyond simple, one-line questions and architecting a request that gives the model the context, constraints, and guardrails it needs to produce a reliable and useful result.
This is a skill, and it involves several layers of technique. At the most basic level, it means providing clear and specific instructions. Vagueness is an invitation for hallucination. Don’t ask, “Tell me about the economy”. Ask, “Summarise the key factors affecting Australian GDP growth in the last quarter, according to mainstream economists. Provide a balanced view, mentioning both positive and negative indicators”.
While most mainstream LLMs in the market today generally handle vague prompts much better than their counterparts from 12 months ago, there is one inescapable reality: they tend to perform better on and with input that is similar to what they were trained on. To understand how to take advantage of this, you need to know a little bit about the “secret sauce” that turns a raw, base LLM into a helpful assistant like ChatGPT, Claude, or Gemini. This process is a form of fine-tuning, often called instruction fine-tuning.
Think of the initial, massive training process (what we discussed earlier) as the model going to a library and reading every book, article, and website in the world. After this, it has an immense knowledge of language, facts, and concepts, but it’s like a brilliant but aimless scholar—it doesn’t necessarily know how to be a helpful assistant. It knows what a question is and what an answer is, but it hasn’t been taught how to be polite, safe, or how to follow specific, nuanced instructions.
That’s where instruction fine-tuning comes in. The AI labs take this raw model and put it through a second, highly specialised “finishing school”. In this phase, they don’t just feed it random text from the internet. Instead, they train it on a carefully curated dataset of high-quality examples. These examples are structured as instruction, desired_output pairs.
For instance, an instruction might be, “Explain the concept of photosynthesis in simple terms,” and the desired output would be a perfectly crafted, simple explanation often created by a human (though companies like Scale AI have made synthasising this content much easier and faster using, wait for it: other LLMs). They create tens of thousands of these examples, covering everything from writing poetry to explaining complex code to refusing to answer harmful requests. In the end, this is what an instruction-tuned LLM is doing when you provide a prompt. It sees your prompt and goes about completing it as if it were the first part of an instruction, desired_output pair.
A critical part of this is often a technique called Reinforcement Learning from Human Feedback (RLHF). In this stage, the model generates several possible answers to a prompt. Human reviewers then rank these answers from best to worst. This feedback is used to train a “reward model,” which essentially learns to predict which kinds of answers humans will prefer. The LLM is then further fine-tuned using this reward model as a guide, nudging its internal parameters to produce outputs that are more helpful, coherent, and aligned with human expectations.
In summary, the model first learns how to speak the language, then it learns how to structure its outputs to match the setting (a bit like a human learning a new language and then learning how to have a conversation vs writing an essay in the new language).
So, why does this matter to you as a user?
Because it fundamentally changes the model’s nature. It is no longer just a next-word predictor, like your phone’s autocomplete; it has been explicitly trained to recognise and follow instructions. This is where you can gain a massive advantage. When you write a prompt, you are not just providing a query; you are providing a new “instruction”. By structuring your prompt to mimic the format the model was fine-tuned on, you are playing to its strengths.
Let’s explore two powerful techniques specifically designed to combat hallucinations.
Technique 1: Grounding and Forcing Citation
One of the biggest reasons models hallucinate is that they try to answer questions from their vast, internal “memory” of training data, which might be outdated or incorrect. The solution is to forbid them from doing so. You can ground the model by providing it with a specific text and instructing it to answer only based on that information.
This is a powerful way to use an LLM for summarisation or analysis of a document you provide. Let’s see it in action. Imagine you have the following snippet of a press release:
“Flying Teabag Inc. today announced its third-quarter results. Revenue reached $50 million, a 15% increase year-over-year, driven by strong performance in its ‘Mainland Australia’ division. The company’s new ‘Project Big Dipper’ is slated for a Q1 2026 launch. CEO Lady Grey expressed optimism but noted that regulatory hurdles in the Asian market remain a significant challenge”.
Now, let’s compare two ways of prompting.
Summarise the latest news from Flying Teabag Inc.
This prompt is risky because it’s an open invitation for the model to “fill in the gaps.” It might pull outdated information from its training data about “Project Big Dipper” or invent details about the regulatory hurdles. So let’s try a more grounded, focused approach:
""" [Paste the press release here in tripple quotes] """
Based ONLY on the text I have provided above (inside the tripple quotes), answer the following questions.
1. What was the company's revenue and what was the growth rate?
2. What is the expected launch quarter for Project Titan?
3. What specific market presents a challenge?
For each answer, provide my question as a heading and then your answer, followed by a direct quote from the text as a citation to support your statement. If the answer to a question is not available in the text or you cannot find a specific quote that supports an answer, you must state 'Information not available in your source'.
This prompt is far superior for several reasons:
- It’s Grounded: The instruction
Based ONLY on the textstrictly forbids the model from using its internal knowledge. - It Forces Citation: Asking for a direct quote makes the output self-verifying. You can instantly see if the model’s answer is supported by the source.
- It provides an “Out”: The instruction to state
Information not availableis critical. It gives the model a safe and correct path to follow when it encounters a data void, preventing it from inventing an answer.
Technique 2: Structuring Prompts for Contentious Topics
This is where things get really interesting. What about topics where even a live web search might return a minefield of biased sources, misinformation, and conflicting narratives? A simple question on a contentious topic can easily lead to a biased or oversimplified answer.
Here, the goal of the prompt is to force the model to move beyond a simple “answer” and instead act as a neutral synthesiser of information, helping you understand the landscape of the debate. Let’s take the highly contentious 2020 U.S. Presidential election as an example. You might prompt your LLM with he following:
Did Donald Trump win the 2020 election?
This is a high-risk prompt. It demands a simple “yes” or “no” answer on a topic that is factually settled but mired in public controversy, and with a lot of content asserting that he did win (arguably more content) than the LLM might get distracted by. A simple “no” is factually correct regarding the certified result, but fails to inform the user about the context of the controversy they are likely asking about. A “yes” would be a dangerous piece of misinformation. The model is forced into a corner.
A safer a nd more structured approach that would likely be more resilient to hallucinating an answer (or, at best, providing a factually dubious answer in the event you were asking this very shortly after the event in question):
Provide a neutral and factual summary regarding the outcome of the 2020 US Presidential election by following these specific steps: 1. First, state the officially certified winner of the election as recognised by the US government and all 50 states. 2. Next, neutrally summarise the public claims made by Donald Trump and his campaign regarding the election's integrity. 3. Then, summarise the findings from major court rulings, as well as statements from election officials and federal agencies (such as CISA), in response to these claims. 4. Finally, conclude with a single sentence that summarises the gap between the claims and the official findings.This structured prompt is far more robust because it fundamentally changes the task.
- It Deconstructs the Question: It breaks a politically loaded question into a series of more objective, fact-based sub-queries.
- It Separates Fact from Claim: It explicitly asks the model to distinguish between the certified, legal outcome and the public claims made, treating them as two separate things to report on.
- It Demands Balance: By asking for the claims and the findings of official bodies, it forces the model to present both sides of the public conversation, which is essential for true understanding.
- It Prioritises Nuance over Simplicity: The goal is no longer a simple “win” or “lose” answer but a comprehensive overview that informs the user about the reality of the situation. This is how you use an LLM to build understanding rather than just get an answer.
By using techniques like Grounding and Structured Prompting for Contentious Topics, you are no longer just asking a question; you are engineering a reliable process. You are leveraging your understanding of how the model was trained to construct a prompt that guides it toward factual accuracy and away from confident fiction. This is the very essence of effective “Context Design and Engineering”.
The Corporate Brain: High Stakes and High Rewards with RAG on Internal Data
The same principles of grounding LLMs in external data can be applied within an organisation to create what many are calling a “corporate brain.” By using Retrieval-Augmented Generation over a company’s own internal knowledge bases—its reports, its documentation, its wikis, its intranet—businesses can create a powerful conversational interface to their own proprietary information.
The potential is immense: an engineer could ask for a summary of the last three performance tests on a specific software module, a new marketing hire could ask for the key takeaways from last quarter’s campaign analysis, and any employee could ask for a simple explanation of a complex HR policy.
However, while the rewards are high, the stakes are arguably even higher. An LLM hallucinating about public information is one thing; an LLM hallucinating about a company’s sensitive internal data or critical operating procedures is another entirely. I’ve worked with several teams building these systems, and the challenges are significant.
First and foremost is the issue of data quality and curation. The RAG system will only be as good as the documents it has access to. If your company’s shared drive is a chaotic swamp of outdated reports, contradictory policy documents, and unlabeled drafts, then the LLM will simply provide a more articulate and confident interface to that same chaos. A successful implementation requires a serious, ongoing commitment to data governance, ensuring that the knowledge base is clean, up-to-date, and well-structured (and, most importantly, well understood).
Second, access control and security are paramount. When dealing with sensitive, role-specific information, the system must be architected to rigorously enforce user permissions. The RAG system needs to know that an employee in the sales department does not have the clearance to ask the “corporate brain” to summarise the minutes from a confidential board of directors meeting.
Finally, the risk of internal misinformation must be actively managed. Imagine an employee at a manufacturing plant asking the system for a safety procedure and the LLM confidently hallucinating an incorrect step that could lead to injury or equipment damage. For critical applications, this means rigorous testing, validation, and often keeping a “human in the loop” to review the AI’s output before it’s acted upon. Despite these hurdles, a well-designed internal RAG system, guided by meticulous Context Design and Engineering, can be a truly transformative asset for an organisation.
A Mindful Approach: Your Guide to Seeking Truth in the AI Age
We have journeyed deep into the machinery of language models, explored the roots of their fabrications, and outlined the technological and user-side strategies to fight back. We’ll wrap up with a practical guide for you, the mindful user. How should you approach these tools, especially when the information really matters? The fundamental shift required is to move from being a passive consumer to an active, critical interrogator of the AI’s output.
When you’re dealing with high-stakes information, your level of scepticism should be at its highest. This is especially true in several key domains. For news and current events, the sheer speed at which information develops makes it a minefield. Early reports are often contradictory, and misinformation spreads faster than truth. Rely on established, reputable news organisations and be deeply sceptical of LLM-generated summaries of breaking news.
For medical, legal, and financial advice, the answer is simple: don’t use a standalone LLM as your primary source. A hallucinated piece of medical advice from a chatbot, which studies have shown can happen, could have devastating consequences. The risk is not worth the convenience.
Finally, in academic and scientific research, while these tools can be phenomenal research assistants, they are notoriously unreliable sources themselves. There are documented cases in research papers where LLMs have invented dozens of non-existent references. As a rule, you must always track down and verify the original source for any academic claim.
To help you in this process, I recommend internalising a simple, five-step checklist every time you receive a piece of information from an LLM that you intend to rely on.
- Demand the Source. If you are using a search-augmented system, are there clear citations? If not, be immediately sceptical. If there are, your work has just begun. Click on them. Read the original source material. Does the LLM’s summary accurately reflect the nuance and context of the source?
- Run a Plausibility Check. Does the information pass a basic common-sense test? If an AI tells you something that seems extraordinary, revolutionary, or too good to be true, it almost certainly is. Extraordinary claims require extraordinary evidence.
- Insist on Corroboration. Never, ever rely on a single source, even if that source is a sophisticated AI synthesising multiple other sources. Take the core claims from the LLM’s response and verify them independently. Use a traditional search engine, check with known experts, and look for multiple, reputable sources that confirm the information.
- Acknowledge the Potential for Bias. Always consider the topic. Is it a controversial or politicised issue? Be actively aware that the LLM’s response may be skewed based on the dominant perspective in its training data or the top search results. Consciously seek out alternative and dissenting viewpoints.
- Interrogate Your Prompt. Finally, look at your own role in the conversation. Could the way you phrased your question have been ambiguous? Could it have led the model astray? Try rephrasing your prompt with more detail, a different persona, or clearer constraints and see if the answer changes in a meaningful way. or even take the output from one LLM and get the second LLM to fact check/validate the first output (though, keep in mind that some LLMs have universal biases that could be manifesting across both LLMs.
Final Thoughts
AI hallucinations are not a temporary bug that will soon be “fixed” (and, in my opinion, is not a bug at all, despite being a problematic element to have to contend with, much like gravity was to the Wright brothers during the early days of aviation). They are a fundamental and emergent property of how the current generation of Large Language Models works. They are a direct consequence of their probabilistic architecture and their training on the vast, messy, and beautifully flawed corpus of human language. They are, at their core, sophisticated mimics, not truth-seekers.
This reality, however, should not lead us to discard these tools. It should lead us to engage with them more wisely. The future of reliable AI is not a future where the models are perfect; it is a future where the human-machine interface is built on a foundation of transparency, verifiability, and critical thinking. Systems that integrate real-time search and provide meticulous sourcing are a major leap forward, transforming the LLM from an opaque oracle into a powerful and auditable assistant.
Ultimately, the most critical component in this entire system is, and will remain, you. By cultivating a habit of mindful scepticism, by mastering the art of Context Design and Engineering, and by refusing to outsource your own critical judgment, you can safely and effectively navigate this new technological frontier. The goal is not to find an AI that will give you all the answers, but to use these remarkable tools to help you ask better questions and to find those answers for yourself.
And remember, no matter what an LLM tells you, there is simply no justifiable reason to put milk in black tea.
