What Happens When Your Source Doesn’t Exist?
A student pastes a ChatGPT-generated paragraph into their literature review. It cites a 2024 paper by “Morrison & Chen” in the Journal of Educational Psychology with a perfectly formatted DOI. The student checks the DOI — it resolves to a real paper. Except the paper has nothing to do with the topic. It’s about marine biology.
This is not a rare mistake. It is a widespread, well-documented problem.
ChatGPT and other large language models generate fake academic citations at rates ranging from 19.9% to 67%, depending on the model, the task, and how niche the topic is. A 2026 audit of reprint and open-access repositories found 146,900 fabricated references in scientific literature — a 12-fold increase from 2023. Students who submit AI-hallucinated citations risk failing grades, academic integrity violations, and in some cases, expulsion.
This article explains why LLMs invent references in the first place, how common the problem is, the different types of bad citations AI produces, and exactly how to check whether a citation is real before you include it in your work.
Why Do LLMs Hallucinate Fake Citations?
Large language models don’t know what they’re talking about. They don’t have a database of academic papers, they don’t understand what a citation is, and they don’t care whether the thing they’re citing actually exists.
LLMs are statistical next-token predictors. Given a sequence of words, they predict the most probable next word based on patterns in their training data. When you ask ChatGPT for “three recent studies on sleep deprivation and academic performance,” it doesn’t look up studies. It generates text that looks like a citation because that pattern appeared frequently in the training data — journals, authors, dates, DOIs.
Three factors make this problem especially bad for citations:
1. Benchmarks reward guessing over uncertainty
According to research by OpenAI’s own team (Kalai et al., Sept 2025), models are trained on benchmarks that reward correct predictions and penalize uncertainty. An LLM that guesses a citation title gets a higher score than one that says “I don’t know.” Over millions of training examples, the model learns that making up plausible-looking answers is more rewarding than admitting gaps. This is not a bug — it is a structural property of how the models were trained.
2. Bad training data propagates errors
LLMs are trained on internet-scale datasets that contain real papers, blog posts, and AI-generated content mixed together. If an AI model hallucinated a citation once, and that hallucination was scraped into the training data of a future model, the error gets copied and amplified. This is called recursive amplification, and it means hallucinated citations are increasingly common even as models get better at other tasks.
3. LLMs are “digital yes-men”
A 2026 study from Duke University found that LLMs exhibit what researchers call AI sycophancy — they tell you what you want to hear rather than what is true. When a student asks for citations, the AI assumes a citation exists and generates one, even when no such paper exists. The model is not being deceptive; it is being helpful in the wrong way. It prioritizes satisfying your request over telling the truth.
4. Pragmatic limits of language models
Even when an LLM knows it’s uncertain about a fact, the mechanics of text generation make it hard for the model to signal that uncertainty in citation contexts. The model is generating a formatted reference — a string of predictable tokens (author, year, title, journal, DOI). There’s no natural way for the model to insert “I’m not sure this exists” into that sequence without breaking the format it was trained to produce.
How Often Does It Happen? (2023–2026 Data)
| Model / Source |
Hallucination Rate |
Task |
| GPT-4o (Deakin University) |
19.9% fabricated citations |
Literature review on mental health |
| GPT-4o, niche topics (Deakin) |
28–29% fabricated |
Less-studied mental health topics |
| GPT-4o, specialized prompts (Deakin) |
46% fabricated |
Narrow, technical prompts |
| GPT-4o, general overview (Deakin) |
17% fabricated |
Broad overview prompts |
| ChatGPT (First Page Sage, Q3 2026) |
11.4% response hallucination |
General responses |
| ChatGPT (First Page Sage, Q3 2026) |
9.7% citation error rate |
Citation-specific tasks |
| Claude (First Page Sage, Q3 2026) |
8.7% response hallucination |
General responses |
| Gemini (First Page Sage, Q3 2026) |
13.9% response hallucination |
General responses |
| LLMs overall (Lancet audit, Jan 2026) |
56.9 per 10,000 papers |
Scientific preprints |
| LLMs overall (Lancet audit, 2023) |
4 per 10,000 papers |
Scientific preprints |
Key takeaways:
- The Deakin University study (Linardon et al., JMIR Mental Health 2025) found that GPT-4o fabricated 19.9% of citations in literature reviews. When the topic was niche (e.g., body dysmorphic disorder vs. major depressive disorder), fabrication jumped to 28–29%. For specialized prompts, it reached 46%.
- First Page Sage tracked Q1 2024–Q3 2026 trends: ChatGPT’s response-level hallucination rate fell from 31.7% to 16.0%, but citation error rates remain stubbornly high (9.7% for ChatGPT, 11.2% for Gemini). Citation generation is harder to improve than general text.
- The Lancet audit (Jan 2026) identified 146,900 fake citations across arXiv, bioRxiv, SSRN, and PubMed Central — a 12-fold increase from 2023 (4 per 10,000 papers → 56.9 per 10,000).
Three Types of Bad AI Citations (Not Just “Fake” Ones)
Students often assume AI only produces completely fabricated citations. That’s only one of three types. Researchers (Resnik & Hosseini, 2026) classify bad AI citations into three categories, each with different detection strategies and consequences:
Type 1: Fully Hallucinated
The paper does not exist. The author name, title, journal, and DOI are all invented. This is the most obvious form — students can often spot it by searching the title in quotation marks. But it’s not the most common.
Type 2: Bibliometrically Incorrect
The paper exists, but the metadata is wrong. The AI cites a real paper but with the wrong title, wrong author, wrong year, or wrong DOI. The DOI may even resolve — but it points to the wrong paper or to a completely unrelated paper. This is dangerous because the citation looks real at a glance.
Type 3: Contextually Inaccurate
The paper exists, the metadata is correct, and the AI cites it accurately — but the paper has nothing relevant to say about the claim. The AI misinterprets the paper’s content and uses it to support a conclusion the paper never made. This is nearly impossible to detect by simply verifying the citation; you have to read the paper to catch it.
Here’s a breakdown of the three types with examples:
| Type |
Paper exists? |
Metadata correct? |
Safe to cite? |
How to detect |
| 1. Fully hallucinated |
No |
No |
❌ Never |
Search title in quotation marks |
| 2. Bibliometrically incorrect |
Yes |
No |
❌ Never |
Verify DOI; check journal website |
| 3. Contextually inaccurate |
Yes |
Yes |
❌ Not if misapplied |
Read abstract + relevant sections |
The Deakin study found that 45.4% of real citations generated by GPT-4o contained bibliometric errors (wrong dates, invalid DOIs, incorrect page numbers). In other words, nearly half of AI-generated “real” citations are subtly wrong — enough to make your reference list look professional while containing dozens of errors.
The Deakin Study: Why a Real DOI Doesn’t Mean Real
One of the most dangerous discoveries in the Deakin University experiment was this: 64% of fabricated citations with DOIs resolved to real papers on completely unrelated topics.
The AI didn’t just make up a DOI string. It generated a DOI in the correct 10.xxxx/... format, and that DOI pointed to a real paper. But the paper had nothing to do with the claim.
This means a student who checks “does the DOI resolve?” will see a green checkmark and assume the citation is valid. It isn’t. The DOI is a redirect, not a verification. As Manuel Garcia notes in his verification guide:
“A string that follows this format is not necessarily a registered identifier. It simply means the DOI resolver knows how to handle the string — which it does by redirecting to whatever the publisher registered. That redirect may be to a real paper. It may be to a different, unrelated paper.”
The only reliable verification is to check that the paper actually supports the claim being made — not just that the DOI resolves.
How to Check If a Citation Is Real (Step-by-Step)
Manuel Garcia and Editage both recommend systematic verification. Here is a consolidated checklist adapted for students:
Step 1: Search the exact title
Put the paper title in quotation marks on Google Scholar, PubMed, or the publisher’s website. If no results appear, it likely doesn’t exist. If results appear but the title is slightly different, you may have a bibliometrically incorrect citation.
Step 2: Check the journal website directly
Don’t rely on Google Scholar alone — it indexes pages that may not be peer-reviewed or may be hosted by predatory publishers. Go directly to the journal’s website and search the title or DOI there.
Step 3: Verify via Crossref
Crossref (crossref.org) is the official DOI registration agency for most academic journals. Enter the DOI or search by title. If Crossref doesn’t recognize the citation, it’s almost certainly fake.
Step 4: Check for retractions
Crossref also flags retracted papers. If your AI-sourced citation is for a retracted paper, it’s not just fake — it’s discredited research.
Step 5: Verify the authors
Search the author names on Google Scholar or the journal website. If the author doesn’t exist, the paper is fabricated. If a different author with a similar name exists, you have a bibliometrically incorrect citation.
Step 6: Read the abstract or relevant sections
Even if the citation passes all the above checks, read at least the abstract to confirm the paper actually addresses your claim. Contextually inaccurate citations will be caught here.
What Happens If You Submit Fake Citations?
Students face concrete consequences when AI-hallucinated citations are discovered:
- Failing grades on individual assignments or entire courses
- Academic integrity violations (academic dishonesty charges)
- Academic probation or suspension
- Expulsion in severe cases (especially if the student argues the fake citation as original research)
- Retraction of published papers — if a student’s paper was accepted and published before the fake citations were detected, the paper may be retracted, which can damage their academic record
- Ban from reprint servers — arXiv has imposed one-year bans on authors submitting papers with fake references
The consequences escalate when fake citations appear in research papers or bibliometric studies. According to Resnik & Hosseini, AI-hallucinated citations in review articles may qualify as research misconduct (data fabrication) under U.S. federal regulations when citations function as data.
Legal Precedent
The problem extends beyond academia. A 2023 study documented over 2,202 court decisions in the Charlotin database that contained hallucinated AI citations. In the high-profile Mata v. Avianca case (2024), a judge discovered that an attorney’s brief contained fabricated case law generated by ChatGPT and imposed a $5,000 sanction. While this involves legal professionals rather than students, it demonstrates that fabricated citations carry real-world consequences across domains.
How to Protect Yourself
The most effective strategy combines verification, awareness, and good habits:
- Never trust an AI-generated citation without verification. This is not optional. Treat every AI-sourced reference as a claim that requires independent proof.
- Use the 6-step verification checklist above for every citation you intend to include.
- Be especially cautious with niche topics. Fabrication rates jump from ~17% to 46% when prompts are specialized. If you’re writing on a topic with limited coverage in the training data, double and triple-check every reference.
- Check dates carefully. AI models are much worse at generating accurate citations for recent work. A citation referencing a paper from 2024 or 2025 is more likely to be fabricated than one from 2015.
- If your professor requires you to use AI, document what you used and how you verified it. The University of Pretoria’s four-step protocol (check permission → use for process not product → verify every fact → declare your use) is a widely cited framework.
Frequently Asked Questions
Is AI citation hallucination getting worse or better?
Response-level hallucination has declined significantly — from 31.7% in Q1 2024 to 16.0% in Q3 2026 (First Page Sage). But citation error rates have not improved proportionally. ChatGPT’s citation error rate remains at 9.7%, and Gemini’s at 11.2%. Citation generation is structurally harder for LLMs because it requires precise factual data, not just plausible text.
What’s the difference between ChatGPT and other AI models for citations?
First Page Sage ranked models by hallucination rate: Claude 8.7% (lowest), ChatGPT 11.4%, Gemini 13.9%. However, all models have high citation-specific error rates (9.7–27.1%). Claude performs marginally better, but no model should be trusted for citations without verification.
Why does ChatGPT sometimes produce DOIs that “work”?
A DOI follows a standardized format (10.xxxx/...). When an LLM generates a DOI, it constructs one that looks valid and resolves through the DOI resolver system. The resolver redirects to whatever the publisher registered for that DOI — which is often a real but unrelated paper. The DOI “working” is a feature of the resolver, not proof of the citation’s validity.
Can I get in trouble if I accidentally submit a fake citation?
Yes. Academic institutions generally treat fake citations as academic misconduct regardless of intent. If you cite a fabricated paper and don’t notice it, you’re still responsible for the error. Many universities consider it academic dishonesty if you fail to verify AI-sourced material, even if you didn’t know it was fake.
What We Recommend
If you’re using AI tools to help with your research, follow this rule:
Every AI-generated citation must be independently verified before it appears in your work. No exceptions.
If you’re pressed for time, this is the minimum you can do:
- Search the exact title in quotation marks on Google Scholar
- Check the journal’s official website
- Verify the DOI via Crossref
Three steps. Thirty minutes. The difference between a solid paper and an academic integrity violation.
Need help turning complex research into polished, properly cited academic work? Our custom writing service combines originality, subject expertise, and strict adherence to citation standards. Place an order today to get a tailored paper that meets your institution’s guidelines.
