Anthropic Asked Claude for a Citation. Claude Made One Up.

On a Thursday, in a federal court in Northern California, a lawyer representing Anthropic filed something you don't see often: an apology. The apology was for a citation that never existed, built by the company's own chatbot, in a filing meant to defend the company.

The case is Anthropic's ongoing copyright fight with music publishers, including Universal Music Group. Somewhere in the paperwork built to help Anthropic's side was a citation that Claude had simply invented and handed over, formatted and confident, like it always does.

Anthropic said so itself. In the filing, the company admitted Claude "hallucinated the citation" with an inaccurate title and inaccurate authors. The source was never real to begin with, dressed up to look like one.

Here's the part that should make you put your coffee down. Anthropic's lawyers had a safeguard for exactly this. They ran a manual citation check. It did not catch this error, nor did it catch several other errors Claude's hallucinations had caused. A human sat down, presumably with the actual job of catching fake citations, and the fake citation got past them anyway.

Anthropic apologized and called it "an honest citation mistake and not a fabrication of authority." A fine distinction to draw, and probably a necessary one for a company being sued over how its models handle copyrighted material. It's the kind of line that means a great deal to the lawyers writing it and roughly nothing to the judge now holding a filing with a made-up source in it.

The mechanism, again

A language model knows what a citation looks like. That is the whole of its knowledge. Ask it for a source and it produces something shaped exactly like one, a title and authors as plausible as anything sitting on a real shelf. Whether the thing exists is not a question the model is built to ask itself. It optimizes for the next word that fits, and a fabricated title fits just as smoothly as a real one. Fluency is the product. Truth is a separate line item, and nobody ordered it.

What makes this one sting is who got caught by it. This was Anthropic, the company that built Claude, that presumably understands its failure modes better than anyone on Earth, filing a document in its own defense with a hallucinated citation sitting inside it. The fox built the henhouse.

And they still had a human check the work, and the human still missed it. The lawyer doing the checking did the job. The problem is catching hallucinations by hand at all: a fabricated citation is built to look exactly like a real one. This is harder than proofreading for typos. You're trying to spot the one reference, out of a stack of correct ones, that was never written by anyone at all. Eyes get tired. Formatting looks fine. It slides through.

It got worse

This wasn't Anthropic's only run-in with its own chatbot in this case. Lawyers representing Universal Music Group and the other publishers accused Anthropic's expert witness, Olivia Chen, an Anthropic employee, of using Claude to cite fake articles in her testimony. Same tool. Same case. A separate person, hit by the same failure mode.

Federal judge Susan van Keulen ordered Anthropic to respond to the allegations. So somewhere in the docket of one lawsuit, a company is now on record twice, explaining to a judge why its own AI kept inventing sources for the people paid to argue on its behalf.

Meanwhile, the market keeps betting the other way. Harvey, a company that uses generative AI models to help lawyers with exactly this kind of work, is reportedly in talks to raise over $250 million at a $5 billion valuation. Somewhere between those two stories is the whole shape of the moment: an industry raising billions to put more of this in more filings, while the company that arguably knows the failure mode best just got caught by it twice in the same case.

The 2am version

All this took was a fluent chatbot, a citation-shaped sentence and a manual check that trusted its own eyes a little too much. That's the trap, and it doesn't care whose name is on the letterhead. It caught a company that built the technology in the first place.

If it can happen to the people who trained the model, it can happen to you writing a term paper at 2am with six tabs open and a deadline in four hours. The citation your chatbot hands you will look exactly right, a real-sounding author and a title that fits your argument, in formatting indistinguishable from every real source you've ever copied down. You won't know it's empty until someone downstream goes looking and finds nothing there. Anthropic found out in front of a federal judge, twice. Most people find out in smaller rooms. It stings just the same.

A sharper eye won't save you. Eyes get tired, and a fabricated source is built to survive a skim. The fix is checking every citation against the record that actually exists — the paper is real and still says what you think it says, no quiet retraction since you found it — before you file anything with a judge's name on the other end of it.

Boring is the goal. Boring is never having to write "an honest citation mistake" in a filing with your name on it.