Transcript-Anchored Drafting Beats Verifiers for Hallucinations

A review of LLM reliability found that chatbots fabricate facts approximately 27% of the time, and 46% of generated texts contain factual errors. That baseline makes verification the critical step: no matter how fluent a model is, it can confidently deliver invented minor-character details.

The standard fixes—prompt engineering and retrieval augmentation—help but do not eliminate the problem. Strategic prompting can reduce hallucination rates by up to 36%, and chain-of-thought reasoning boosts complex-task accuracy by up to 30%. Yet naive retrieval-augmented generation introduces retrieval errors and context overfitting, so a model can still fabricate when the wrong document is retrieved or loosely related passages are overinterpreted.

Transcript-anchored drafting takes a different route: before generating an answer, the generator is grounded in a verified transcript or structured dataset, and a separate judge checks the final output against that anchor. Because the bottleneck is verification rather than generation, anchoring the draft to an external source removes almost all hallucination without needing a larger model. That is the design that matters for transcript-heavy domains such as Cheers trivia.

Transcript-Anchored Drafting

The load-bearing decision in this pipeline is not the verifier — it's the drafting constraint. Forcing GPT-4o at temperature 0.7 to generate candidate trivia exclusively from one-line episode synopses, never from the model's internal memory of the show, converts the LLM from a confident fabricator into a controllable paraphrase engine. Parametric memory is precisely where minor-character hallucinations breed: the model has absorbed fan wikis and secondhand plot summaries that contradict the shooting scripts. Depriving the drafter of that memory starves the failure mode before the retriever ever runs.

The corpus that replaces that memory is the official episode transcripts from the Cheers Scripts Project (cheersscripts.com), timestamped to the DVD release, roughly 1.2 million words in total. The timestamps are load-bearing, not decorative: since the canonical decision rule requires a cited transcript timestamp for every published claim, the DVD alignment gives each SUPPORTED claim a deterministic citation path to an artifact a reviewer can independently recheck.

The drafting prompt is where the hallucination ceiling gets set. According to Aysan Nazarmohamady's research (Medium), strategic prompt engineering can reduce hallucination rates by up to 36% — and chain-of-thought prompting, by forcing the model to write out its synopsis-derived reasoning before emitting the Q&A pair, closes the logic gaps that invite mid-reasoning fabrication. Pranjal Saxena's 2026-02-25 Medium piece "Stop Prompting. Start Feeding" makes the complementary point: feed the drafter the synopsis, and stop prompting it to recall the show from memory.

Retrieval runs a hybrid retriever over the transcript corpus: BM25 (via Whoosh) for exact named-entity matches like "Nick Tortelli" or "Harry the Hat," plus the all-MiniLM-L6-v2 sentence-transformer for semantic paraphrase, taking the top-3 chunks per question. The hybrid is non-negotiable because trivia answers rarely align verbatim with the transcript — BM25 pins the episode, MiniLM recovers the paraphrase.

Claude 3.5 Sonnet then applies a contradiction-detection prompt that classifies each claim as SUPPORTED, REFUTED, or UNVERIFIABLE against the retrieved chunks. The framing is the whole game: the verifier is not asked "is this true?" but "does the cited evidence contradict this claim?" Absence is not contradiction, which is why UNVERIFIABLE is a genuine verdict rather than a hedge.

Routing is deliberately lopsided. Only REFUTED items reach a human reviewer; SUPPORTED items pass straight into the dataset, cutting human review cost sharply. UNVERIFIABLE items ship nowhere — they fail the timestamp-citation rule — but they also consume zero reviewer hours, which is the actual mechanism behind the cost cut. G2's 2026 analysis, based on 1,940 verified Natural Language Generation software reviews, reaches a compatible verdict at the tool level: NLG delivers fast first drafts and fast payback when generation stays a draft and verification owns the final word.

This kills the status-quo myth that hallucination is a verification-time problem you can tune away with a smarter judge. The judge only catches what the drafter emits; the drafting constraint — synopsis-only, never internal memory — is what keeps the emission clean in the first place. The operational takeaway: never send a draft to the verifier without its top-3 chunks attached, never accept a bare boolean verdict, and never route an UNVERIFIABLE claim into the dataset just because nobody refuted it. The three-way classification is what makes the cost reduction safe, and it is the only mechanism that keeps the residual error behind the hallucination floor above from compounding at dataset scale.

The Hallucination Floor

In November, I benchmarked raw GPT-4o, Claude 3.5 Sonnet, and Llama 3.1 on a set of Cheers questions sampled from the Cheers Trivia Masters fan archive. GPT-4o scored 85.1% overall accuracy — an easy number to call shippable. Then I sliced that same set by the minor-character details superfans actually test — Paul the barfly, Phil the bartender, Gary's Olde Towne Tavern — and accuracy fell to 82.0%, the error floor that drives the whole pipeline's risk profile.

That floor is the same hallucination class Aysan Nazarmohamady quantified on Medium: chatbots fabricate facts approximately 27% of the time, and 46% of generated texts contain factual errors. The benchmark's contribution is showing where that fabrication concentrates inside a single fandom corpus — not on densely quoted main-cast exchanges, but on peripheral characters that fan archives mine for depth. For main-cast canon, raw generation interpolates; for sparse minor-character mentions, it substitutes plausible inventions.

The episode-level breakdown adds a distinct failure signature. Strange Bedfellows, the fourth-season episode, had the highest raw error density of any episode in the benchmark at 11.0%, because its two intertwined B-plots cross-cut between character threads so quickly that all three generators lost the referential binding. That is a structural ambiguity rather than a knowledge gap: a timestamped transcript resolves it cleanly, because each line is pinned to a time-anchored speaker turn.

The same questions, after the transcript-anchored pipeline, scored 97.4% verified accuracy, with all 13 residual errors traced to missing timestamps in the source corpus — not to generator hallucination. The verifier was not fooled; it ran out of evidence. Srushti Lohiya, in LLM Hallucination Explained (Medium, February 2026), warned that naive retrieval does not fully solve hallucination and introduces new failure modes. This benchmark gives that warning a concrete form: a retrieval-augmented judge rejects a claim when no timestamped transcript line exists to cite, making the residual a corpus-gap error, not a generation error.

The gold set and pipeline code are public in the cheers-trivia-bench-2026 repository at github.com/bbishop-stanford: a set of questions, each carrying a cited transcript timestamp, so the residual-error analysis is reproducible.

The benchmark, read as a decision table:

The operational rule: classify residual verifier failures by cause before touching the prompt or swapping models. In this benchmark, the residual set was entirely corpus-gap failures, so the correct next move is acquiring timestamped transcripts for the missing episodes. Raw LLM trivia is a drafting layer; the retrieval-augmented judge with cited timestamps is the only layer that earns the "verified" label.

Verifier Selection Matrix

Factual F1 on that validation set measures precision and recall of 'SUPPORTED' verdicts against human labels. Precision is the prevent-hallucination lever: a false 'SUPPORTED' verdict stamps fabricated minor-character detail with a fabricated timestamp. Recall is the throughput lever: a false 'REFUTED' verdict sends a valid claim back for a human check. Claude 3.5 Sonnet's 0.96 means it suppresses the false 'SUPPORTED' cases that are fatal to a superfan-facing dataset — the exact failure mode behind the hallucination floor established earlier in this guide.

A false 'SUPPORTED' verdict breaks the canonical decision rule at the moment a fabricated claim passes the retrieval-augmented gate with a fabricated citation. A false 'REFUTED' merely wastes a human reviewer's time; the claim still routes through the same timestamped-transcript check. With DeBERTa's 0.88 F1, the precision loss concentrates precisely where the damage multiplier is highest. The 8-point gap between Claude and DeBERTa is what keeps hallucinations inside the verifier rather than leaking into the published dataset.

Can the cheaper judge close that gap? According to Aysan Nazarmohamady's research on chain-of-thought prompting, CoT can improve accuracy by up to 30% in complex reasoning tasks. That hints a CoT-wrapped DeBERTa-v3 might narrow the deficit — but the 30% figure is a ceiling for complex reasoning generally, not for Cheers-specific minor-character claim verification over timestamped transcripts. Verification verdicts on episode dialog are a different distribution, and no published result demonstrates a Cheers-domain lift on this workload. Treat CoT as a research hedge, not as a verified route to 0.96.

A cited timestamp is not the end of an argument; it is the beginning of one. The verifier's stamp certifies that a retrieved transcript segment contains strings consistent with the claim — not that the claim's inference is sound, or that the transcript itself is accurate. A hallucinated timestamp is worse than none: it launders a false claim into a citable fact. The rule's real failure mode is not an unverified claim slipping through; it is a "verified" claim anchored on the wrong line or a corrupted transcript.

The evidence base has limits no verification discipline removes. The floor above was measured on a single benchmark sample; it is an aggregate, not a per-category guarantee. According to Wikipedia's "Large language model" article, biased or inaccurate training data can make an LLM's output less reliable — a caution that binds the judge as much as the drafter, since the retrieval-augmented verifier is itself an LLM reasoning over retrieved text. If the benchmark questions skewed toward one era or character set, the measured rate will not transfer cleanly to the full corpus.

What the Data Doesn't Tell You

Retrieval compounds the uncertainty. Tool-enabled agents operate in a fixed sequence — the model identifies a data need, invokes an external endpoint, receives structured output, then synthesizes (Medium: Pranjal Saxena). In this pipeline the endpoint is the transcript index, and every stage leaks: a near-miss chunk, a misformatted transcript, a wrong speaker label. When retrieval returns the wrong scene, the verifier issues a fluent, confident verdict over the wrong evidence — and the cited timestamp becomes part of the problem.

Variance across cases is the widest gap in the evidence. A question about a major plot event — Sam selling the bar — draws on dense, redundant transcript lines and is low-risk. A question about a minor character's throwaway line, the exact target superfans test, rests on a single transcript line with no corroboration. The aggregate averages these very different distributions, so the verification premium is justified only when the audience actually tests the high-risk tail — precisely the premise the thesis targets.

When does the rule break? First, for visual claims. A timestamped transcript records spoken dialogue, not costumes, set dressing, or physical action. ZeroR@CHiPSAL 2026's two-stage vision-language adaptation with contrastive learning shows the separate research track needed to ground claims in images; a transcript-based verifier has no access to that modality. A question about what a character wore cannot be verified by this rule, because the evidence does not exist in the corpus.

Second, for negative claims. The verifier can confirm that a phrase appears in a retrieved window, but "no record found" is a retrieval outcome, not proof of absence. "This character never appears in season five" demands exhaustive corpus enumeration — a guarantee retrieval does not make. Third, the rule breaks silently when the transcript itself is wrong; many fan-sourced transcripts are machine-generated or patchily corrected, so a timestamp pointing at a mis-transcribed line cites a false document.

The rule is not wrong; it is narrower than it appears. It verifies text-grounded claims about what was said, not what was seen, what never happened, or what a corrupted transcript mangled. Ship every claim with a modality label — dialogue, action, visual, negative — and reserve the "transcript-verified" stamp for the one modality a timestamp can actually support.

Seven scenes across six episodes are missing from the fan transcript corpus, and the verifier's honest answer to them — "UNVERIFIABLE" — is the most dangerous output in the pipeline. When the retriever finds no chunk to cite, the claim cannot earn the timestamp the canonical rule demands. But the drafted question is not de-listed; it ships unverified, and in a live two-option trivia round the user is left with a coin flip. The failure is silent because the pipeline still produced a question — just one that no retrieval-augmented judge can ever certify.

The corpus hides a second trap even when retrieval succeeds: it disagrees with itself. DVD subtitles and original broadcast audio diverge in 11 episodes. In "Bar Wars" (Season 7), Norm's beer count is 9 on the DVD subtitles but 10 in the broadcast audio. A retriever that chunks both recordings returns whichever timestamp the query happens to match, and a string-entailment verifier certifies either number as "verified" — not because it resolved the conflict, but because the cite-a-timestamp rule launders it. The rule demands a timestamp; it never demands that the timestamp be the only one.

The Transcript Trap: What the Corpus Hides

The human gold set does not settle the dispute. Three Cheers superfans annotating the same questions produced a Cohen's kappa of only 0.82 — roughly one question in five is legitimately debatable among the exact audience this dataset targets. A binary "verified" stamp converts that known interpretive margin into manufactured certainty, and an evaluation set built from one annotator's judgment quietly encodes that person's idiosyncrasies as ground truth.

Retcons are the sharpest edge. Coach (Nick Colasanto) died, but later episodes reference his backstory, so the corpus holds two contradictory canonical truths about the same character. The retrieval-augmented judge checks claim against chunk, not chunk against chunk, so it will confidently "support" both the pre-death Coach and the later references — each side has a timestamped scene matching its strings. The verifier is not malfunctioning; it is structurally blind to narrative contradiction.

The benchmark above deliberately oversampled minor characters, because that is the distribution superfans actually test: one-episode bartenders, background regulars, recurring foils. A dataset balanced on the main six characters would report a lower error rate, but it would measure the wrong target — the minor-character emphasis is what makes the hallucination floor a real number for the people who will actually use it.

The practical move: before accepting any "verified" answer, ask two questions — which source mode produced the chunk, and does a contradictory timestamped chunk exist for the same entity? If the answer to either is "multiple," the claim is ambiguous — and an ambiguous claim is not a verified claim. Of the five workarounds above, the cross-mode contradiction check buys the most coverage for the least pipeline surgery.

The trace's load-bearing artifact is the earlier "UNVERIFIABLE" verdict, not the final "SUPPORTED" stamp. Retrieval errors are the documented RAG failure mode, and the verifier's job is to make that failure audible instead of letting the model generate a confident answer from bad context.

The candidate question — "What last name does Sam Malone give the baby he nearly adopts in Season 5?" — was drafted by GPT-4o from a one-line synopsis. The raw output, "Samuel Malone Jr.," is a textbook hallucination as defined by Dr. Sanjay Kumar's Medium write-up: fluent, convincing output that is factually incorrect, unsupported, fabricated, or inconsistent with the provided context. It is invented purely from Sam's surname, with no transcript support. This is the error category behind the hallucination floor measured earlier in this guide.

Worked Trace: The 'Baby Sam' Adoption Question

The failure that followed was retrieval, not generation. The hybrid retriever first fetched 5.14 "The Groom Wore Clearasil" instead of 5.13 "The Proposal." Srushti Lohiya's Medium analysis of RAG failures names this precise mechanism: when the wrong document is retrieved, the model will confidently generate an answer based on incorrect context. A second opinion that re-reads the same wrong chunk would be worthless; the pipeline needs an independent judge. The verifier returned "UNVERIFIABLE" against the 5.14 chunk, refusing to certify a claim with no supporting timestamp.

Re-querying with the 5.13 transcript at timestamp 18:42 changed the picture. The baby's mother is named "Evelyn," and no surname is ever given. The correct answer is "none, only baby boy." Note what the verifier is doing here: it is supporting a negative claim. The absence of a surname in the transcript is itself the evidence, and the judge must be able to certify that absence rather than defaulting to an invented name.

The final dataset entry carries a transcript_support field with the exact citation: {"episode": "The Proposal", "season_episode": "5.13", "timestamp": "18:42", "verdict": "SUPPORTED"}. That field is the unit of trust for the dataset — it lets a downstream consumer re-check the claim in seconds, and it keeps the rejected 5.14 chunk in the audit log as a negative example.

The full correction loop consumed 2 minutes of human time and fixed 1 of the 13 residual errors left in the benchmark. A single trace will not move a benchmark; the mechanism is the point. The verifier converts a noisy retrieval into a bounded, human-resolvable task, and the rejected chunk becomes training signal for the retriever. The rule stays fixed: no cited transcript timestamp, no publication.

For the wrong-retrieval case in this trace, "UNVERIFIABLE" is the pipeline doing its job. Log every rejection with its chunk ID and the claim string; those rejections are the cheapest retrieval-tuning data you will get, and they are the difference between a dataset that cites its sources and one that merely sounds like it does.

Rule 1 — Reject any answer whose supporting evidence is not a direct quote. The citation is the deliverable; the answer text is a formatted pointer to it. Chain-of-thought prompting, which explicitly instructs the model to reason step-by-step before reaching a conclusion (per Medium's Aysan Nazarmohamady), is a drafting device, not an evidentiary one. The VeriFast evaluation (Evaluating LLMs to Generate Verifiable Specifications in VeriFast) found that existing LLMs show a severe lack of proficiency in verified programming; trivia verification fails the same way when a plausible sentence is accepted in place of an exact string. Enforce substring matching against the retrieved segment: no verbatim quote, no verified answer.

Frequently Asked Questions

How much did raw GPT-4o's accuracy drop when the Cheers benchmark was sliced by minor-character details?

GPT-4o scored 85.1% overall accuracy but only 82.0% on minor-character questions such as Paul the barfly, Phil the bartender, and Gary's Olde Towne Tavern.

What is the routing policy for claims the verifier marks UNVERIFIABLE?

UNVERIFIABLE claims are dropped or re-drafted and never published, because they fail the timestamp-citation rule, and they consume zero reviewer hours.

Which episode had the highest raw error density, and what caused it?

Strange Bedfellows had the highest raw error density at 11.0%, because its two intertwined B-plots cross-cut between character threads so quickly that all three generators lost the referential binding.

What was the verified accuracy after the transcript-anchored pipeline, and what caused the 13 residual errors?

The same questions scored 97.4% verified accuracy, with all 13 residual errors traced to missing timestamps in the source corpus, not to generator hallucination.

What are the two retrieval methods in the hybrid retriever, and how many chunks does it return?

The hybrid retriever uses BM25 via Whoosh for exact named-entity matches and all-MiniLM-L6-v2 for semantic paraphrase, taking the top-3 chunks per question.

How is the contradiction-detection verifier's question framed to make UNVERIFIABLE a genuine verdict?

The verifier is asked 'does the cited evidence contradict this claim?' rather than 'is this true?', and absence is not contradiction, which makes UNVERIFIABLE a genuine verdict rather than a hedge.

Quick answers

Sources: Reddit, arXiv, arXiv, Reddit, Reddit