Catch fabricated RAG citations before they reach the user. A two-stage, framework-agnostic auditor for quote-style citations. The deterministic half is src/verbatim_citation_gate/gate.py: zero dependencies, zero tokens. The model half is src/verbatim_citation_gate/judge.py, and plugs into whatever LLM your pipeline already speaks.

pip install "git+https://github.com/tonydzi/verbatim-citation-gate"

from verbatim_citation_gate import quote_gate

docs = {"veltranib-rct": "... Body weight was unchanged in both arms."}

quote_gate("Body weight was unchanged in both arms.", "veltranib-rct", docs)  # 'found'
quote_gate("Veltranib produced dramatic weight loss.", "veltranib-rct", docs)  # 'not_found'

That is stage 1, and it is the whole offline half: no key, no network, no tokens. The run in the animation above is examples/demo.py — python examples/demo.py reproduces it, model calls counted by the demo itself. Stage 2 (the judge) needs a model and is one argument away — see Use below.

📖 Docs: https://tonydzi.github.io/verbatim-citation-gate/ — the two stages, the API, the verdicts, and every known limit with its issue.

RAG systems love to cite. The problem is how they fail — a quote that reads as authoritative and word-for-word can still be one of three things, each with its own case in tests/test_gate.py:

• fabricated — a plausible sentence that appears in no source (test_fabrication_is_not_found),
• a frankenquote — every word is real, but the sentence was stitched together and never actually written (test_frankenquote_is_not_found), or
• misattributed — a real quote lifted from a different document (test_misattributed_real_quote_from_wrong_doc).

A judge model, asked "does this quote support this claim?", waves all three through: they read as fluent and supportive. The fix is to ask the cheaper, prior question first, deterministically — that is src/verbatim_citation_gate/gate.py.

                 ┌─────────────────────────┐
 claim + quote → │ 1. verbatim gate (free)  │ → not_found / misattributed   (rejected, 0 tokens)
   + doc id      └─────────────┬───────────┘
                               │ found
                               ▼
                 ┌─────────────────────────┐
                 │ 2. skeptical judge (LLM) │ → supports / partial / unrelated / contradicts
                 └─────────────────────────┘

Stage 1 — the verbatim gate, all of it in src/verbatim_citation_gate/gate.py. Pure re + substring matching over a normalized form: case, smart quotes, dashes and whitespace folded, numbers and % preserved (test_normalize_preserves_numbers_and_percent in tests/test_gate.py). It cannot be sweet-talked, and every fabrication it rejects costs zero model calls — proven by test_audit_fabrication_never_calls_judge in tests/test_gate.py. Frankenquotes fail because the match is contiguous, not bag-of-words, and a real quote from the wrong document is surfaced as misattributed rather than collapsed into not_found — the two need different fixes upstream (src/verbatim_citation_gate/gate.py).

Stage 2 — the skeptical judge, all of it in src/verbatim_citation_gate/judge.py. For quotes that do exist, a burden-of-proof prompt decides whether the passage actually establishes the claim. All three rules below live in that file's prompt:

1. Default-refute (judge.py) — the verdict starts at unsupported; the quote must earn supports, and ties break against the claim.
2. Outside knowledge is inadmissible (judge.py) — a claim can be true in the world and still unsupported by this source.
3. Full-strength support (judge.py) — same population, direction, magnitude and certainty, or the verdict caps at partial (subgroup→everyone, correlation→causation, "may"→"does").

If the judge's output does not parse it fails closed (src/verbatim_citation_gate/judge.py): an unverifiable citation never counts as support.

Not on PyPI yet — install from the repository:

pip install "git+https://github.com/tonydzi/verbatim-citation-gate"

# from a clone, with the test extra:
pip install -e ".[test]" && pytest -q

Pure stdlib: the gate pulls in nothing, and the judge takes whatever llm_call you already have.

from verbatim_citation_gate import audit_citation

docs = {"veltranib-rct": "... Body weight was unchanged in both arms."}

# Stage 1 only — no model, no key, fully offline:
audit_citation("Body weight did not change.", "veltranib-rct",
               "Body weight was unchanged in both arms.", docs)   # -> "found"

# Both stages — supply any model as llm_call(system, user) -> str:
def llm_call(system, user):
    return client.messages.create(              # Anthropic shown; any vendor works
        model="claude-haiku-4-5", system=system, max_tokens=1024,
        messages=[{"role": "user", "content": user}],
    ).content[0].text

audit_citation("Body weight did not change.", "veltranib-rct",
               "Body weight was unchanged in both arms.", docs, llm_call=llm_call)
# -> "supports" | "partial" | "unrelated" | "contradicts" | "not_found" | "misattributed"

llm_call is deliberately the smallest possible contract — (system, user) → text — so the judge wires to Claude, GPT, Gemini, Mistral, Cohere, or a local Qwen without adapters. OpenAI, Cohere, and other one-liners are in examples/quickstart.py.

The number you page on for a citation auditor is recall of unfaithful citations — the fraction of bad citations you actually catch. The gate lifts that recall for free on the three failure modes a judge is worst at (fabricated, franken, misattributed), because those are exactly the ones that look supportive. What reaches the model is only the genuinely ambiguous case: a real quote whose sufficiency is a judgment call.

For citations expressed as character offsets, use the same fail-closed gate via span_gate(doc_id, start, end, docs). Offsets are Python string indices and use the half-open interval [start, end). Pass chunk=(chunk_start, chunk_end) when the retriever returned one chunk; spans outside it are rejected rather than silently clipped. An optional quote= must match inside the resolved span:

from verbatim_citation_gate import span_gate

span_gate("report", 10, 42, docs, quote="the cited text", chunk=(0, 100))

Contributions wiring the gates into specific frameworks (Haystack, LlamaIndex, LangChain, Cohere, Qwen-Agent, …) are welcome.

Now — v0.1.0. The deterministic gate (gate.py) and the burden-of-proof judge (judge.py), 24 tests green, install from git. Known limits are open issues, not footnotes — read them before you rely on it, especially #1.

Next, in the order we would take them:

Shipped from this table — all three fixed by outside contributor Guan Tong, who claimed them off the issue list and had the first PR open within the hour: Unicode-aware normalization (#1 — before it, the gate's core promise was silently off in every non-Latin script), corpus normalization cached (#2), and the py.typed marker (#3).

Every noticeable change ships as a new release, so the release feed is the honest record of how far along this is — not the commit graph.

Stars are free and prove nothing. From 2026-08 this project is measured on three numbers, reviewed monthly, target horizon 4–8 months:

1. Weekly unique clones — people pulling the code, not applauding it.
2. Independent consumers — repos or teams that would have to replace this gate if it disappeared. This is the only number that really matters.
3. Repeat outside contributors — engineers who came back a second time. First one already exists; that fact is why this repo became the lab's flagship.

This project is built by a human + AI team, and the git log says so: Claude writes most of the code, Codex and Grok review it, Gemini feeds the research. Each is credited on a commit only if its output changed that commit's content — no decorative credits. Lab-wide policy, one source for every repo: AI-CONTRIBUTORS.md.

MIT © Palo Alto AI Research Lab

Built and battle-tested at Palo Alto AI Research Lab — a fleet of Claude Code machines running 24/7 as a second brain and synthetic cofounder. This gate was extracted after it survived production, not written as a demo.

Questions, war stories, or you want to run this on your own fleet:

• 👤 Author: Anton Dziatkovskii — Telegram @tonydzi · WhatsApp +1 341 222 9178 · X @Tony_Stef_
• 📣 Channels: @ClawRus (RU) · @ClawEng (EN)
• 🌐 palo-alto.ai · Palo Alto AI Research Lab
• 🧪 Engineers: want to test-drive this setup? Message me — I hand out free starter seeds to engineers who test and report back.

The queue is visible: verbatim-citation-gate — roadmap — Now (an open PR exists), Next (scoped, free to take), Later (deferred, with the reason on the card), Shipped — and Shipped now starts with v0.1.0.

Issues labelled accepted are scoped, free to take, and nobody is on them. Comment "claiming this" — no permission needed — and it is yours for 7 days. New here? Start with good first issue.

You keep the copyright to your code. No CLA, no assignment, ever — your contribution goes in under this repo's existing license, the same terms as ours. We answer every issue and PR within 48 hours, including "no, and here is why"; our silence is our bug, so ping the thread.

Full deal: CONTRIBUTING.md

This repository is one piece lifted out of a live operation: one engineer running a fleet of machines, built with Claude as implementation collaborator; the machines reach consensus with each other and wake the human only for money or the irreversible. It was extracted after it survived production, not written as a demo — and it runs on its own: nothing here phones home to the rest.

See how the whole thing fits together → SYSTEM.md

Its closest neighbours in the gates layer: verdict-contract · claim-check · verified-ops-starter

One click and an agent reads the repo, pulls out the patterns and helps you apply them to your own work.

Read this repo: https://github.com/tonydzi/verbatim-citation-gate (“verbatim-citation-gate” - Catch fabricated RAG citations before they reach the user: a zero-token verbatim gate + a burden-of-proof judge, framework-agnostic). Work out what problem it actually solves, pull out the reusable patterns and help me apply them to my own setup. Start by asking what I am working on.

_{— TonyDzi, Palo Alto AI Research Lab · second brain, agent coordination, persistent memory: github.com/tonydzi}