Motivation
Two things go wrong with citations. A citation can point to a source that does not exist. Or it can point to a real source that does not say what you claim.
This demo catches both, on a short sample article with planted errors. The two checks need different things. The reference check works against your reference list alone. The source-claim check also needs the sources themselves, kept as a small local knowledge base.
Set up
Install Claude Code and the skills first (see Getting started). Then clone the demo:
git clone https://github.com/scdenney/ai-for-research.git
cd ai-for-research/demos/reference-checkReference check reference list only
Does each citation exist, resolve, and match an entry? This needs only the manuscript and the reference list, no knowledge base.
/oss:citation-check sample-project/manuscript.md sample-project/references.bibOn this project it flags, among others:
| What | Verdict |
|---|---|
othen2022 — a real-looking article that was never written | Likely fabricated |
vance2018 — cited in the text, no entry in the list | Parity |
kowalski2017 — in the list, never cited | Parity |
hainmueller2014 — a real methods paper | Verified |
The last row is the control: one real reference that checks out, so you can see the difference between a flag and a clean pass.
Source-claim check needs a knowledge base
This is the deeper check, and unlike the reference check it has a prerequisite. Does the cited source actually support the claim it is attached to? To answer that, the tool has to read the source, so the sources have to be on disk first, as a small knowledge base of Markdown files in sources/md/ (already built for this demo). Without one, the check refuses to run rather than guess.
The research-repo skill builds that knowledge base for you, one Markdown file per source. When a source will not convert cleanly from PDF, such as a book or another hard-to-get work, write a short Markdown file that summarizes it faithfully. The checker reads that just as well.
With the knowledge base in place, run:
/oss:fact-check sample-project/manuscript.mdOn this project it flags:
| Claim | Verdict |
|---|---|
| "compulsory voting also raises political knowledge" | Contradicted |
| "higher social trust causes higher turnout" | Overclaim |
| "a modest, short-lived rise in youth turnout" | Supported |
A citation can be real, well-formed, and resolve cleanly, and still be attached to a claim its source never makes. Only reading the source catches that.
Check by hand
Start with the reports, but check their findings yourself. Open and read each source a report quotes. A "likely fabricated" label can be a real paper the tool could not reach. And whether a "causes" should become "is associated with" is a judgment only you can make.
Every planted error is listed in ANSWER-KEY.md. Then test the tool. Add a real reference and check that it verifies. Fix a claim and check that Contradicted turns to Supported. Delete a source file and check that the source-claim check refuses to run.