Choosing a tool

Best AI fact-checking tools: five tests, and where Arbiter falls short

AI assistants state wrong answers in the same confident voice as right ones. A tool that claims to fix this must do more than make the answer sound careful. Here are five requirements, with a test for each.

1. The answer rests on sources you can see

Test: ask where each statement came from. A source you cannot open is not a source.

Arbiter: the verdict is computed from named sources that you or your assistant supplied. If a claim is not backed by a source in the record, it is not in the verdict.

2. Confidence is a number derived from the evidence

Test: ask how sure the answer is and why. Confident wording is a writing style, not a measurement.

Arbiter: each verdict carries separate scores for support, opposition and uncertainty, computed from the weighted sources.

3. "The evidence is mixed" is an allowed answer

Test: ask a question the sources genuinely disagree on. A tool that always produces a clean answer is hiding the disagreement.

Arbiter: reports contested and insufficient evidence as such.

4. The reasoning can be audited afterward

Test: a week later, can you walk from the conclusion back to who said what, and when?

Arbiter: every investigation saved as a session keeps its sources, claims and verdicts in your account. The audit trail lists each claim with who made it and how strongly the evidence supports and opposes it. It is readable as JSON through the REST API on every plan; the spreadsheet export needs a paid plan.

5. It works with the assistant you already use

Test: does checking an answer mean leaving your assistant and retyping everything?

Arbiter: connects over the Model Context Protocol (MCP), the open standard AI assistants use to call outside tools. It works with Claude, ChatGPT, Cursor, Windsurf and other assistants that support MCP. See The evidence-weighing MCP server.

Where Arbiter falls short

It does not change the model. The assistant can still misread a source when it lists the claims; the verdict is only as good as that list.
It does not know the world. We measure the sources you give us against what they say.

Start

See the five tests met on a real case: a full fraud investigation, worked end to end. Then connect Arbiter to Claude in five minutes, or open the app — free, no card required. Current plans are on the pricing page.