How to read these numbers

These are real, automated scores over a versioned 18-question golden set — not a curated highlight reel. They are regenerated in CI and published with the last-run date, so a regression here is a public, falsifiable signal. Headline: citation precision 97% and recall 74% on the goldens scored so far. Faithfulness (38%) comes from cite-check, which scores each cited claim strictly against its source paper and counts anything not clearly grounded as unsupported — a deliberately conservative bar. Some axes are still maturing: discovery and screening are v2 — discovery_recall in particular is noisy (it depends on the LLM's query draw and live-provider availability), so read it as a directional signal, not a gate. Not every golden runs in every sweep — see the last-run line above. Current focus: discovery query targeting, faithfulness, and golden coverage.

Public Eval Dashboard

Citation evaluation, in public.

18 golden SLR questions, 6 metrics, the latest commit-of-record run for each. Designed so a regression is a public signal — not a hidden one.

Last run6 Jul 2026, 11:13 UTCcommit17e0c4d6 of 18 goldens have data at this commit

Citation recall

76%

Did the agent find the papers it should have?

Citation precision

100%

Of the papers it cited, how many were the right ones?

Claim faithfulness

51%

Are its claims actually supported by the papers it cites?

Expected-claim coverage

23%

Does the draft mention the canonical findings reviewers expect?

Discovery recall (v2)

20%

For outbound runs: did the discoverer find the expected papers across providers?

Screening precision (v2)

Of the papers the screener admitted, what fraction were the expected ones?

By question

Most recent run per (question × metric).

Question	Recall	Precision	Faithfulness	Coverage
000-tdd-web-frameworks	100%	100%	69%	67%
001-llm-code-review-security	50%	100%	55%	50%
002-rag-architecture-patterns	80%	100%	63%	40%
004-chain-of-thought-effectiveness	100%	100%	54%	25%
005-rag-vs-finetuning-knowledge	75%	100%	63%	25%
007-prompt-engineering-techniques	100%	100%	60%	0%
008-llm-agent-evaluation	100%	100%	40%	0%
009-llm-copilot-developer-productivity	33%	100%	38%	0%
010-rag-architecture-evolution	50%	100%	40%	0%
017-rag-hallucination-grounding	0%	0%	27%	0%

How this works

Why the dashboard exists, how a number gets here, and what each metric is actually measuring.

Where these numbers come from

Every score here is produced by running the actual Thoth agent — the same pipeline that powers the live app (plan → discover/retrieve → screen → assess → draft → cite-check) — headlessly over the versioned golden set. Nothing on this page is hand-entered. Each run writes one row per (golden, metric) to the EvalRun table, stamped with the git commit and timestamp shown at the top of this page; the page always renders the most recent run, so the figures are pinned to exactly the code that generated them.

The agent is LLM-provider-agnostic— a run uses whichever model is configured. The free, public demo runs on a free-tier model (Mistral) for $0; some refreshes run on a frontier model (Claude) for higher fidelity. Either way the model doesn't assign the scores by fiat: recall and precision are set-overlap against the golden's expected papers, and faithfulness is the fraction of per-claim cite-check verdicts that come back supported — so the numbers measure what the agent actually produced on that run, not a model grading itself.

Lifecycle

Each golden lives in evals/golden/*.yaml — a question, a small expected corpus, and a list of claims a competent reviewer would surface.
A scheduled GitHub Action (Monday 06:00 UTC, also manually triggerable) runs the full Thoth agent loop — plan → retrieve → assess → draft → cite-check — headlessly against every golden.
Each run writes one row per (golden, metric) to the EvalRun table. This page reads the latest row for each pair — no cherry-picking, no averaging across history.

Philosophy

Public, not hidden.: If a commit makes the agent worse, the regression shows up here — not buried in CI logs only the author reads.
Vacuous-true scoring.: When a golden doesn't assert on a particular metric (no expected papers, no expected claims), that metric returns 1.0 for that golden so it doesn't drag the average down.
Advisory regression watch.: Each metric is logged with its delta from the highest historical score for that golden, but the check is advisory, not a CI gate. Empirical result: Mistral's free-tier non-determinism on 4–5-item denominators produces ±25–40% per-metric variance run-to-run even when agent code is byte-identical, so no single threshold cleanly separates real regressions from sampling noise. The dashboard is the authoritative public signal; the workflow status reflects only catastrophic failure (empty sweep, infrastructure outage).

The 6 metrics, in detail

Citation recall · expected ∩ cited / expected: Of the papers the golden says should be cited, how many did the agent actually surface? Hardest metric to game — it requires retrieval and screening to land on the right papers from the corpus, not just write plausible prose.
Citation precision · expected ∩ cited / cited: Of the papers the agent did cite, how many were on the expected list? Easy to score 100% by being maximally conservative; the trick is doing it alongside high recall.
Claim faithfulness · SUPPORTED / total claim checks: After drafting, the cite-check stage asks an LLM whether each extracted claim is actually supported by the paper excerpt it cites. This is the most direct measure of hallucination — and the metric we watch most closely.
Expected-claim coverage · keyword hits / expected claims: Literal substring match: does the draft contain each expected-claim phrase from the golden? Deliberately brittle — paraphrasing scores zero — but useful for catching outright omissions of canonical findings. Treat low scores here as a prompt for inspection, not as proof of failure.
Discovery recall · expected ∩ discovered / expected: Outbound mode only.Of the papers the golden expects, how many did the discoverer surface from the live providers (OpenAlex / arXiv / Exa) before screening? Informational — the discoverer's queries are LLM-generated and vary per run, so read it as a directional signal, not a hard gate.
Screening precision · expected ∩ screened-in / screened-in: Outbound mode only.Of the discovered papers the screener voted to include, how many were on the golden's expected list? Measures whether the include/exclude judgement keeps the corpus on-topic.