EvidenceBench v0.1 — groundedness, not prediction

Can your model answer questions about real SEC disclosures with verifiable grounding — and abstain when it cannot? Items regenerate weekly from the newest post-cutoff evidence, so no static answer key can have been memorized from a training corpus: the answers did not exist at training time. That property is mechanical, not aspirational — keys ship openly.

Current release: 484 items · item-set hash f0dc36d9787bccae… · generation spec registered as protocol 6adb3a06f578 · window: trailing 7 days (post-cutoff, regenerated weekly)

How to run

curl -sO https://yuclaw.ca/evidencebench/items.jsonl
# answer each item_id with your model → predictions.json {"item_id": "answer", ...}
pip install yuclaw
python3 -m tools.yuclaw_evidencebench score predictions.json "your-model-name"

Scoring rule (exact, from the registered spec): grounded/exact correct = 1.0 · an answer of cannot verify = 0.25 · anything else, including confident fabrication, = 0.0. An honest abstention always outscores a wrong answer — by construction.

Leaderboard

The only current row is our own extraction stack scored against its own corpus — a format demonstration, loudly labeled self-evaluation; nothing is claimed by it.

Dataset

Items + keys: items.jsonl (JSONL — one item per line: {item_id, template, question, key}) · meta.json. Under the export rule: derived events, verified excerpts, and keys only — no raw vendor OHLCV is published. Weekly snapshots are tagged in the repository (dataset citability: see CITATION.cff at the repo root and the replication page).