BENCHMARKS
How the eval set is scored.
This page will publish the question set, what counts as a correct citation, what the failures were, and what's still unfixed. Publishing our own failure cases is the proof.
The question set
Twenty-four questions drawn from Broadland's recorded documents. Each item has a hand-verified source citation in the recorded text.
What counts as a correct citation
The cited section must match the expected source. A confident wrong citation is a fail. Refusal is acceptable versus a wrong guess; an answerable question that is refused is flagged.
Failures
The miss list — questions that cited the wrong section, refused when they shouldn't, or diverged from the recorded claim — will live here once the scored run is written up.
Still unfixed
Open gaps against the eval set will be listed here rather than smoothed over. Nothing on this page is a projection.