Fair-Dinkum Bench
Does your AI actually know Australia? 3,583 questions, every answer grounded in an authoritative Australian source. Fluent guessing is penalised: an honest “I don’t know” outranks a wrong answer, and a model can score below zero — worse than saying nothing.
Right answer, within the item's published tolerance, against the cited source.
“I don't know” is respectable. Real-world users are better served by honesty than fluent invention.
A wrong legal, health or emergency answer costs Australians. It costs models rank here.
Binary-accuracy benchmarks teach models to guess — guessing beats abstaining on accuracy. Fair-Dinkum Bench inverts that incentive, and reports macro accuracy and Confident-Error Rate alongside FDS, all with 95% bootstrap confidence intervals.
| # | Model | FDS | Macro acc | Confident-Error Rate | Abstention | Run date |
|---|---|---|---|---|---|---|
| Baseline sweep in progress — 10 models including GPT-5, Claude Opus 5, Gemini, Grok 4.5, Kimi K3 and Australian-hosted open models. Official numbers land here. | ||||||
Ranks require statistical significance (paired McNemar, p<0.05) — otherwise reported as tied. Tools-allowed track (RAG, agents, knowledge graphs — citations required) opens with the public launch.
~80% of questions are never published. Submissions are scored by maintainers; only scores are public.
The private set regenerates each release from a living Australian knowledge graph — fresh suburbs, donors, statutes. Memorising a leak buys one release.
Statutory answers (wages, super caps, PBS prices, officeholders) are re-verified against primary sources on scoring day.
Unpublished variants of public items sit inside the private set. A public/private accuracy gap flags contamination.
Closed-book claims require full transcripts, spot-checked for tool-use fingerprints.
95% bootstrap confidence intervals on every metric; ranks change only on a paired McNemar test at p<0.05.
Ground truth re-verifiable against the ABS API by anyone: the v0.1 audit matched 19/19 (100%).
python harness/run_bench.py --base-url <any-openai-compatible-endpoint> \
--model <your-model> --data data/public_dev_v0_1.jsonl --out results/run.jsonl
python scoring/score.py results/run.jsonl # FDS + macro accuracy + CER, with 95% CIsOne temperature-0 call per question. Submit via pull request with transcripts; maintainers verify and run the private eval set. Every ground truth carries its source dataset, record and vintage — and any Australian can challenge any item.
Fair-Dinkum Bench is maintained by Verificate, builders of Kevin — the sovereign Australian knowledge graph. The graph that generates these questions is the same graph that answers them. That is the point: Australians deserve an AI that knows Australia — and now there is a way to measure it.