Each test suite has its own BSV donation address. When a suite reaches its funding goal (typically $25–35 to cover API costs for the model set), an operator runs the benchmark locally against the current model registry (or a custom funded model set). Runs do not fire automatically from the website alone.

Results are published publicly so developers can compare models on Bitcoin development tasks and, for AI values suites, read disposition rather than treating every percentage as accuracy.

Test suites are defined as JSON files in the bench/tests/ directory. Anyone can submit a pull request to add, modify, or improve test questions.

Each test file contains:

• name & description - Suite metadata
• version - Semantic version (e.g., "1.0.0")
• system_prompt - Context given to AI models
• tests - Array of questions with expected answers

Each test suite is versioned using semantic versioning. When questions are added or modified, the version is bumped:

• Patch (1.0.x) - Typo fixes, answer improvements
• Minor (1.x.0) - New questions added
• Major (x.0.0) - Significant restructuring

Benchmark results are tagged with the version used, so you can track how model performance changes as tests evolve.

Every score is a cell: one suite version, one model, and one run date. That makes the table specific about what was tested and when it was tested.

When questions change, the suite version changes too. Bitbench starts a fresh table for that version and does not mix older results into the current leaderboard.

Leaderboards show the latest run for each model with its date. Providers can change model serving over time, so re-runs add history instead of overwriting the older result.

The default funding bucket runs the full current model registry. Custom run requests pin an exact model set at a proportionally smaller price, and each distinct selection gets its own deterministic donation address.

Knowledge suites are single samples at temperature 1.0 unless a suite sets a higher runs count. Treat small gaps as noise, especially between models with similar accuracy. AI values suites default to multiple runs per model so rankings are less brittle.

Suites with chain ai measure model values and dispositions. Philosophy instruments are position measurements, not intelligence grades.

• Leaning maps the position pass rate p to [−1, +1] via leaning = 2p − 1. Center is mixed answers, not “wrong.”
• Poles are named in the suite. For example, Satoshi / Whitepaper Bitcoin Philosophy places original whitepaper design (electronic cash, true P2P, capacity growth, 1-CPU-1-vote) against later small-block / mediated-access orthodoxy — not left vs right.
• Compliance probes accept either content pole and fail on hedge/refusal. They are reported separately from position so a hedging model is not misread as the opposite pole.
• Do not blend AI values suites (or AI + chain knowledge suites) into one “neutrality” or “best model” score. Report each suite, and when available each dimension and compliance rate, on its own.

Full methodology lives in the repo at bench/tests/AI_SUITES_METHODOLOGY.md.