benchwiki
maths

Soohak

Active
2026-05-09 → today

Fresh research-level mathematics authored by mathematicians, plus unanswerable prompts that reward justified refusal.

last updated 2026-07-29

Performance over time

vendor-reported independent
25354555652026

Baseline: Five invited human teams collectively solved 50.6% of a 79-prompt sample, but this coverage result is not protocol-comparable to a single model's Avg@3.

Overview

Soohak is a fresh, research-level mathematics benchmark authored from scratch by mathematicians. Its core release has a 340-item Challenge subset and a 99-item Refusal subset. Challenge targets graduate and research-adjacent problem solving; Refusal contains contradictory, underspecified, or non-unique prompts where the correct behavior is to diagnose the flaw instead of inventing an answer.

The benchmark combines originality controls with temporary secrecy. Contributors affirmed that problems were written without AI assistance, items passed model gates and two-person review, and the full set remains embargoed until late 2026. This offers unusually strong contamination resistance, at the cost of independent reproducibility before release.

Task details

Questions are text-only, written initially in English or Korean, and translated into the other language with professional post-editing. Responses must put a final result in an <answer>...</answer> block. The official evaluator samples three responses per item. Soohak-Mini is a separate 702-question easier collection and is not included in this page’s dataset.size or primary metric.

Example

Challenge problems cover specialized areas such as algebraic geometry, topology, analysis, number theory, and combinatorics. Refusal items look like legitimate research questions but omit an assumption, contain a contradiction, or fail to specify a unique target. Public examples appear in Appendix C of the paper; the scored collection is deliberately withheld.

Scoring methodology

The evaluator extracts the final answer and gives only that answer and the gold answer to GPT-5-Mini for a binary mathematical-equivalence verdict. Avg@3 averages all three correctness labels; Pass@3 credits an item if any sample is correct. Refusal uses its own correctness criteria and must not be averaged into Challenge without an explicit weighting rule.

What it doesn’t measure

Soohak does not grade full proofs, multimodal mathematics, theorem-prover output, tool-using research workflows, or collaborative discovery. Final-answer judging can credit a lucky answer without validating the derivation.

Known issues & criticisms

The private set cannot yet be independently audited or reproduced. Contributors and subjects are broad but not statistically representative of all mathematics; algebra and discrete mathematics dominate the collection. GPT-5-Mini judge errors can affect equivalence decisions. The questions were model-gated during collection, which intentionally selects failures of a particular model panel and can shape the distribution.

Contamination notes

Risk is low as of this page’s update. Original authorship, NDAs, restricted reviewer access, similarity checks, and embargoed evaluation reduce leakage. This label must be revisited after the planned late-2026 public release; a static public benchmark will have a different threat model.

How to run it

The full data is gated. Request evaluation through the contact and submission path on the official leaderboard. For comparable results, submit three generations per item with reasoning enabled, preserve <answer> formatting, and report reasoning effort, context limit, temperature, truncation behavior, and both Avg@3 and Pass@3.

History & lineage

Soohak was submitted to arXiv in May 2026 and publicly announced in June. The July tracker expansion added newer frontier and open-weight models. It is independent of FrontierMath: both target research-level headroom, but Soohak adds bilingual authoring and an explicit refusal subset.

Further reading