benchwiki
reasoning

CommonsenseQA

Saturated
2018-11-02 → today

Five-choice questions derived from ConceptNet relations, designed to require everyday knowledge beyond a supplied passage.

last updated 2026-07-25

Performance over time

vendor-reported independent
50607080902018201920202021202220232024human baseline 88.9%

Baseline: Majority vote of five crowd workers on the official test set in the original paper; random choice among five answers is 20%.

Overview

CommonsenseQA is a five-choice question-answering benchmark built to test everyday background knowledge without giving the model a passage containing the answer. Its questions cover relations such as locations, causes, purposes, desires, capabilities, and effects. The source structure comes from ConceptNet, but crowd workers turn that structure into natural-language questions.

Construction starts with a source concept and several target concepts connected to it by the same ConceptNet relation. For example, several places may all have an AtLocation relation to the same concept. Workers write a question for which one target is correct and the related targets are plausible distractors. They then add one more ConceptNet distractor and one manually written distractor, producing five choices total. This makes random elimination harder than drawing unrelated answers from a vocabulary (original paper).

The benchmark exposed a large gap at launch: the paper’s BERT-Large baseline reached 55.9% against an 88.9% human reference. That gap later closed. The KEAR system reported 89.4% on the hidden test set in 2021, using an ensemble plus retrieved ConceptNet, dictionary, and training-data knowledge (Xu et al., 2021). Saturation therefore means the fixed leaderboard stopped separating the strongest systems; it does not mean models possess broad, robust human common sense.

Task details

An item contains a question, a question_concept, five answer strings labeled A-E, and one gold label. The model must return the label or select the highest-scoring option. There is no supporting passage in the standard task.

The current official Hugging Face release contains 12,102 rows: 9,741 train, 1,221 validation, and 1,140 test (dataset card). The original paper reports 12,247 collected questions. The public release and standard harness use the smaller 12,102-row version; benchmark reports should not silently substitute the paper’s collection count for the released split sizes.

The paper studied two partition strategies:

  • Random split, the main leaderboard setup, randomly partitions questions.
  • Question-concept split, which separates question concepts across partitions and tests transfer to unseen source concepts.

The official repository states that its leaderboard is for the random split. lm-evaluation-harness also implements the random split, but evaluates the labeled validation set because the official test answers are withheld.

Example

The KEAR paper gives this genuine CommonsenseQA item and records both the base-model failure and KEAR’s corrected selection:

Question: What has a surface with many sides?

A. tetrahedron

B. object

C. geometry problem

D. lake

E. triangle

The correct answer is A, tetrahedron. The paper reports that a DeBERTa base system chose E, triangle, while KEAR chose the correct answer after retrieving ConceptNet, dictionary, and related training examples (case study). The item illustrates both the intended reasoning and a limitation: external lexical knowledge can directly supply the distinction.

Scoring methodology

Chance accuracy is 20% because every question has five choices. Scoring is exact-match accuracy: map the prediction to A-E, award one point for the gold label and zero otherwise, then average over questions. There is no partial credit and no judge model.

Official leaderboard results are computed on 1,140 hidden random-split test questions. Standard lm-evaluation-harness runs instead use 1,221 labeled validation questions and an MMLU-style prompt. For log-likelihood-capable models, the harness scores the five label choices and reports acc; generative API evaluations may depend on answer extraction. Those values should not be presented as hidden-test leaderboard results.

The 88.9% human reference is the majority vote of five workers, not an estimate that every valid question has one universally accepted answer. The KEAR 89.4% point is a 39-model ensemble with external retrieval. It demonstrates benchmark saturation under an allowed system setup, not that a single closed-book model surpassed individual humans.

What it doesn’t measure

CommonsenseQA does not measure free-form explanation, whether a model can recall an answer without choices, embodied interaction, long-horizon planning, or consistency across paraphrases. It samples English concepts and relations available in ConceptNet and questions written by a particular crowd-worker pool. Results do not establish multilingual, cross-cultural, social, or physical common sense in general.

Known issues & criticisms

  • The construction leaves annotation signals. All questions are written while workers view a source concept and related targets. This yields harder distractors than random sampling, but it also ties wording and answer distributions to the annotation process. The original paper notes that models can exploit dataset artifacts and reports above-chance behavior even for weak baselines (Talmor et al., 2019).
  • Some choices are ambiguous or duplicated in meaning. Commonsense facts are context-dependent, and several related ConceptNet targets can remain defensible. Exact match forces one label even when ordinary language permits more than one reading.
  • ConceptNet bounds the benchmark’s worldview. Relations, English concept filtering, and graph coverage shape which kinds of common sense can become questions. Knowledge missing or unevenly represented in ConceptNet is not tested.
  • Retrieval changes what the score means. KEAR’s human-parity result uses ConceptNet, Wiktionary, 17 QA datasets, repeated seeds, and a 39-model ensemble. It is a valid system result, but not comparable to a single closed-book zero-shot LM without qualification (KEAR paper).
  • The authors built a harder successor because parity was brittle. CommonsenseQA 2.0 explicitly argues that pretrained models exploit artifacts to reach human parity while still failing adversarial examples. It replaces five-choice ConceptNet-derived QA with model-in-the-loop, gamified yes/no question collection (CommonsenseQA 2.0).

Contamination notes

Risk is high for modern pretrained models. The official test labels remain hidden, which prevents the simplest answer-key lookup, but the benchmark has been static since 2018. Its training and validation questions, answer choices, and labels are public on GitHub and Hugging Face and have been reused in training and evaluation mixtures.

Hidden labels do not remove semantic contamination: test questions and choices can appear without their answer keys, ConceptNet relations can appear in training, and near-duplicate public questions can teach benchmark-specific patterns. Conversely, a model can legitimately know the underlying fact from ordinary text. These mechanisms are difficult to separate after pretraining.

Use the official hidden test when available, disclose whether the system retrieves ConceptNet or other corpora, name every fine-tuning dataset, and run exact or near-duplicate overlap checks. A public-validation harness score is useful for local regression testing but weak evidence for uncontaminated commonsense ability.

How to run it

lm-evaluation-harness uses the verified task flag commonsense_qa:

pip install "lm_eval[hf]"
lm-eval run \
  --model hf \
  --model_args pretrained=meta-llama/Llama-3.1-8B \
  --tasks commonsense_qa \
  --num_fewshot 0 \
  --batch_size auto

The current task configuration points to tau/commonsense_qa, declares train and validation splits, and has no test split. The prompt lists A-E choices and asks for Answer:. Record the harness version, chat-template settings, shot count, and whether malformed generations were counted wrong.

To inspect the public data:

from datasets import load_dataset

ds = load_dataset("tau/commonsense_qa")
print(ds["validation"][0])

History & lineage

  • 2018-11-02: CommonsenseQA is first released on arXiv with ConceptNet-derived, crowd-written five-choice questions.
  • 2019: The work appears at NAACL-HLT; the random-split hidden-test leaderboard becomes the primary comparison.
  • 2021-12-06: KEAR reports 89.4% hidden-test accuracy, slightly above the 88.9% human reference, using retrieval and a large ensemble.
  • 2022-01: Several original authors release CommonsenseQA 2.0, a distinct yes/no benchmark collected through a model-in-the-loop game to target questions that fool contemporary systems.

CommonsenseQA 2.0 is a methodological successor, not a score-compatible revision: it changes the answer format, collection process, examples, and human baseline. This page therefore records it as a lineage variant but leaves successor null rather than implying the two accuracy numbers can share one chart.

Further reading