benchwiki
coding

CodeXGLUE

Deprecated
2021-02-09 → today

A broad code-intelligence suite spanning understanding, retrieval, completion, translation, repair, generation, and summarization.

last updated 2026-08-03

Performance over time

vendor-reported independent
515252021

Baseline: No single human baseline exists across the ten heterogeneous tasks; several component datasets use human-authored code or labels rather than a common human score.

Overview

CodeXGLUE is a GLUE-style collection for code intelligence. Microsoft assembled 10 tasks across 14 datasets covering code understanding and generation: clone and defect detection, cloze prediction, completion, repair, translation, code search, text-to-code generation, summarization, and documentation translation. Its lasting value is breadth and standardized baselines, not one headline number.

The suite is marked deprecated for frontier comparison. The datasets and evaluators remain useful for task-level research, but the official challenge was designed around fine-tuned encoder, decoder, and encoder-decoder models of 2021. A table of unrelated metrics cannot honestly be read as a modern general coding score. The chart therefore uses only the paper’s code-summarization subtask—overall smoothed BLEU across six CodeSearchNet languages—and must not be interpreted as overall CodeXGLUE performance.

Task details

Each component has its own input and output. Examples include classifying whether two functions are clones, retrieving code from a natural-language query, completing the next line, translating Java to C#, repairing a buggy function, generating Java from a description, and writing a docstring. Some components are multilingual; others use one programming language or one natural language. The official repository provides separate directories, preprocessing, baselines, and evaluators.

Example

In the code-translation component, a system receives a Java method and must emit functionally corresponding C# code. Evaluation compares the generated text with a reference using exact match and generation metrics. That is fundamentally different from the defect-detection component, which returns a binary label for a C function; the example illustrates why a single suite-wide result would be misleading.

Scoring methodology

CodeXGLUE deliberately preserves each research task’s established metric. Classification uses accuracy or F1; retrieval uses MAP or MRR; language modeling uses perplexity and accuracy; generation tasks use exact match, BLEU, CodeBLEU, or edit similarity. Official test labels were withheld and submissions were evaluated by the maintainers. For the selected chart axis, Table 13 averages smoothed BLEU across Ruby, JavaScript, Go, Python, Java, and PHP summarization sets under a shared encoder-decoder setup. Report the component dataset and metric, not merely “CodeXGLUE.”

What it doesn’t measure

The suite does not test repository-scale software engineering, interactive debugging, tool-using agents, long-running execution, or whether generated patches pass a real project’s tests. Its generation metrics often compare text to references rather than executing code, and its components do not combine into a validated measure of overall developer productivity.

Known issues & criticisms

Metric heterogeneity is the central limitation: a point of BLEU is not commensurate with a point of F1 or MRR. Several tasks inherit known weaknesses of reference-based text metrics, where semantically equivalent code can score poorly. The paper also combines newly introduced datasets with older corpora, so provenance, duplication, and exposure differ by component. The repository’s email-based official test submission process and static leaderboard are no longer a practical current-model tracker.

Contamination notes

Risk is medium, but it must be assessed per component. Withheld official test answers reduce direct label leakage for the challenge, while public source projects, prior datasets, training splits, and repository history may appear in model training corpora. CodeXGLUE is static and has no rolling clean split.

How to run it

Clone the official repository and select one task; there is no meaningful one-command suite score:

git clone https://github.com/microsoft/CodeXGLUE.git
cd CodeXGLUE
# Follow the README in the chosen task directory, for example Code-Code/Defect-detection.

Pin the repository commit and report the component, dataset, split, preprocessing, baseline implementation, and exact metric. Official hidden-test evaluation instructions are documented in the root README.

History & lineage

CodeXGLUE was announced with its paper on 2021-02-09 and published at ESEC/FSE 2021. It followed the benchmark-suite pattern of GLUE while adapting it to program understanding and generation. Rather than having one direct successor, its tasks later split into specialized executable-code, multilingual, repository-level, and agentic benchmarks.

Further reading