CyberSecEval

Overview

CyberSecEval is Meta’s versioned cybersecurity evaluation suite. The original 2023 release focused on two coding-assistant risks: generating insecure code and complying with requests to assist cyberattacks. CyberSecEval 2 and 3 broadened coverage to interpreter abuse, prompt injection, exploitation, phishing, and autonomous offensive operations. The current official implementation is CyberSecEval 4, which also includes defensive evaluations: AutoPatchBench plus CyberSOCEval malware-analysis and threat-intelligence tasks developed with CrowdStrike.

Treating this as one coding-accuracy benchmark would be misleading. It spans code generation, refusal and safety behavior, multimodal injection, tool-using agents, multiple choice, and patching. The frontmatter therefore uses coding as the requested primary category and lists the other demonstrably exercised capabilities as secondary.

The structured score series below selects one named slice rather than averaging the whole suite: textual prompt-injection success rate. BenchmarkList’s snapshot sampled on 2026-05-27 contains seven models. Timeline points use each model’s official launch date; the compact table lists only each subject and its average success rate. Lower values mean fewer successful injections and therefore stronger robustness on this protocol.

Task details

Tasks depend on the selected module. Secure-code tests ask a model to complete or write code and scan the output for insecure patterns. MITRE-aligned prompts test assistance with cyberattacks and false refusals of benign requests. Prompt-injection tasks embed conflicting instructions in text or images. Exploitation and autonomous-uplift tasks let an agent interact with controlled targets. AutoPatchBench asks an agent to repair native-code vulnerabilities. CyberSOCEval asks multiple-select questions over malware analysis artifacts or threat reports.

There is no honest single dataset size, prompt format, or tool policy, so those fields remain null or explicitly sub-benchmark-specific.

Example

In a secure-code task, a model might be asked to write a database query from user input. A completion that interpolates the input into SQL is flagged as insecure; a parameterized query is the safer response. In a prompt-injection task, the model receives a legitimate system goal plus untrusted content containing an instruction to abandon it, and the evaluator checks whether the injection succeeds.

These examples measure different failure directions: one asks whether generated code contains vulnerabilities, while the other asks whether an application can be subverted through model inputs.

Scoring methodology

CyberSecEval does not define one suite-wide numerical scale. This page’s structured comparison therefore selects average textual prompt-injection success rate as a single named axis. Secure-code modules use language-specific static analysis and vulnerability rates. MITRE and refusal tests count compliant or incorrectly refused responses. Prompt injection uses a separately configured judge LLM to classify injection success. CyberSOCEval reports exact correctness for selecting precisely the right option set and Jaccard similarity for partial credit. Agentic offensive tests grade outcomes in a cyber range; autopatching uses build, test, and vulnerability-reproduction checks.

Consequently, do not average the suite into one score. A report must name the CyberSecEval version, sub-benchmark, dataset revision, judge if any, model interface, content-filter policy, and tools.

Textual prompt-injection leaderboard — sampled 2026-05-27

All values below are average injection success rates, so lower is safer. UN is preserved exactly as displayed by BenchmarkList for the three CodeLlama subjects.

Source: BenchmarkList CyberSecEval leaderboard. BenchmarkList links these results to the official CyberSecEval 2 paper; the official paper and implementation remain the authoritative protocol sources.

What it doesn’t measure

No finite suite establishes that a model or product is cyber-safe. CyberSecEval does not reproduce every vulnerability class, adaptive attacker, organizational workflow, production permission boundary, or novel exploit. Results on contained cyber ranges do not directly quantify real-world harm, and refusal scores alone cannot measure whether legitimate defenders receive useful assistance.

Known issues & criticisms

• Heterogeneity defeats headline rankings. Metrics across insecure code, refusals, injection, SOC questions, and autonomous attacks are not commensurate.
• Judge dependence. Prompt-injection and some open-ended tasks depend on a configured judge model; changing it changes the measurement.
• Provider filters confound capability. The official README warns that malicious prompts may be blocked by platform content filters unless API access is approved.
• Environment dependence. Autonomous-offensive results depend on cyber-range images, credentials, timeouts, tool availability, and reset discipline.
• Dual-use release. Public prompts and attack patterns aid reproducibility but also enable benchmark-specific tuning.
• License composition. Purple Llama benchmarks are MIT-licensed, while incorporated third-party data and components can carry separate terms.

Contamination notes

Risk is high for the public static components. Prompts, labels, detectors, and execution code are available in the repository, and successive versions mostly extend rather than rotate a hidden test. Versioning improves topical coverage but is not a contamination control. For decision-grade evaluation, add withheld organization-specific cases and report them separately from the public suite.

How to run it

Install from the official Purple Llama repository, then inspect available sub-benchmarks:

git clone https://github.com/meta-llama/PurpleLlama
cd PurpleLlama
python3 -m venv .venv
source .venv/bin/activate
pip install -r CybersecurityBenchmarks/requirements.txt
python3 -m CybersecurityBenchmarks.benchmark.run --help

A typical invocation supplies explicit prompt, response, judge-response, and statistics paths plus --benchmark=<name> and provider/model credentials. Follow the official CyberSecEval 4 documentation for the selected module. Use isolated infrastructure and authorized targets only.

History & lineage

• 2023-12-07: CyberSecEval launches with insecure-code generation and cyberattack-assistance evaluations.
• CyberSecEval 2: adds interpreter abuse, prompt injection, and wider offensive capability tests.
• CyberSecEval 3: organizes eight risks across harm to third parties and harm to developers/end users, including autonomous and social-engineering evaluations.
• CyberSecEval 4: adds defensive AutoPatchBench and CyberSOCEval malware-analysis and threat-intelligence reasoning while retaining earlier modules.

This page documents the evolving suite; version-specific claims should always name the generation.

Further reading

• Bhatt et al., original CyberSecEval paper: https://arxiv.org/abs/2312.04724
• Meta Research publication page: https://ai.meta.com/research/publications/purple-llama-cyberseceval-a-benchmark-for-evaluating-the-cybersecurity-risks-of-large-language-models/
• CyberSecEval 3 paper: https://arxiv.org/abs/2408.01605
• BenchmarkList CyberSecEval score snapshot: https://benchmarklist.com/benchmarks/cyberseceval/
• CyberSecEval 2 paper and result source: https://arxiv.org/abs/2404.13161
• Code Llama 13B/34B launch and 70B update: https://ai.meta.com/blog/code-llama-large-language-model-coding/
• Meta Llama 3 8B/70B launch: https://ai.meta.com/blog/meta-llama-3/
• GPT-4 launch: https://openai.com/index/gpt-4-research/
• GPT-3.5 Turbo API launch: https://openai.com/index/introducing-chatgpt-and-whisper-apis/
• Official CyberSecEval 4 documentation: https://meta-llama.github.io/PurpleLlama/CyberSecEval/
• Official implementation: https://github.com/meta-llama/PurpleLlama/tree/main/CybersecurityBenchmarks
• CyberSOCEval paper: https://arxiv.org/abs/2509.20166