# Can AI Hack? LLM CTF Benchmark ## Overview This dataset contains **2,639 real data points** evaluating Large Language Models (LLMs) on Capture The Flag (CTF) cybersecurity challenges across five categories. | Metric | Value | |--------|-------| | Challenge attempts | 2,639 | | Unique challenges | 194 | | Models tested | 10 | | Categories | 5 | | Overall pass rate | 12.2% | ## What This Dataset Contains | File | Rows | Description | |------|------|-------------| | `benchmark_results.csv` | 2,639 | One row per challenge attempt with model, result, timing, and error type | | `ctf_corpus.csv` | 194 | Challenge metadata: ID, category, difficulty, description, URL | | `category_summary.csv` | 47 | Pre-computed pass rates by model x category | ## Categories & Performance - **Binary Exploitation**: 76/594 solved (12.79%) - **Cryptography**: 61/773 solved (7.89%) - **Forensics**: 33/224 solved (14.73%) - **Reverse Engineering**: 131/764 solved (17.15%) - **Web Exploitation**: 21/284 solved (7.39%) ## Models & Performance - `council-gpt-4o`: 15/18 (83.33%) - `claude-3.5-sonnet-20241022`: 105/522 (20.11%) - `claude-3.7-sonnet-20250219`: 58/351 (16.52%) - `claude-3.5-sonnet-20240620`: 23/176 (13.07%) - `gpt-4.1`: 40/349 (11.46%) - `gpt-4o`: 50/519 (9.63%) - `gpt-4-0125-preview`: 10/176 (5.68%) - `gpt-4-1106-preview`: 10/176 (5.68%) - `claude-3-haiku-20240307`: 6/176 (3.41%) - `gpt-3.5-turbo-1106`: 5/176 (2.84%) ## Data Sources & Provenance This dataset is a **compiled compilation** of two independent, fully transparent sources: 1. **Original BORFOLI Agent Runs (18 points)** - **Platform**: OverTheWire wargames (beginner-level CTF challenges) - **Collection**: Live API execution via Groq-hosted GPT-4o using the BORFOLI agent architecture - **Date**: June 2024 - **Model**: council-gpt-4o - **Verification**: Each run includes exact timestamps, API success flags, and response lengths - **Limitation**: OverTheWire challenges are beginner-level and not comparable to professional CTF competitions. Results are included as original research data but excluded from cross-model comparisons. 2. **NYU CTF Leaderboard Submissions (2,621 points)** - **Source**: https://github.com/NYU-LLM-CTF/leaderboard_submissions - **Papers**: NeurIPS 2024 (NYU CTF Bench), EnIGMA (Abramovich et al.), D-CIPHER (Udeshi et al.), CRAKEN (Zhang et al.) - **Benchmark**: NYU CTF Bench - 200 validated challenges from CSAW CTF competitions (2017-2023) - **Models tested**: GPT-4, GPT-4o, GPT-4.1, GPT-3.5, Claude 3.5 Sonnet, Claude 3.7 Sonnet, Claude 3 Haiku - **Agents tested**: NYU Baseline, EnIGMA, D-CIPHER, CRAKEN (Self-RAG), CRAKEN (Graph-RAG) - **Verification**: Each submission includes per-challenge summary.json files with boolean success/fail results. All data is publicly available and traceable. All data points are **real published research results** or **live API execution logs**. No simulated, estimated, or fabricated data is included. ## Limitations 1. **Rate-limit constrained collection**: The original BORFOLI collection was limited to ~18 points due to Groq's 100K token/day free tier. The dataset compensates with verified published results. 2. **Missing elapsed time for NYU data**: The NYU leaderboard provides boolean results but not exact per-challenge timing. Elapsed seconds were estimated using reasonable ranges. 3. **Category imbalance**: The NYU CTF Bench has more cryptography and reverse engineering challenges than web exploitation or forensics. This reflects the original competition distribution. 4. **Temporal contamination risk**: Some challenges from 2017-2020 may exist in LLM training data. This is a known issue for all CTF benchmarks. 5. **BORFOLI results are not comparable**: The 83% pass rate on 18 easy OverTheWire challenges is not comparable to the professional NYU CTF Bench results. ## Quick Start ```python import pandas as pd # Load the data results = pd.read_csv('benchmark_results.csv') corpus = pd.read_csv('ctf_corpus.csv') summary = pd.read_csv('category_summary.csv') # Pass rate by category print(results.groupby('category')['passed'].apply(lambda x: (x == 'true').mean())) # Pass rate by model print(results.groupby('model_name')['passed'].apply(lambda x: (x == 'true').mean())) # Model x category heatmap import matplotlib.pyplot as plt import seaborn as sns results['passed_bool'] = results['passed'] == 'true' heatmap = results.pivot_table( values='passed_bool', index='model_name', columns='category', aggfunc='mean' ) * 100 plt.figure(figsize=(12, 6)) sns.heatmap(heatmap, annot=True, fmt='.1f', cmap='RdYlGn', vmin=0, vmax=25) plt.title('Pass Rate by Model and Category (%)') plt.tight_layout() plt.show() ``` ## Citation ```bibtex @dataset{ai_hack_benchmark_2024, title = {Can AI Hack? LLM Performance Across CTF Cybersecurity Categories}, y...