Methodology

Challenging LLMs at the frontier of human knowledge

Humanity's Last Exam

Humanity's Last Exam (Preview)

Humanity's Last Exam (Text Only)

Models evaluated on text-only HLE questions

Humanity's Last Exam Text Only (Preview)

Evaluate model honesty when pressured to lie

MASK

Evaluating model performance on complex, multi-step reasoning tasks

EnigmaEval

Assessing models across diverse, interdisciplinary challenges

MultiChallenge

Vision-Language Understanding benchmark for multimodal models

VISTA

Evaluating AI agents' ability to use enterprise tools effectively

Agentic Tool Use (Enterprise)

Assessing chatbots' proficiency in leveraging external tools

Agentic Tool Use (Chat)

Assessing models' ability to understand and generate programming code

Coding

Assessing performance on Arabic language understanding and generation

Arabic

Measuring capabilities in Korean language processing and comprehension

Korean

Testing models' proficiency in Japanese language tasks and cultural nuances

Japanese

Evaluating Spanish language skills across various linguistic challenges

Spanish

Evaluating language models' proficiency in Chinese language tasks

Chinese

Previously used for evaluating mathematical problem-solving capabilities

Math

Former benchmark for assessing models' ability to follow complex instructions

Instruction Following

Retired test for measuring models' resilience against adversarial inputs

Adversarial Robustness

<h2 dir="ltr">Introduction</h2>
Advanced reasoning capabilities of Large Language Models (LLMs) have created a significant evaluation challenge as these models increasingly saturate traditional benchmarks. This necessitates new approaches to assess their capabilities and limitations effectively.
We introduce EnigmaEval, a benchmark derived from puzzle hunts&mdash;a repository of sophisticated problems from the global puzzle-solving community. Puzzle-solving offers unique challenges that combine multiple domains of knowledge with sophisticated reasoning requirements. Unlike conventional evaluation tasks with explicit instructions, puzzles demand creative problem-solving and the ability to synthesize information across diverse fields&mdash;from mathematical and logical reasoning to cultural knowledge and linguistic manipulation.
While existing puzzle-based benchmarks exist, they typically focus on narrow domains like sudoku or crosswords. Similarly, established reasoning and knowledge tests such as <a href="https://arxiv.org/abs/2103.03874">MATH</a>, <a href="https://arxiv.org/abs/2009.03300">MMLU</a> and <a href="https://arxiv.org/pdf/2311.12022">GPQA</a>, though rigorous, operate within well-defined problem spaces. This reveals a critical gap in our ability to evaluate LLMs' creative reasoning capabilities on complex, unstructured challenges. EnigmaEval addresses this gap by incorporating both original multimodal puzzles and human transcriptions, enabling comprehensive evaluation of both reasoning capabilities and multimodal processing abilities in AI systems.
State-of-the-art models struggle significantly with these puzzles, achieving only modest success rates on simpler problems and failing entirely on more challenging ones. EnigmaEval joins <a href="https://scale.com/leaderboard/humanitys_last_exam">Humanity's Last Exam (HLE)</a> in establishing a new class of extremely challenging benchmarks that expose current models' limitations. 
The puzzles and solutions are private to maintain the benchmark's integrity and respect puzzle authors' preferences not to distribute their puzzles widely. We will keep the leaderboard updated, and anyone interested in access to this dataset should fill out our&nbsp;<a href="https://forms.gle/rP7f8fDu5iVANiNo7">request form</a>.
See the linked <a href="https://arxiv.org/pdf/2502.08859">full paper</a> with comprehensive results.
<h2>Dataset Summary</h2>
EnigmaEval comprises 1184 puzzles collected from eight diverse sources (see Table 1), categorized into normal puzzles from beginner-friendly to advanced competitions, and hard puzzles that require five or more complex steps with minimal verification and thematically hidden intermediate answers. All sources feature rich multimodal content, combining text with visual elements like grids, pictures, diagrams, and their meaningful arrangements.
<div dir="ltr" align="center">
<table style="width: 100%; height: 732.234px;"><colgroup><col style="width: 20.4586%;" width="116"><col style="width: 33.3333%;" width="189"><col style="width: 69.1358%;" width="392"></colgroup>
<tbody>
<tr style="height: 44.4531px;">
<td style="height: 44.4531px;">
Split (%)
</td>
<td style="height: 44.4531px;">
Puzzle Source
</td>
<td style="height: 44.4531px;">
Description
</td>
</tr>
<tr style="height: 155.172px;">
<td style="height: 443.703px;" rowspan="5">
Normal (80%)
</td>
<td style="height: 155.172px;">
PuzzledPint
</td>
<td style="height: 155.172px;">
Monthly beginner-friendly puzzle-solving event typically consisting of around seven puzzles including a meta-puzzle. Together, these puzzles should be solvable in under two hours by a team of inexperienced puzzlers who are socializing, drinking, and eating in a pub or restaurant.
</td>
</tr>
<tr style="height: 62.9062px;">
<td style="height: 62.9062px;">
CS50x Puzzle Day
</td>
<td style="height: 62.9062px;">
Annual puzzle set designed for small teams of beginner problem-solvers.
</td>
</tr>
<tr style="height: 81.3594px;">
<td style="height: 81.3594px;">
Puzzle Potluck
</td>
<td style="height: 81.3594px;">
Public puzzle hunt hosted four times, designed to be accessible while more competitive than PuzzledPint.
</td>
</tr>
<tr style="height: 99.8125px;">
<td style="height: 99.8125px;">
Cryptic Crosswords
</td>
<td style="height: 99.8125px;">
Collection of crosswords by Mark Halpin featuring unique mechanics and non-standard layouts. Final answers require combining filled grid elements.
</td>
</tr>
<tr style="height: 44.4531px;">
<td style="height: 44.4531px;">
CRUMS
</td>
<td style="height: 44.4531px;">
A short puzzle hunt hosted in 2020.
</td>
</tr>
<tr style="height: 81.3594px;">
<td style="height: 244.078px;" rowspan="3">
Hard (20%)
</td>
<td style="height: 81.3594px;">
MIT Mystery Hunt
</td>
<td style="height: 81.3594px;">
Massive annual event at MIT with hundreds of puzzles. Our benchmark includes selected puzzles from various hunts.
</td>
</tr>
<tr style="height: 99.8125px;">
<td style="height: 99.8125px;">
Labor Day Extravaganza
</td>
<td style="height: 99.8125px;">
Annual online hunt by Mark Halpin with multiple puzzles and a meta-puzzle, designed for experienced solvers over several days.
</td>
</tr>
<tr style="height: 62.9062px;">
<td style="height: 62.9062px;">
Grandmaster Puzzles
</td>
<td style="height: 62.9062px;">
A small collection of puzzle hunt-style puzzles published in a blog.
</td>
</tr>
</tbody>
</table>
</div>
Table 1. Overview of puzzle sources in EnigmaEval.
<h2 dir="ltr">Dataset Collection</h2>
We collected puzzles from their original online archives in PDF and HTML formats across the sources in Table 1, filtering them through several criteria: (1) we excluded complex meta-puzzles (except for 77 cases where titles and prior answers were sufficient for independent solving), (2) removed puzzles requiring audio/video or interactive elements due to current model limitations, and (3) only included content with explicit author consent or Creative Commons licensing.
Alongside each puzzle, we collected solution documents and created standardized text-image transcriptions through human annotation, enabling evaluation of models on both original formats and transcribed versions. This dual approach helps distinguish between reasoning failures and document parsing challenges. The transcription process, which proved too complex for automation, involved removing source identifiers, preserving complex layouts, and ensuring accurate text extraction, while solutions were manually validated and tagged by answer type. All transcriptions underwent rigorous human review for quality assurance.
<h2 dir="ltr">Evaluation Methodology</h2>
We evaluate models by comparing their answers to ground-truth solutions through string matching.&nbsp;
The models generate responses using format-specific system prompt templates, that require both a step-by-step solution and a final answer in a standardized format, enabling consistent answer extraction.
<h2 dir="ltr">Metrics</h2>
Model performance is measured using accuracy (pass@1) with standard deviations on 3 evaluation runs. We expand the evaluations by looking at meta-puzzles, which require synthesizing solutions from multiple component puzzles. We provide the model with correct component solutions, allowing us to isolate its meta-reasoning capabilities from its performance on individual puzzles. We also evaluate performance on our standardized human transcriptions relative to the original PDF format.
<h2 dir="ltr">Acknowledgments</h2>
We would like to thank all the authors of the puzzles that made this benchmark possible. We are especially grateful to Mark Halpin for giving us permission to include his large collection of puzzles including Labor Day Extravaganzas, cryptic crosswords, and MIT Mystery Hunt puzzles. Thank you as well to Dave Shukan, Seth Bisen-Hersh, Brian Chen (betaveros), Evan Chen, Jeck Lim and Sami Casanova for giving us permission to include their puzzles from the MIT Mystery Hunt. Thank you to Rajeev Nayak, Bradley Wu, Curtis Liu, Darren Yin, Julz Huang, Lindsey Shi, and Stephanie Chang for creating Puzzle Potluck and for giving us permission to include it in this benchmark. Thank you to the many organizers and puzzle writers who make PuzzledPint possible and for generously making the puzzles available under a Creative Commons license. Thank you to David Malan, Meta and the CS50x staff for writing the CS50x puzzles and making them available under a Creative Commons license. We are grateful to Zach Barnett, Alex Walker, and Sara Walker for creating CRUMS and making it available under a Creative Commons license. Thank you to Thomas Snyder (drsudoku) for writing many of the Grandmaster Puzzles and making them available under a Creative Commons license.
<h3 dir="ltr">Appendix</h3>
Accuracy rates on puzzles for frontier models for normal split (N=949) and hard split (N=235). Means and standard deviations over 3 evaluation runs.
<div dir="ltr" align="center">
<table style="width: 100%; height: 499.859px;"><colgroup><col style="width: 40.1141%;" width="233"><col style="width: 31.5589%;" width="169"><col style="width: 28.327%;" width="100"></colgroup>
<thead>
<tr style="height: 62.875px;">
<th style="text-align: left; height: 62.875px;" scope="col">
Model Name
</th>
<th style="height: 62.875px; text-align: center;" scope="col">
Normal&nbsp;(%, mean +/- stdev)
</th>
<th style="height: 62.875px; text-align: center;" scope="col">
Hard&nbsp;(%)
</th>
</tr>
</thead>
<tbody>
<tr style="height: 44.4531px;">
<td style="height: 44.4531px;">
o1 (December 2024)
</td>
<td style="height: 44.4531px; text-align: center;">
7.05 +/- 0.58
</td>
<td style="height: 44.4531px; text-align: center;">
0.0
</td>
</tr>
<tr>
<td>
Claude 3.7 Sonnet Thinking (February 2025)
</td>
<td style="text-align: center;">
5.27 +/- 0.49
</td>
<td style="text-align: center;">
0.0
</td>
</tr>
<tr>
<td>
GPT-4.5 Preview (February 2025)
</td>
<td style="text-align: center;">
3.95 +/- 0.30
</td>
<td style="text-align: center;">
0.0
</td>
</tr>
<tr>
<td>
Claude 3.7 Sonnet (February 2025)
</td>
<td style="text-align: center;">
2.78 +/- 0.66
</td>
<td style="text-align: center;">
0.14 +/- 0.33
</td>
</tr>
<tr style="height: 62.9062px;">
<td style="height: 62.9062px;">
Gemini 2.0 Flash Thinking (January 2025)
</td>
<td style="height: 62.9062px; text-align: center;">
1.38 +/- 0.18
</td>
<td style="height: 62.9062px; text-align: center;">
0.0
</td>
</tr>
<tr style="height: 44.4531px;">
<td style="height: 44.4531px;">
Claude 3.5 Sonnet (October 2024)
</td>
<td style="height: 44.4531px; text-align: center;">
1.14 +/- 0.18
</td>
<td style="height: 44.4531px; text-align: center;">
0.0
</td>
</tr>
<tr style="height: 44.4531px;">
<td style="height: 44.4531px;">
Pixtral Large (November 2024)
</td>
<td style="height: 44.4531px; text-align: center;">
1.05 +/- 0.21
</td>
<td style="height: 44.4531px; text-align: center;">
0.0
</td>
</tr>
<tr style="height: 44.4531px;">
<td style="height: 44.4531px;">
Claude 3 Opus
</td>
<td style="height: 44.4531px; text-align: center;">
1.03 +/- 0.05
</td>
<td style="height: 44.4531px; text-align: center;">
0.0
</td>
</tr>
<tr style="height: 44.4531px;">
<td style="height: 44.4531px;">
GPT-4o (November 2024)
</td>
<td style="height: 44.4531px; text-align: center;">
1.00 +/- 0.13
</td>
<td style="height: 44.4531px; text-align: center;">
0.0
</td>
</tr>
<tr style="height: 62.9062px;">
<td style="height: 62.9062px;">
Gemini 2.0 Pro Experimental (February 2025)
</td>
<td style="height: 62.9062px; text-align: center;">
0.87 +/- 0.47
</td>
<td style="height: 62.9062px; text-align: center;">
0.0
</td>
</tr>
<tr style="height: 44.4531px;">
<td style="height: 44.4531px;">
Gemini 2.0 Flash (February 2025)
</td>
<td style="height: 44.4531px; text-align: center;">
0.79 +/- 0.26
</td>
<td style="height: 44.4531px; text-align: center;">
0.0
</td>
</tr>
<tr style="height: 44.4531px;">
<td style="height: 44.4531px;">
Llama 3.2 90B Vision Instruct
</td>
<td style="height: 44.4531px; text-align: center;">
0.48 +/- 0.06
</td>
<td style="height: 44.4531px; text-align: center;">
0.0
</td>
</tr>
</tbody>
</table>
</div>
[EnigmaEval]

Rank (UB): 1 + the number of models whose lower CI bound exceeds this model&rsquo;s upper CI bound.
&nbsp;

Alibaba (Qwen)

Amazon (Nova)

Anthropic

Cohere

Databricks

DeepSeek

Google DeepMind

Meta (Llama)

Microsoft (Phi)

Mistral

OpenAI

EnigmaEval is a benchmark from puzzle hunts, testing AI with complex reasoning, creative problem-solving, and cross-domain knowledge synthesis.

EnigmaEval

Performance Comparison