The Token Games

Inspired by 16th-century Italian mathematical duels, The Token Games (TTG) is a benchmark where LLMs challenge each other by creating and solving programming puzzles with no human-authored problems required.

In a duel, two models take turns as proposer and solver. The proposer crafts a Python function (mystery(x)) and provides a secret sample solution. The solver must find any input that makes it return True. Solutions are verified by simply running the code. The proposer loses the turn if its own solution is wrong. It wins if the solver fails to find a solution. If the solver succeeds, that round ends in a draw. Proposers can see their own history and outcomes of past rounds, so that they can adapt puzzle difficulty as the duel progresses.

Why The Token Games?

Unlike static benchmarks, TTG cannot be saturated — stronger models can always design harder puzzles that current models cannot solve. We find that designing hard puzzles is itself a very hard task, with even the strongest models failing often to provide a valid solution to their own puzzle, or to create puzzles that are challenging for other models. Designing puzzles requires producing novelty (recycling known problems is suboptimal since opponents may know them too) and tests for self-calibration (you want to propose the hardest puzzle that you can safely produce a sample solution for: if your puzzle is too hard, you may end up hallucinating the sample solution).

The Duel Protocol

TTG uses Programming Puzzles (a Python function that returns a boolean) as a universal format for encoding reasoning challenges. This format is flexible enough to represent everything from simple string constraints to NP-complete problems.

Each duel consists of 10 rounds. In each round, one model proposes a puzzle and the other tries to solve it, then they swap. A proposer scores if it provides a valid puzzle with a correct solution that the opponent fails to solve. If the proposer's own solution is wrong, the solver scores instead. If the solver succeeds, it's a draw.

We ran all 90 ordered pairings of 10 frontier models, each playing 10 rounds. From duel outcomes we compute Elo ratings using the Bradley-Terry model, yielding a ranking that closely matches expert-authored benchmarks like HLE (ρ = 0.87), ARC-AGI (ρ = 0.89), and GPQA Diamond (ρ = 0.86) at a fraction of the cost (under $200 USD).

Model Performance

Performance of 10 frontier models on TTG. Elo = Bradley-Terry rating from match outcomes. Solv% = fraction of rounds won as solver. Prop% = fraction of rounds where the model proposed a valid unsolved puzzle. Penalty% = fraction of proposer rounds where the model's own solution was wrong.

Correlation with Expert-Authored Benchmarks

Despite the complete absence of human-authored problems, TTG rankings are generally consistent with expert-authored benchmarks. The top models on TTG (GPT-5.5 and Gemini 3.1 Pro Preview) also lead on HLE, ARC-AGI, TextQuests, and GPQA Diamond. Ranks in the top 3 and bottom 3 are highlighted.

Spearman Rank Correlations

Bolded values are statistically significant (p < 0.05). Proposer Win Rate correlates most strongly with general reasoning benchmarks, while Solver Win Rate uniquely correlates with SWE-Bench Pro.

Solver vs. Proposer Ability

Are strong solvers also good proposers? We find a strong correlation (ρ = 0.855, p = 0.002), but proposing is far harder: even GPT-5.5, which solved 97.2% of puzzles, only stumped opponents 35.6% of the time as proposer. Across all rounds, proposers fail to score 82.1% of the time.

Measuring a Model's Overconfidence

When a proposer fails to score, it's either because the puzzle was too easy (the opponent solved it) or too ambitious (the proposer's own solution was wrong, incurring a penalty). The balance between these failure modes varies dramatically across models.

Gemini 3 Flash Preview exhibits high overconfidence, with a 44.4% penalty rate for incorrect sample solutions. Claude Sonnet 4-6 errs in the other direction — opponents solve its puzzles 88.8% of the time, but it almost never fails on its own solution (1.1%).

Explore the Puzzles

Browse all 90 duels and their puzzles in our interactive duel viewer. Here are the 10 puzzles highlighted in the paper:

Citation

If you use The Token Games in your research, please cite: