Building a tool to compare LLM benchmarks
Updated 23 October 2026
Contents
Key takeaways
- Model Atlas compares published language-model benchmarks using capability weights that readers can adjust for work such as coding, writing or tool use.
- The default score combines relative rankings within benchmark groups, preserving result order while discarding how large the performance gaps are.
- Lower coverage requirements can favor models whose published results cover their strengths, so missing evidence affects how readers should interpret rankings.
- Release-page results remain separate when test methods do not support comparison, rather than automatically filling gaps in a model’s combined score.
The model I would choose for coding may differ from the one I would choose for writing. I built Model Atlas with Codex to compare published LLM benchmarks using weights for the work I have in mind. It started as a Codex artifact. I couldn’t find another tool that let me make this comparison.
LLM benchmark coverage in October 2026
The full Model Atlas index includes the benchmark catalog, published scores, inference settings and coverage controls. The figures below come from the same compiled dataset as the interactive tool and update with each build.
| Model Atlas measure | 23 October 2026 |
|---|---|
| Benchmark families | 178 |
| Capabilities | 12 |
| Scored evaluation profiles | 71 |
| Source-linked benchmark observations | 1,682 |
| Model and agent configurations with scored evidence | 572 |
| Separately reported source results | 4,202 |
In the 23 October 2026 Model Atlas snapshot, Artificial Analysis’s HLE profile contains 53 configurations. Its IFBench profile contains 16 configurations in the same snapshot. A model can have a published result for one test and none for another.
The October refresh adds Kimi K3.1 Pro, DeepSeek V4.1 Flash 1017, MiniMax M3.1 Flash, Ling 3.1 Flash 1019 and NVIDIA’s Bolt E4B configurations. It also adds AA’s CritPt v2 evaluation, which lets models run Python while solving physics problems. HealthBench and HealthBench Professional each have their own comparison because their question sets and judges differ.
Compare LLMs with capability weights
I set the capability weights to match the work I have in mind. The scored profiles draw on Artificial Analysis, Arena Creative Writing, MathArena, JobBench and MCP-Atlas. For example, JobBench tests professional work. MCP-Atlas tests agents using tools through the Model Context Protocol. In the benchmark catalog, Compare selects one evaluation profile and requires complete coverage. This shows the models tested together under that profile. Use in index adds the profile to a combined score. New profiles start unselected. The default comparison uses six established profiles, with model coverage refreshed as new results become available.
How the benchmark score works
By default, each profile converts published scores to percentile ranks within its fixed group of tested configurations. Tied results share the middle rank. Percentiles retain the order of results but discard the size of the gaps between them. Filtering to open weights does not recalculate those ranks against a smaller group.
The index averages the available profile scores within each capability, with equal weight for each profile. My capability weights then determine the combined score. Missing results reduce coverage, which accounts for the selected profiles and their capability weights. At a lower coverage threshold, a model can rank using only part of the selected evidence. That can favour models whose reported results cover their strengths.
Keeping model release results in context
The catalog reviews benchmarks from release pages and model cards from Google, Meta, OpenAI, Anthropic, DeepSeek, NVIDIA and other publishers. It records test methods and limits, including benchmarks that cannot yet support a comparison. A benchmark’s inclusion in a release table alone does not establish that its scores can be combined with another source’s results. For example, Meta’s 2026 Muse Spark Open release reports 61.8% on JobBench and 89.2% on MCP-Atlas. Neither benchmark-owner leaderboard in the October snapshot lists Spark Open. Its JobBench result therefore remains a separate release record. Its MCP-Atlas result enters a separate profile for Meta’s six-model comparison, where Spark Open ranks first. That result does not fill a gap in its Artificial Analysis configuration or determine its rank across other capabilities.