Building a tool to compare LLM benchmarks
Contents
The model I would choose for coding may differ from the one I would choose for writing. I built Model Atlas with Codex to compare LLM benchmarks. It started as a Codex artifact. Model Atlas lets me set weights for those capabilities and compare published results for specific model configurations. Something I couldn’t find anywhere else so far.
The full Model Atlas index includes the benchmark catalog, published scores, inference settings, and coverage controls. The snapshot dated 15 October 2026 has 134 benchmark families across 12 capabilities. The index scores 37 evaluation profiles with enough comparable results. They contain 730 observations across 139 configurations.
Compare LLMs with capability weights
I set the capability weights to match the work I have in mind. The scored profiles draw on Artificial Analysis, Arena Creative Writing, MathArena, JobBench and MCP-Atlas. For example, JobBench tests professional work. MCP-Atlas tests agents using tools through the Model Context Protocol. In the benchmark catalog, Compare selects one evaluation profile and requires complete coverage. This shows the models tested together under that profile. Use in index adds the profile to a combined score. The 15 profiles added in the latest review start unselected, so the wider catalog does not silently change the default comparison.
How the benchmark score works
By default, each profile converts published scores to percentile ranks within its fixed group of tested configurations. Tied results share the middle rank. Percentiles retain the order of results but discard the size of the gaps between them. Filtering to open weights does not recalculate those ranks against a smaller group.
The index averages the available profile scores within each capability, with equal weight for each profile. My capability weights then determine the combined score. Missing results reduce coverage, which accounts for the selected profiles and their capability weights. At a lower coverage threshold, a model can rank using only part of the selected evidence. That can favour models whose reported results cover their strengths.
Keeping model release results in context
The catalog reviews benchmarks from 14 release pages and model cards from Google, Meta, OpenAI, Anthropic, DeepSeek and NVIDIA. It records test methods and limits, including benchmarks that cannot yet support a comparison. Another 366 release results remain separate from the combined score. A benchmark’s inclusion in a release table alone does not establish that its scores can be combined with another source’s results. For example, Meta reports Muse Spark Open at 61.8 on JobBench and 89.2 on MCP-Atlas. The snapshot’s benchmark-owner leaderboards do not list Spark Open. Its JobBench result therefore remains a separate release record. Its MCP-Atlas result enters a separate profile for Meta’s six-model comparison, where Spark Open ranks first. That result does not fill a gap in its Artificial Analysis configuration or determine its rank across other capabilities.