--- title: "Building a tool to compare LLM benchmarks" description: "Build an LLM benchmark comparison with Model Atlas. Set capability weights, check evidence coverage, and compare published results for model configurations." date: 2026-10-13 author: "Philipp D. Dubach" categories: - "AI" keywords: - "LLM benchmark comparison" - "AI model comparison tool" - "LLM leaderboard" - "open-weight LLM benchmarks" - "custom LLM ranking" - "Model Atlas" - "capability weights" - "benchmark coverage" - "percentile ranks" - "model configurations" - "Artificial Analysis" - "Arena Creative Writing" - "MathArena" - "JobBench" - "MCP-Atlas" - "published benchmark scores" - "benchmark normalization" - "model release benchmarks" type: "Project" canonical_url: "https://philippdubach.com/posts/model-atlas-capability-weights/" source_url: "https://philippdubach.com/posts/model-atlas-capability-weights/index.md" --- # Building a tool to compare LLM benchmarks *Philipp D. Dubach ยท Published October 13, 2026* --- The [model I would choose for coding](/posts/jev-model-router-for-pi/) may differ from the one I would choose for writing. I built Model Atlas with Codex to compare LLM benchmarks. It started as a Codex artifact. Model Atlas lets me set weights for those capabilities and compare published results for specific model configurations. Something I couldn't find anywhere else so far. Open [Model Atlas](https://philippdubach.com/model-atlas/) to choose capability weights and compare model configurations. The interactive embed shows the top five configurations with complete evidence coverage. Scores are relative to the observed cohort; 100 is not perfect capability. The [full Model Atlas index](/model-atlas/) includes the benchmark catalog, published scores, inference settings, and coverage controls. The snapshot dated 15 October 2026 has 134 benchmark families across 12 capabilities. The index scores 37 evaluation profiles with enough comparable results. They contain 730 observations across 139 configurations. ## Compare LLMs with capability weights I set the capability weights to match the work I have in mind. The scored profiles draw on [Artificial Analysis](https://artificialanalysis.ai/methodology/intelligence-benchmarking), [Arena Creative Writing](https://arena.ai/leaderboard/chat/text/creative-writing), [MathArena](https://matharena.ai/?comp=arxiv--august), [JobBench](https://job-bench.github.io/) and [MCP-Atlas](https://labs.scale.com/leaderboard/mcp_atlas). For example, JobBench tests professional work. MCP-Atlas tests agents using tools through the Model Context Protocol. In the benchmark catalog, Compare selects one evaluation profile and requires complete coverage. This shows the models tested together under that profile. Use in index adds the profile to a combined score. The 15 profiles added in the latest review start unselected, so the wider catalog does not silently change the default comparison. ## How the benchmark score works By default, each profile converts published scores to percentile ranks within its fixed group of tested configurations. Tied results share the middle rank. Percentiles retain the order of results but discard the size of the gaps between them. Filtering to open weights does not recalculate those ranks against a smaller group. The index averages the available profile scores within each capability, with equal weight for each profile. My capability weights then determine the combined score. Missing results reduce coverage, which accounts for the selected profiles and their capability weights. At a lower coverage threshold, a model can rank using only part of the selected evidence. That can favour models whose reported results cover their strengths. ## Keeping model release results in context The catalog reviews benchmarks from 14 release pages and model cards from Google, Meta, OpenAI, Anthropic, DeepSeek and NVIDIA. It records test methods and limits, including benchmarks that cannot yet support a comparison. Another 366 release results remain separate from the combined score. A benchmark's inclusion in a release table alone does not establish that its scores can be combined with another source's results. For example, [Meta reports Muse Spark Open](https://research.meta.ai/blog/muse-spark-open) at 61.8 on JobBench and 89.2 on MCP-Atlas. The snapshot's benchmark-owner leaderboards do not list Spark Open. Its JobBench result therefore remains a separate release record. Its MCP-Atlas result enters a separate profile for Meta's six-model comparison, where Spark Open ranks first. That result does not fill a gap in its Artificial Analysis configuration or determine its rank across other capabilities. --- ## Frequently Asked Questions ### How can I compare LLM benchmarks with my own weights? Model Atlas lets you set capability weights and combine published benchmark results for specific model configurations. In the catalog, Compare selects one evaluation profile with complete coverage; Use in index adds that profile to a combined score. ### How does Model Atlas calculate its model ranking? By default, Model Atlas converts published scores to percentile ranks within each profile's fixed group of tested configurations. It averages available profile scores within each capability, then applies the capability weights to calculate the combined score. ### How do missing benchmark results affect the ranking? Missing results reduce coverage, which accounts for the selected profiles and their capability weights. A lower coverage threshold can admit models with only part of the selected evidence and favour models whose reported results cover their strengths. --- Canonical: https://philippdubach.com/posts/model-atlas-capability-weights/ This file is the canonical machine-readable variant of https://philippdubach.com/posts/model-atlas-capability-weights/. Author: Philipp D. Dubach (https://philippdubach.com/).