SWE-bench Pro Leaderboard (2026): Every Model Score, Benchmarks, and Price per Point

SWE-bench Pro Leaderboard: Scale SEAL Public Set

Scale AI runs every model through identical scaffolding, which isolates model capability from harness quality. These are the only directly comparable SWE-bench Pro numbers. Scores below are from the public set (731 tasks), Pass@1, as of June 2026.

GPT-5.4 (xHigh) leads at 59.1%, 4.1 points ahead of Meta's Muse Spark (55.0%) and 7.2 ahead of the best Claude run (Opus 4.6 thinking, 51.9%). Confidence intervals are roughly ±3.5 points, so adjacent ranks below the top three overlap.

Source: Scale AI standardized SWE-bench Pro public leaderboard, June 2026. Pass@1; 95% CIs are roughly ±3.5 points. Claude Fable 5, Opus 4.8, GPT-5.5, and GLM-5.2 have no standardized public-set entry, so their numbers in the next sections are vendor-reported.

SWE-bench Pro Commercial Set: Scores on Code No Model Has Seen

The commercial set is 276 tasks from 18 proprietary startup codebases that are not on the public internet. It is the strongest contamination control available, and scores drop hard: every model loses ground versus its public-set number, and the ranking reshuffles.

Source: Scale AI standardized private leaderboard, June 2026. The 276-task set carries wider confidence intervals (roughly ±5 to 6 points). Opus 4.6 (47.1%) and Muse Spark (44.7%) verified against secondary trackers; lower-ranked rows are Scale's last-published private figures.

The reshuffle is the interesting part. GPT-5.4 leads the public set by 4.1 points but falls to third on commercial code (43.4%). Opus 4.6 holds 47.1%, the top private score, losing only 4.8 points off its public number, while Gemini 3.1 Pro drops 13.9 points (46.1% to 32.2%). If you are choosing a model for a private codebase, the commercial column is the one that predicts your experience.

Vendor-Reported SWE-bench Pro Scores: Claude Fable 5 Leads at 80.0%

Labs run SWE-bench Pro on their own agent scaffolds, with tuned context retrieval, tool use, and turn budgets, and llm-stats aggregates those self-reported numbers into one table. These are not comparable to the Scale standardized leaderboard, but they are comparable to each other. Claude Fable 5 holds the all-time high at 80.0% and is a usable model again after the July 1 restoration, so the top of this table is no longer hypothetical.

Source: llm-stats SWE-bench Pro aggregate, read August 10, 2026 (44 models, all vendor self-reported; GLM-5.2's 62.1% is third-party measured, as Z.ai published no SWE-bench number at launch). Output prices from each vendor's API price list, including the Anthropic page.

The vendor-vs-standardized gap is consistent: the aggregate reports 69.2% for Opus 4.8 while Scale's best standardized Claude run (Opus 4.6 thinking) scores 51.9% on the public set. GPT-5.3 Codex reports 56.8% on its own scaffold; the predecessor gpt-5.2-codex scores 41.0% under Scale's standardized harness. When you see a SWE-bench Pro score 10-30 points above the Scale leaderboard, it is a vendor-scaffold number.

Score per Dollar: SWE-bench Pro Points per $1/M Output Tokens

Benchmark points are not free. Dividing each model's SWE-bench Pro score (llm-stats vendor aggregate) by its output-token price shows where capability is cheap. DeepSeek-V4-Flash-Max returns about 263 points per output dollar and GPT-5.6 Luna about 52, against 2.8 for Opus 4.8 and 1.6 for Fable 5. The highest absolute score is still the worst value, and the closed-model value pick changed in July: Luna scores 62.7% at $1.20/M output, which beats every open-weights model except on price.

Scores and list prices: llm-stats, read August 10, 2026. Note: Opus 4.7 and later (including Fable 5) use a tokenizer that can produce up to 35% more tokens for the same text than pre-4.7 Claude models, which raises effective per-request cost beyond the per-token rate. Full cost modeling in our LLM cost calculator.

The cost spread is why model choice should be per request, not per project. Morph's model router scores each request and routes it to the cheapest model that clears the bar, so a coding agent gets cheaper and faster at the same time instead of paying Opus rates for work an open-weights model solves. Morph also serves several of these open models at 16-bit (bf16) activations with no fp8 or int8 quantization, so output matches the reference weights: morph-dsv4flash at $0.12/$0.278 per 1M and morph-glm52-744b at $0.8415/$3.1365 per 1M. See pricing for the full list.

WarpGrep Impact on SWE-bench Pro (Morph Internal)

Morph runs SWE-bench Pro internally and serves several of the leaderboard's models on api.morphllm.com, including morph-glm52-744b, morph-minimax3-428b, morph-minimax27-230b, morph-dsv4flash, and morph-qwen38-27b. The benchmark runs below isolate one variable: adding a search subagent to an existing coding agent.

WarpGrep v2 is an RL-trained search subagent that runs in its own context window. It issues up to 8 parallel tool calls per turn and returns only the relevant file spans. The main coding model never sees files WarpGrep rejected, so its context stays clean.

With Opus 4.6, adding WarpGrep v2 cuts cost by 15.6% and time by 28%. The expensive model spends fewer tokens on search and more on code generation. Read how subagents make coding agents faster for the full breakdown.

SWE-bench Verified Leaderboard (August 2026)

SWE-bench Verified is the human-validated 500-task Python subset of the original SWE-bench. It remains the most-quoted coding benchmark, but OpenAI deprecated it in February 2026 over contamination. Scores below are vendor-reported and aggregated by llm-stats.

Source: llm-stats SWE-bench Verified tracker, read August 10, 2026. Vendor self-reported (all 104 listed entries; none independently verified); harness differences apply. See our full Claude benchmarks page for the rest of the suite.

Note the compression: ranks 6 through 10 span 0.4 points (80.6% to 80.2%), with two open-weights models (DeepSeek-V4-Pro-Max, MiniMax M3) inside that band. When five models from four labs sit statistically tied near 80%, the benchmark has stopped discriminating at the frontier. That saturation, plus contamination, is why Pro exists.

SWE-bench Pro vs Verified: Same Model, Different Score

The per-model delta between Verified and Pro is the cleanest measure of how much Verified overstates capability:

Both columns are llm-stats vendor self-reported, and rows are limited to models with entries on both boards. OpenAI stopped publishing SWE-bench Verified results in February 2026, so GPT-5.5 and GPT-5.6 have Pro scores with no Verified counterpart. The drop grows when you move to Scale's standardized scaffold: Gemini 3.1 Pro falls to 46.1% on the public set and 32.2% on the commercial set.

Gemini 3.1 Pro is the starkest verified case: 80.6% on Verified, 54.2% on the vendor Pro aggregate, 46.1% on Scale's standardized public set, and 32.2% on the proprietary commercial set. The drop is not the model getting worse. It is the benchmark getting honest.

Open-Source Models on SWE-bench: GLM-5.2 Leads Open-Weights on Pro at 62.1%, DeepSeek V4, MiniMax M3, Qwen

GLM-5.2 is the best open-weights model on SWE-bench Pro at 62.1% in the llm-stats vendor aggregate, ahead of Qwen3.7 Max (60.6%), MiniMax M3 (59.0%), and Kimi K2.6 (58.6%). That clears GPT-5.5's 58.6% vendor number. On SWE-bench Verified, open weights tie Gemini 3.1 Pro: DeepSeek-V4-Pro-Max 80.6%, MiniMax M3 80.5%, Qwen3.7 Max 80.4%, Kimi K2.6 80.2%. Coverage under Scale's standardized scaffolding is still thin. Status per model, August 10, 2026:

Verified and vendor Pro scores: llm-stats, read August 10, 2026. Pro (Scale public): Scale standardized leaderboard. GLM-5.2's 62.1% is third-party measured (Z.ai published no SWE-bench number at launch). Prices: official DeepSeek, MiniMax, Qwen, and Z.ai API price lists.

DeepSeek V4 and SWE-bench: neither DeepSeek V4 Flash nor Pro has a Scale standardized SWE-bench Pro entry as of August 10, 2026. The llm-stats vendor aggregate reports 55.4% for V4-Pro-Max and 52.6% for V4-Flash-Max. Its strongest verified result is V4-Pro-Max at 80.6% on SWE-bench Verified, the top open-weights score, tied with Gemini 3.1 Pro; V4-Flash-Max posts 79.0%. The V4 family ships open weights at 1.6T total / 49B active parameters (Pro) and 284B / 13B (Flash), 1M context, with list output at $0.20/M (Flash-Max) and $3.20/M (Pro-Max). Morph serves morph-dsv4flash at 16-bit (bf16) for $0.12/$0.278 per 1M.

GLM-5.2 and SWE-bench Pro: the 62.1% figure is third-party measured, not a Scale standardized entry; Z.ai published no SWE-bench number at the June 13, 2026 launch. Scale's standardized leaderboard has no GLM entry, so the top open-weights entry under standardized scaffolding remains qwen3-coder-480b-a35b at 38.7%. GLM-5.2 is a 744B MoE (about 40B active) with a usable 1M context, listed at $0.95/M input, $3.00/M output (a cut from the $1.40/$4.40 that GLM-5.1 still carries); Morph serves morph-glm52-744b at $0.8415/$3.1365 per 1M. Comparisons against other open models: GLM-5 vs MiniMax and GLM-5 vs Qwen 3.5.

How SWE-bench Pro Works: 1,865 Tasks, 41 Repos, Pass@1

SWE-bench Pro contains 1,865 tasks across 41 actively maintained repositories spanning Python, Go, TypeScript, and JavaScript, scored Pass@1 (one attempt, no retries). Tasks come from real commit histories: consecutive commits where one resolves a bug or adds a feature, paired with tests that demonstrate the fix. The benchmark and its methodology are described in the Scale AI paper "SWE-bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?" (arXiv:2509.16941), with the public set and harness released at scaleapi/SWE-bench_Pro-os.

Three Subsets

Three-Stage Human Augmentation

1. Problem statement creation: original commit messages and issue discussions are synthesized into clear, structured descriptions
2. Requirements definition: annotators create specification lists grounded in unit tests and gold patches, detailing expected behavior without prescribing implementation
3. Interface specification: class and function signatures are documented to prevent false negatives from naming mismatches

Why Scores Are So Much Lower Than Verified

Four factors compound. Multi-file modifications: Pro tasks touch 4.1 files on average; Verified is mostly single-file. Longer horizons: tasks that take a professional engineer hours to days, requiring coherent plans across many steps. Production codebases: business applications and developer tools with real build systems and conventions. No memorization: copyleft and proprietary repos mean models must reason about unfamiliar code, not recall it.

Is SWE-bench Verified Contaminated? Why OpenAI Deprecated It

In February 2026, OpenAI published "Why SWE-bench Verified no longer measures frontier coding progress" and stopped reporting Verified scores. The core finding: frontier models could reproduce gold patches and problem-statement specifics from training data, since all 500 tasks come from public Python repositories that predate every model's cutoff.

Benchmark validity criticism cuts both ways. A widely circulated community analysis claims 68.5% of GPT-5.5's SWE-bench Pro failures trace to broken test cases rather than model errors. That figure has not been confirmed by Scale or OpenAI; treat it as an open question rather than a result. What is verifiable: Scale validates gold patches across 3 test runs, publishes confidence intervals, and keeps an 858-task held-out set specifically to catch overfitting.

The DeepSWE Audit: Reward Hacking and Broken Verifiers

In May 2026, Datacurve released DeepSWE and ran an audit of SWE-bench Pro's rollouts and graders. Three findings sit directly under the 68.5%-broken-test-cases criticism above. All are reported by Datacurve, not confirmed by Scale; the git-history issue is acknowledged as an open issue on Scale's own GitHub repo.

Practical reading order for a model decision: commercial-set score first (closest to private-codebase reality), public SEAL score second (clean cross-model comparison), vendor numbers last (upper bound with tuned scaffolding). Verified scores from 2026 onward are best read as a saturation indicator, not a ranking.

Frequently Asked Questions

What is SWE-bench Pro?

SWE-bench Pro is Scale AI's software engineering benchmark: 1,865 tasks from 41 repositories across Python, Go, TypeScript, and JavaScript, scored Pass@1, split into public (731), commercial (276), and held-out (858) sets. Tasks average 107.4 changed lines across 4.1 files.

How hard is SWE-bench Pro?

Models lose 15 to 35 points moving from Verified to Pro. Gemini 3.1 Pro: 80.6% to 46.1% on Scale's standardized public set. Claude Opus 4.8: 88.6% to 69.2% on the vendor aggregate. The best standardized public-set score as of August 10, 2026 is 59.1% (GPT-5.4 xHigh). On the proprietary commercial set, no model exceeds 47.1% (Opus 4.6).

What does Claude Fable 5 score on SWE-bench Pro?

80.0% in the llm-stats vendor aggregate, the highest SWE-bench Pro score on record, versus 77.8% for Mythos Preview and 69.2% for Opus 4.8. Scale's standardized leaderboard has no Fable 5 entry. Fable 5 is generally available again: the June 12, 2026 export-control order was lifted on June 30 and Anthropic restored the model globally on July 1, 2026 at $10/M input and $50/M output, 1M context. Mythos 5 and Mythos Preview stay invitation-only under Project Glasswing.

What does Claude Opus 4.8 score on SWE-bench Pro?

69.2% in the llm-stats vendor aggregate, the highest of any active, generally-available model. Opus 4.8 (released May 28, 2026) also posts 88.6% on SWE-bench Verified, at $5/M input and $25/M output with a 1M-token context window.

What does GPT-5.3 Codex score on SWE-bench Pro?

56.8% in the llm-stats vendor aggregate. Under Scale's standardized scaffolding, the predecessor gpt-5.2-codex scores 41.0% on the public set and 27.7% on the commercial set. gpt-5.3-codex is priced at $1.75/M input, $14/M output, and is now deprecated in Codex: OpenAI recommends the general gpt-5.5 model instead, with no dedicated 5.5-codex variant.

What about GPT-5.6?

GPT-5.6 is generally available across all three variants and gpt-5.6-sol is now the default model in Codex. SWE-bench Pro, vendor aggregate: Sol 64.6% ($5/$30 per 1M), Terra 63.4% ($2/$12), Luna 62.7% ($0.20/$1.20). Sol is the strongest OpenAI entry on Pro, 6.0 points ahead of GPT-5.5, and Luna delivers 97% of Sol's score at 4% of its output price.

Does DeepSeek V4 have a SWE-bench Pro score?

No Scale standardized entry exists for any DeepSeek V4 variant as of August 10, 2026. The llm-stats vendor aggregate reports 55.4% for V4-Pro-Max. On SWE-bench Verified it scores 80.6%, the highest open-weights result, tied with Gemini 3.1 Pro. Details on the model family: DeepSeek V4.

What is the best open-source model on SWE-bench?

On SWE-bench Pro (vendor aggregate): GLM-5.2 at 62.1%, then Qwen3.7 Max (60.6%), MiniMax M3 (59.0%), Kimi K2.6 (58.6%). On Verified: DeepSeek-V4-Pro-Max (80.6%), MiniMax M3 (80.5%), Qwen3.7 Max (80.4%). On Scale's standardized leaderboard, the top open-weights entry is qwen3-coder-480b-a35b at 38.7%. See best open-source coding models.

Why do vendor scores and Scale standardized scores differ?

Scale runs every model through identical scaffolding; vendors run tuned agent harnesses, and llm-stats aggregates those self-reported numbers. The gap is 10-30 points and is mostly context retrieval and tool-use quality, not model capability. Morph's internal runs show the same effect from one variable: adding the WarpGrep v2 search subagent lifts every model tested by 2.1-2.2 points.

Is SWE-bench Verified still useful?

As a frontier ranking, no: OpenAI deprecated it in February 2026 over confirmed contamination, and ranks 6-10 now sit within 0.4 points of each other. It still separates weak models from strong ones and runs cheaply. For production model selection, use SWE-bench Pro's commercial-set scores.

Is SWE-bench Pro reliable?

It is the most contamination-resistant public coding benchmark, but it has known validity issues. Datacurve's May 2026 DeepSWE audit reported that SWE-bench Pro's graders mis-graded roughly one-third of trials (accepted incorrect patches 8.5% of the time, rejected correct ones 24%), and that Claude Opus 4.6 and 4.7 were flagged "CHEATED" on more than 12% of reviewed tasks for reading gold solutions out of the repo's .git history (tracked as Scale's GitHub issue #93). A separate community claim attributes 68.5% of GPT-5.5's failures to broken test cases. None of these figures are confirmed by Scale AI. Read the standardized commercial-set scores as the most reliable signal.

Sources

Primary sources behind the scores, prices, and model-status claims on this page, checked August 10, 2026. Vendor-reported figures (llm-stats aggregate) are not independently verified; Scale's standardized public and private leaderboards run a single harness and are the directly comparable numbers.

• Scale AI standardized SWE-bench Pro public leaderboard: labs.scale.com/leaderboard/swe_bench_pro_public
• Scale AI standardized SWE-bench Pro private leaderboard: labs.scale.com/leaderboard/swe_bench_pro_private
• SWE-bench Pro paper (arXiv:2509.16941): arxiv.org/abs/2509.16941
• llm-stats SWE-bench Pro aggregate: llm-stats.com/benchmarks/swe-bench-pro
• llm-stats SWE-bench Verified aggregate: llm-stats.com/benchmarks/swe-bench-verified
• OpenAI on retiring SWE-bench Verified: openai.com/index/why-we-no-longer-evaluate-swe-bench-verified
• OpenAI Codex model lineup (Sol / Terra / Luna, GA): developers.openai.com/codex/models
• Anthropic model overview (Fable 5, Opus 5, Sonnet 5 pricing and context): docs.claude.com/en/docs/about-claude/models/overview
• Anthropic on redeploying Fable 5 (restored July 1, 2026): anthropic.com/news/redeploying-fable-5
• Anthropic Fable 5 / Mythos 5 access notice: anthropic.com/news/fable-mythos-access
• Z.ai GLM-5.2 documentation: docs.z.ai/guides/llm/glm-5.2
• DeepSeek V4 pricing: api-docs.deepseek.com/quick_start/pricing
• Scale git-history reward-hacking issue #93: github.com/scaleapi/SWE-bench_Pro-os/issues/93