SWE-bench Verified: all 500 tasks

By repository

By difficulty

Two incidents, and the rule fixed before replaying

• ·On 15 October, DeepSeek’s API was partially unavailable from 00:11 to 05:04 UTC, according to DeepSeek’s status page and its notice to API customers. 43 tasks started between 00:27 and 03:58 UTC ended without a single model call completing — the agent never got to work on them. We paused the run at 03:58 UTC.
• ·Before replaying anything, we wrote the rule down: a task whose mission got no model call at all, or whose agent never started, is not an attempt — its folder is set aside, never deleted, and the task is played again. Every task that got at least one model call counts as it is, even if the outage cut it short: 3 counted tasks were running inside the outage window with answers from the model: sphinx-doc__sphinx-9229 (not resolved), astropy__astropy-13398 (not resolved), sympy__sympy-22714 (resolved).
• ·During the pause, our server’s nightly image cleanup deleted the agent’s runtime image, which no container was using at that moment. When we resumed at 05:03 UTC, the first 5 replayed tasks stopped before the agent started (“No such image”); under the same rule they were set aside and replayed again. From 05:18 UTC the agent ran on a rebuilt image with the same package.json and lockfile, Node 22.23.3 and Playwright 1.59.1; the engine itself is the frozen source, unchanged.
• ·In all, 48 runs were set aside for 43 distinct tasks, all kept on our server and counted nowhere. Each of these 43 tasks counts once in the 346/500, with the result of its replay: 31 resolved. They are marked in the list below. 402 counted tasks ran on the first image, 98 on the rebuilt one.
• ·A first full run, started on 14 October with the previous engine, was stopped after 23 tasks (see “Correct proofs read as missing” below). It is not counted: the 500 tasks started again from zero.

DeepSeek’s status page for this incident

How we measured

• ·The whole set: all 500 tasks of the Verified test split, no sampling. Before the first run they were sorted by ID and shuffled with a fixed seed (20261014), then played in that order, two at a time.
• ·Agent: the agent engine of 14 October 2026, in “agent” mode with the included model (DeepSeek Flash, API model deepseek-flash (DeepSeek-V4.1-Flash), reasoning off). One run per task, a 30-minute limit, the issue text as the only instruction: no hint, no retry, no human help.
• ·Same engine for all 500 tasks: its source was frozen at the start and mounted unchanged into each task’s container. Runtime image e7975b96e00c, then fcb54cbe1f84 after the incident described above.
• ·Environment and judging as for the 13 October sample: the official SWE-bench Docker image of each task, network limited to the model API, and the agent’s changes (git diff, without the notes the engine keeps for itself) judged unchanged by the official harness (swebench 5.0.2, run_evaluation).
• ·Cost: $64.87 for the 500 tasks — the tokens counted for each model call (323.4 million in, 5.7 million out), priced at DeepSeek’s published rate for that hour, every input token counted as a cache miss. The actual bill is lower.

Limits of this run

• ·One run per task, and language models vary: on the 50 tasks shared with the 13 October sample, 6 were resolved only this time and 5 only the first time. With 500 tasks the 95% confidence interval runs from 65.0% to 73.1%.
• ·In myide you can reply “continue” to a mission; here each task got a single mission of at most 30 minutes (3 reached the limit). 49 of 500 missions ended unfinished: 25 stopped before concluding and 24 concluded with a point they could not prove. Their changes were judged anyway; 21 resolved the task.
• ·9 tasks ended with no change at all in the repository (3 of 50 on 13 October); the harness refused 1 patch that did not apply and could not evaluate 1 task. All count as failures.
• ·Python only, 12 projects: SWE-bench says nothing about web front-ends, other languages or your own codebase. The everyday tasks below cover part of that.

The defects found while measuring, and what became of them

Reference point

DeepSeek reports 73.7% (reasoning off) and 79.0% (maximum reasoning) on the same 500 tasks for DeepSeek-V4-Flash — self-reported, with a harness it does not describe (model card, consulted on 13 October 2026). Not directly comparable: myide calls DeepSeek-V4.1-Flash through its own agent, reasoning off, one run per task and 30 minutes at most. We publish no other tool’s score here: none was measured under these conditions. DeepSeek-V4-Flash model card

SWE-bench Verified, 13 October 2026: a 50-task sample

SWE-bench Verified is a public set of 500 real GitHub issues from 12 open-source Python projects, each judged by the project’s own tests. We drew 50 of them at random, gave each issue once to myide’s agent, and let the official SWE-bench harness judge the result. Measured on 13 October 2026.

Tasks resolved (official harness)

32/5064%

95% confidence interval

50% – 76%

Median agent time per task

58 s

Model cost per task (average, upper bound)

$0.078

The same 50 tasks were played again in the full run, by the 14 October engine: 33 resolved instead of 32 (27 both times, 6 only the second time, 5 only the first), and 7 missions unfinished instead of 23. A difference of one task out of 50 means nothing on its own; fewer unfinished missions is what the 14 October fixes aimed at.