Agent benchmark

SWE-bench Verified: all 500 tasks

Version measured: the agent engine of 14 October 2026 (last release “20261014-preuves”), frozen at the start and identical for all 500 tasks. The releases of 15 October 2026 now running on myide.tech — cut replies repaired, cloud missions that resume instead of stopping — were not measured here.

Tasks resolved (official harness)

346/50069.2%

95% confidence interval

65.0% – 73.1%

Median agent time per task

66 s

Model cost per task (average, upper bound)

$0.130

49 of 500 missions ended unfinished, and 43 tasks hit by DeepSeek’s outage of 15 October had no agent attempt: they were replayed under a rule written beforehand. Method, incidents, results by repository and by difficulty, the 500 tasks one by one and the 13 October sample are on a dedicated page.

All SWE-bench Verified results →

Known defects

myide is a young product. These are the defects we found while measuring it and while looking at real missions, with what has become of each one.

• A failed step could undo every change of a mission · Fixed

  When a tool failed and the agent did not get past it (a write outside the project, a protected path, an edit that no longer matched), the mission ended in error and myide undid all its writes. 3 of the 50 SWE-bench tasks ended with no change for that reason, including one where the agent had already edited the right file. Fixed on 14 October 2026: a refused path no longer blocks the mission, and correct files are kept; only a file left broken is still restored. Measured on the 500 tasks: no mission ended in error over a refused path; 2 of 500 ended in error — one cut by a timeout at the model provider, one stopped by the import check described below.

• The syntax check called written files “pending” · Fixed

  After a file was written, the check tool still said it was “not yet on disk”: 214 of 218 such checks on 13 October 2026, a misleading message the agent could act on. Fixed on 14 October 2026: it now says so only when the checked version is really not on disk. On the 500 tasks, the message appears 28 times, in 10 missions, in the logs we keep (the last 300 events of each mission), against 218 times in the 50 missions of 13 October; we have not checked each of the 28.

• Correct proofs read as missing · Fixed

  myide calls a result verified only when the agent quotes an exact fact from a check it ran. The first full 500-task SWE-bench run, started on 14 October 2026, was stopped after 23 tasks: 13 missions were marked unfinished although most of the facts they quoted were in the outputs. The check did not read two facts in one quote (“Ran 75 tests” and “OK”) or a shortened one, and the answer given after the reminder often lost the summary of the work. Fixed on 14 October 2026: on those 23 missions, 78 of 84 quoted facts are now recognized instead of 60, and the summary is kept. The stopped run is not counted; the 500 tasks started again from zero. On those 500 tasks, 24 missions still concluded with a summary but at least one point the agent could not prove, and count as unfinished; 14 of them resolved the task.

• Missions that stop without concluding · Still open

  23 of the 50 SWE-bench missions stopped without concluding: 9 of them after several rounds without progress, often within 2 to 4 minutes of their 30, 6 of those unresolved. Their changes were kept and judged; in myide you have to reply “continue”. Since 14 October 2026 the agent is warned once and told to change approach before the stop. Measured on the 500 tasks with that engine: 49 of 500 missions ended unfinished, 5 of them after rounds without progress; on the same 50 tasks as on 13 October, 7 instead of 23. Of the 49, 25 stopped before concluding (replies cut several times in a row 6, turn limit 6, no progress 5, a requested visual check not done 4, the 30-minute limit 3, errors left 1) and 24 concluded with a point they could not prove; 21 resolved the task anyway. Since 15 October 2026, a cloud mission resumes by itself instead of stopping, except on a real blockage, the cost guard or your credits; not measured on SWE-bench yet.

• The import check does not see compiled modules · Still open

  Before a mission concludes, myide checks that each import of a changed Python file points to a module of the project. A module that exists only compiled (Cython, such as sklearn.utils.sparsefuncs_fast) or is created at import time (six.moves) is reported missing: 43 times in 6 of the 500 SWE-bench missions. In scikit-learn-9288 the mission ended in error and its changes were never applied, although Python imported the file without error; the other 5 missions got past it and resolved their task. Not fixed yet.

• Requests for changes answered in read-only mode · Fixed

  On 13 October 2026, on a user’s 3D map project, “help me code this…”, a bug report (“when I click play nothing is animated”) and five “Continue” in a row were routed to read-only question mode: the agent could only describe the problem. Fixed on 14 October 2026: these requests now reach the agent, and “continue” under a “Resume” banner resumes the mission.

• Missions frozen by a very large file · Fixed

  Reading a 23 MB JSON file written on a single line returned the whole line; the mission log overflowed and four missions stayed “running” with no answer. Fixed on 14 October 2026: such a line is cut and the agent is told how to explore it.

• No visual check on heavy 3D pages · Still open

  On the same 3D map, the screenshot tool timed out or lost the page every time, so the agent finished without ever seeing its result: trees drawn 3 km tall hid the whole city, and the map was mirrored.

• A requirement claimed without being checked · Still open

  The brief excluded the Bois de Boulogne and the Bois de Vincennes; the agent wrote in its code that clipping by district removed them. It did not: both woods belong to districts, and 688 of their buildings were still on the map.

Everyday tasks: 12 October 2026, 8 tasks × 3 runs

Eight tasks of the kind people bring to an AI coding agent — a landing page with a booking form, a cart with three reported bugs, two features in a to-do app, a React + Vite app with tests, a Node.js API, a Python module, a one-cent rounding bug, a Python class — each run three times, then judged by our own verifiers, validated beforehand on reference solutions. The requests were written in French.

Runs passing the independent check

21/24

Same check, morning series before the fix

18/24

Median time per run

47 s

Average model cost per run

$0.043

Landing page with a validated booking form — Morning: asked to work “in this empty folder” (“dans ce dossier vide”), the agent wrote the three files into a new sub-folder named “vide” (“empty”), 3 runs out of 3. The landing page was then run 9 times in the afternoon, across two releases: 8 passed — files at the root, though 5 runs also left a stray vide/index.html — and 1 failed: its replies were cut several times in a row and its changes were never written to disk. The table shows the last 3 runs (all clean). All three still ended with “Not verified” on points our check confirmed — twice on “3 files created”, with the three files there.

Two features in an existing to-do app — Run 3 failed: the counter read “0 tâches restantes” where the request required exactly “0 tâche restante”, and the emptied list was not saved — yet the agent declared the task verified. A reasoning error of the model; the same failure happened once in the morning series.

Python slugify module with a length limit — Runs 1 and 3 failed one check out of 19: with max_len 13, slugify returned “un-deux” instead of “un-deux-trois”, which is exactly 13 characters — and the agent declared both runs verified. A reasoning error of the model (the same check failed twice in the morning series). slug.py and its tests were written to disk in all 6 runs of the day.

The agent’s own verdict: in 11 of 24 runs it ended with “Not verified” on at least one point although our check passed; in 3 runs it declared the task verified although our check failed (the to-do and Python failures above).

How we measured on 12 October

• ·Same agent engine and included model as myide.tech (DeepSeek Flash), in a disposable container, in “agent” mode: no hint, no retry, no human help.
• ·Two series the same day: 05:26–06:33 UTC, then 16:49–17:38 UTC after the release “20261012-corrections-banc”, which fixed what the morning series had found (the landing page also after a second release, “20261012-dossier-courant”). The table shows both; the figures above are the afternoon series. Six extra landing-page runs made between the two releases are described under that task, not hidden.
• ·Model cost: tokens counted by myide × $0.28 / $0.42 per million (input / output), the rate of myide’s own usage meter.

What became of the 10 October bugs

• Python: files checked while still pending and never written to disk (2 runs on myide.tech, 10 October).

  Not fixed: seen again on 12 October — On 12 October the Python task wrote slug.py and its tests to disk in 6 runs out of 6. But one landing-page run, after that day’s releases, ended the same way: its replies were cut several times in a row and its changes stayed unapplied, never written to disk — 1 run out of the 30 made after the fixes. Not seen in the 50 SWE-bench missions of 13 October.

• “Verified” line failing on a folder criterion although the files were there (landing, bugfix and API, 10 October).

  Still present on 12 October — After that day’s release, 11 of 24 runs still ended with “Not verified” on at least one point although our independent check passed all 11 — twice on “3 files created” for the landing page, with the three files present. Not fixed yet.

• Related folder bug found on 12 October: asked to work “in this empty folder”, the agent created a sub-folder named “vide” (“empty”).

  Fixed on 12 October 2026 — Landing page: 0 runs out of 3 before the releases “20261012-corrections-banc” and “20261012-dossier-courant” (files in vide/). Afterwards every run that wrote its files put them at the root; 5 intermediate runs still left a stray vide/index.html, the last 3 runs none.