Skip to content
Log in
All research
Engineering

Coding tasks in our agent training harness: probe4 against probe5

On 40 generated coding tasks, the pass rate rose from 15% to 75% after agent training. What this measures, and the important things it does not.

incbury-codeagentsevaluation

What this measures

We generated 40 coding tasks, 8 of each kind:

  • Bug fix: make a failing test suite pass.
  • Feature: add behaviour to an existing small project.
  • New project: create a small project from a description.
  • Explain: answer a question about existing code.
  • No tools: a question Atom should answer directly, without touching the project.

Each task has an automatic check, such as running the project's tests or confirming that no tools were used when none were needed. Atom works on each task in our agent training harness. The harness uses the same agent system prompt and tool definitions as the Incbury Code command-line app, and runs every tool call on a generated project in a temporary folder, with a limit of 30 steps.

What it does not measure

  • It is not an evaluation of the Incbury Code product. The harness runs every tool call without asking for approval. It does not exercise the desktop app, the editor extension, long sessions or context management, and the tasks are small generated projects, not real codebases.
  • It is not a held-out test. The tasks come from the same generators we use to create agent training data, with a different random seed. probe5 was trained on data from those generators, so its gain shows that it learned these task types, not that it improved on unfamiliar ones.
  • It is not the current model. probe4 and probe5 are intermediate checkpoints. Atom v1.2 has not been run on this set.

Results

Both checkpoints were run once, on the same 40 tasks.

Tasks passed by kind and checkpoint
Task kindprobe4probe5
Fix a bug0 / 82 / 8
Add a feature0 / 86 / 8
Start a project0 / 86 / 8
Explain code2 / 88 / 8
Answer without tools4 / 88 / 8
Total6 / 40 (15.0%)30 / 40 (75.0%)

probe5 passed 24 tasks that probe4 failed, and failed none that probe4 passed (exact McNemar test, p < 0.001). The change is far larger than run-to-run variation, within the limits above. Bug fixes remain the weakest kind, at 2 of 8.

The command lines were not saved with the results. The per-task records fit the harness defaults, with no task taking more than 30 steps, but we cannot confirm the other settings.

Next

  • Run Atom v1.2 on this set.
  • Build tasks from generators that never feed training, so we can report a held-out result.
  • Evaluate the shipped Incbury Code command-line app end to end, in an isolated environment with a fixed budget.