Atom v1.2 release report

Atom v1.2 (internal checkpoint probe6c) has served Incbury Chat and the API, as atom, since 1 October 2026. This report covers how it was made, what we measured and how far that evidence goes.

In short: v1.2 scores higher than earlier checkpoints on our internal development set. But that set guided its training, the differences amount to a handful of prompts, and none is statistically established. Public benchmarks for v1.2 have not been run yet.

What changed

Between v1.1 and v1.2 we ran a series of fine-tuning passes, producing internal checkpoints probe2 to probe6. Each pass started from v1.1 and added training examples aimed at failure types we had seen in the product. Every training answer had to pass an automatic check before we kept it: for example a word count, a verified calculation or a required format.

We also changed the serving code around the model between passes: when web search runs, how requests are routed by intent, and how malformed tool calls are cleaned up. So some of the differences between checkpoints come from serving changes rather than from the model.

probe6c is probe6 plus one correctness fix. When we exported to 16-bit weights, the small per-expert offsets in the expert router were being rounded away; the fixed export keeps them. It made no difference on the development set, where both scored 65 of 76 checked turns.

The development set

We measure each checkpoint on a fixed set of 120 prompts that resemble what people ask Incbury Chat. By category: code 15, knowledge 12, instruction following 12, writing 12, maths 10, reasoning 10, search 8, everyday tasks 8, multi-turn 6, safety 6, extraction 6, identity 5, clarifying questions 5, long-form 5.

The prompts go through the live product, including the system prompt, the tools and the serving code. 36 items have an automatic check (38 turns per run, counting multi-turn items); we review the rest by hand, and only the automatic checks are reported here. Each checkpoint was run two or three times, and the runs are pooled.

This is a development set, not a held-out test. Its failures decided which training data we generated, and its scores decided which checkpoint we shipped. An improvement on it shows that we fixed the failures we were targeting. It does not show that Atom got better on tasks it has not seen.

To gauge the evidence, we compared checkpoints item by item, counting an item as better if it passed more of its runs:

• probe6 against probe4: better on 10 items, worse on 4. Two-sided sign test, p ≈ 0.18.
• probe6c against probe6: better on 3 items, worse on 4. No measurable difference.

Neither comparison reaches conventional significance. With 36 checked items and two runs per checkpoint, the set is too small to tell a change of a few points from variation between runs.

Benchmarks

v1.2 has not been benchmarked yet. The table shows the two most recent checkpoints in this series that were, probe3 and probe4. Neither was released.

These figures come from our training-run records. The per-problem outputs are on storage we could not reach while writing this report, so we have not re-checked them.

On the same 300-problem GSM8K subset, the score fell from 32.3% to 29.3%, which is 9 fewer problems solved. We have not yet compared the answers problem by problem, so we cannot say whether this is a real regression or variation between runs. Until v1.2 is measured, treat it as an unresolved regression. HumanEval went from 14.0% to 15.9% (3 more tasks), and IFEval barely moved.

Regressions and open failures in v1.2

On the development set:

• Four items pass less often in v1.2 than in probe6: a number-sequence question, a family-relations puzzle, a request to write a limerick and then count its lines, and a three-way sentiment classification. With two runs per item, these differences may be noise.
• Some items fail on every v1.2 run: a sentence without the letter "e", the same family-relations puzzle ("Sally has 3 brothers. Each brother has 2 sisters"), the limerick request, and a simple counting problem ("I have 3 apples. I eat one and give one away. Then I buy 4 more").
• Clarifying questions: when a request is ambiguous, v1.2 now asks a question, but not always the right one. Asked to "book me a table for tonight", it asked what type of table.

Availability

v1.2 serves every Incbury Chat plan and the API, as model atom. v1.1 is retired.

Next

• Run the public benchmarks on v1.2 and compare GSM8K answers problem by problem with probe3 and probe4.
• Build a held-out set of product-style prompts that never feeds back into training, and report it alongside the development set.