Models / Evaluations

Measured honestly, published plainly.

Every number on this page names the release or checkpoint, the dataset, the sample size and the settings used. Development-set results are labelled as such: they guide our training, so they are not held-out tests.

Public benchmarks

Public benchmarks by release.

Atom v1.2 is serving today. Its public benchmark run has not been done yet, so the latest release-level results are for Atom v1.1.

Benchmark results with the settings used
Release	Suite	Dataset	n	Mode	Decoding	Tools	Score
Atom v1.1if-sft	Instruction following	IFEval	541	Instant	greedy	No	41.8% prompt-level / 52.0% instruction-level (strict)
Atom v1.1if-sft	Maths reasoning	GSM8K test, 300-problem subset	300	Medium	greedy	No	27.0%
Atom v1.1if-sft	Code	HumanEval	164	Instant	greedy, 512-token limit	No	12.2% pass@1
Atom v1.1base	Knowledge	MMLU, 5-shot	14,042	Base model	log-likelihood	No	27.2%
Atom v1.1base	Commonsense and language	HellaSwag / ARC-Easy / ARC-Challenge / PIQA / Winogrande	16,695	Base model	log-likelihood	No	55.5 / 63.3 / 34.6 / 73.5 / 57.6
Atom v1.0	Instruction following	IFEval	541	Not recorded	Not recorded	No	31.6% prompt-level / 41.5% instruction-level (strict)
Atom v1.0	Maths reasoning	GSM8K test, full set	1,319	Not recorded	Not recorded	No	24.9%
Atom v1.0	Code	HumanEval	164	Not recorded	Not recorded	No	13.4% pass@1

Internal checkpoints

Two unreleased checkpoints from the training runs between v1.1 and v1.2, taken from training-run records. GSM8K fell from 32.3% to 29.3% between them, 9 of 300 problems. We have not established whether that change is significant.

Benchmark results with the settings used
Release	Suite	Dataset	n	Mode	Decoding	Tools	Score
Internal checkpointprobe4	Instruction following	IFEval	541	Instant	greedy	No	42.0% prompt-level (strict)
Internal checkpointprobe4	Maths reasoning	GSM8K test, 300-problem subset	300	Instant	greedy	No	29.3%
Internal checkpointprobe4	Code	HumanEval	164	Instant	greedy, 384-token limit	No	15.9% pass@1
Internal checkpointprobe3	Instruction following	IFEval	541	Instant	greedy	No	41.8% prompt-level (strict)
Internal checkpointprobe3	Maths reasoning	GSM8K test, 300-problem subset	300	Instant	greedy	No	32.3%
Internal checkpointprobe3	Code	HumanEval	164	Instant	greedy, 384-token limit	No	14.0% pass@1

Development set

Our development set.

120 prompts across 14 categories, sent to the live product. 36 of them have machine-checkable answers, and the pass rate below covers those only.

Development-set pass rate by checkpoint
Checkpoint	Release	Runs	Checked turns passed	Pass rate
probe2	Not released	3	76 / 114	66.7%
probe3	Not released	3	85 / 114	74.6%
probe3-dpo	Not released	3	84 / 114	73.7%
probe4	Not released	2	59 / 76	77.6%
probe5	Not released	2	58 / 76	76.3%
probe6	Not released	2	65 / 76	85.5%
probe6-dpo	Not released	2	59 / 76	77.6%
probe6c	Atom v1.2	2	65 / 76	85.5%

This is not a held-out test. We wrote these prompts and use failures on them to design training data, so scores here overstate how Atom does on unseen tasks.
Each run checks 38 turns, and passes are summed across runs.
With this few items, differences of a few points may not be significant. The release report gives a paired test for each comparison.

Atom v1.2 release report Preference-optimisation experiment

Agent harness

Incbury Code harness tasks.

40 coding tasks of five kinds, run through the Incbury Code agent loop and its tools against a model checkpoint directly. This tests the harness and the model together, not the Incbury Code product as you install it.

Tasks passed by kind and checkpoint
Task kind	probe4	probe5
Fix a bug	0 / 8	2 / 8
Add a feature	0 / 8	6 / 8
Start a project	0 / 8	6 / 8
Explain code	2 / 8	8 / 8
Answer without tools	4 / 8	8 / 8
Total	6 / 40 (15.0%)	30 / 40 (75.0%)

Neither checkpoint is the one serving today, and these tasks are not held out: probe5 was trained on data from the same task generators. We will publish a run on the current release once it can be done in an isolated environment with a fixed budget.

Read the harness evaluation

Suites

What we measure.

Instruction following

IFEval: 541 prompts with verifiable formatting and content constraints, checked programmatically rather than by a grader model.

Metric: strict accuracy

Maths reasoning

GSM8K: grade-school word problems that need several steps, answered in chat mode with the reasoning shown.

Metric: exact match

Code

HumanEval: 164 Python programming tasks, with the generated function executed against the reference tests.

Metric: pass@1

Knowledge

MMLU: multiple-choice questions across 57 academic and professional subjects.

Metric: accuracy

Commonsense and language

HellaSwag, ARC-Easy, ARC-Challenge, PIQA and Winogrande, scored by log-likelihood over the full test sets.

Metric: normalised accuracy

Methodology

How we run them.

Public benchmarks first

We start with widely used, openly available suites so anyone can compare Atom with other models on the same footing. Test sets are excluded from training data, and we decontaminate our fine-tuning data against them.

Graded by machine where possible

Every published number is checked programmatically: exact answers, unit tests or verifiable constraints. We do not ask a model to mark itself.

Settings on every number

Each result names the checkpoint, mode, decoding and whether tools were allowed. Benchmarks run against the checkpoint on our training hardware, not through the product, so search and the Python sandbox are not used.

Development set kept separate

Our development set guides training, so it is reported apart from public benchmarks and never called held-out. Regressions are reported as measured, with the sample size, rather than explained away.

Architecture details are on the Atom page. We do not publish training infrastructure or costs.

Run your own evaluation.

The best benchmark is your actual work. Try it free.

Try Incbury