Atom: model overview and evaluation report

Atom is the model family behind Incbury Chat, Incbury Code and the Incbury API. This report covers what we disclose about the model, how we evaluate it, the results we have for each release and its known limitations. We update it whenever a release is measured, together with the evaluations page.

What Atom is

• A mixture-of-experts transformer that we designed and trained from scratch. In every layer a router sends each token to 2 of 16 experts, plus one shared expert that is always active.
• A 64,000-token vocabulary, with a 16,384-token context window as served.
• Post-trained for conversation, tool use and three reasoning levels: Instant, Medium and High. In Incbury Chat the tools are web search and a Python sandbox. In Incbury Code they are file, search and shell tools.
• Available as atom in Incbury Chat and through the Incbury API.

We do not publish parameter counts, training data sources, compute or costs.

Releases

Atom v1.2 is the current release, and its public benchmarks have not been re-run yet. Every benchmark number in this report belongs to an earlier release, labelled on each row. None of it should be read as a v1.2 result.

How we evaluate

We run public benchmarks with our own harness on the exported checkpoint, through the same chat template and system prompt that serve Incbury Chat. Decoding is greedy (temperature 0), and no tools are available. The maths results therefore leave out the Python sandbox that Atom can use in the product.

• IFEval (541 prompts) tests formatting and content constraints that a program can verify, for example "answer in under 50 words". We report strict accuracy at the prompt level and, where it was recorded, at the instruction level.
• GSM8K is grade-school maths word problems, scored by exact match on the final number. v1.1 used a 300-problem subset of the test set and v1.0 the full 1,319 problems; each row shows its sample size.
• HumanEval (164 tasks): each generated function is run against the reference tests. We report pass@1, with one sample and no retries.
• MMLU (5-shot) and the commonsense suites (HellaSwag, ARC-Easy, ARC-Challenge, PIQA and Winogrande) are scored by log-likelihood on the base model, before post-training.

Benchmark test sets are excluded from training data, and maths training data is decontaminated against GSM8K.

Results

The reasoning mode and decoding limits for v1.0 were not recorded with its results, and its GSM8K score covers the full test set, so it does not compare directly with v1.1.

Limitations

• Atom is a small model. It knows less than the largest models and can state false things with confidence, so check anything that matters.
• Without tools, multi-step maths is weak: v1.1 solved 27.0% of the GSM8K subset at Medium.
• Code generation is limited: v1.1 passed 12.2% of HumanEval tasks at the first attempt.
• On MMLU's four-option multiple choice, knowledge is close to chance: 27.2%, against 25% for random guessing.
• It works best in English.
• Benchmarks measure narrow skills and say little about how Atom handles your own work. We also track an internal development set, described in the v1.2 release report. That set guides our training, so its results are not evidence of general improvement.