Skip to content
Log in

Research / Directions

Open questions.

What we are investigating, and why. Nothing here is a result yet. When it is, it will be published with its evidence.

  1. Open question

    Targeted data over generic mixtures

    On our development set, targeted, machine-checked training examples have helped Atom more than generic fine-tuning mixtures. We do not yet know how far that carries over to unseen tasks.

    Question: Do gains from targeted data hold up on held-out tasks?

  2. Open question

    Tool use in preference training

    Preference optimisation lowered Atom's development-set score both times we tried it. One hypothesis is that our sampler cannot run the Python Atom writes, so correct tool-using answers were marked as rejected.

    Question: Does preference training help once tool calls are executed during sampling?

  3. Open question

    Asking the right question

    Atom now asks when a request is ambiguous, but its questions are often too vague to help.

    Question: How do we train a small model to ask the one question that matters?

  4. Open question

    Knowledge in a small model

    Atom's general knowledge is weak: v1.1 scored 27.2% on MMLU. Search helps, but only when Atom recognises that it needs to look something up.

    Question: When should a small model search rather than answer from memory?

  5. Open question

    Measuring the agent, not just the model

    Our coding-agent results so far come from a harness run on tasks related to our training data.

    Question: What does a fair, held-out and budgeted evaluation of Incbury Code look like?

Working on similar questions?

We would like to hear from people working on small models, evaluation or tool use.

Contact us