I Built an Open-Source "System One" Decision Model on My DGX Spark in Three Days

A few days ago I wrote about Jev, TypeSafe AI's "System One" decision model, and about using it to guard agent tool calls. The idea wouldn't leave me alone. So I built an open-source one, from scratch, on my own hardware.

It's called Kodiak. You give it a state (a message, a thread, or a JSON record) and a set of typed questions. It answers every question in a single forward pass, in about 8 milliseconds, and every answer comes with a probability you can actually trust. It can't answer outside the options you gave it, and when the information isn't there, it says so.

• Try it: the Kodiak demo
• Code, docs and every decision: github.com/grizzlypeaksoftware/kodiak
• Weights (research preview, Apache-2.0): cortex-agent-llc/kodiak-small-r1-preview

It took three days, one DGX Spark, and a few tens of dollars of cloud inference. Every number below came off my machine.

Kodiak (orange) against the open zero-shot classifiers people use today. It wins big overall and on calibration, and it's slightly behind on never-seen tasks. More on that below.

What this actually is

Most of what software asks LLMs to do isn't writing. It's deciding:

• Which of these 12 intents is this customer message?
• How urgent is this ticket, 0 to 10?
• Which tool should this agent call next?
• Is this a prompt-injection attempt?
• Can this even be answered from what we have?

Sending those to a chat model means paying for slow, token-by-token generation, then parsing the output and hoping it stayed on the menu. Kodiak answers them directly:

from kodiak_s1.hub import Kodiak

kodiak = Kodiak.from_pretrained("cortex-agent-llc/kodiak-small-r1-preview")   # CPU works fine
kodiak.decide(
    "Hi, I ordered the walnut desk two weeks ago. Tracking has said 'label created' for 10 days. "
    "If it can't arrive by Monday, please cancel and refund me.",
    [{"type": "choice", "id": "intent", "text": "What does the customer want?",
      "labels": ["delivery status or expedite", "cancel and refund", "product question"]},
     {"type": "score", "id": "urgency", "text": "How urgent is this?", "min": 0, "max": 10},
     {"type": "choice", "id": "carrier", "text": "Which carrier is shipping it?", "labels": ["UPS", "FedEx", "USPS"]}],
)

On that example it answers cancel and refund (89%), urgency about 6.6 out of 10 with an interval, and for the carrier it abstains: the message never says. That last answer is the whole point. A confident wrong answer is worse than "I don't know," and Kodiak treats "I don't know" as a real, calibrated output.

There are three question types:

• Choice: labels you make up per request; it can't pick anything else.
• Score: any range you choose, with a mean and a 90% interval.
• Abstain: any question can come back as "not answerable from this state," with its own probability.

How I built it

Same approach as the fruit fly project: I wrote a phased spec and worked through it with Claude Code running on the Spark, stopping for review after every phase. My job was making the calls (scope, licensing, what to spend) and reading the evidence. The rule was the same too: no claims without numbers.

Three constraints shaped everything:

1. No LLM backbone. No chat model underneath, and no constrained decoding of one. That's the thing Kodiak is trying to beat.
2. Permissive licenses only, for every dataset and model, so the weights can ship under Apache-2.0.
3. An honest comparison against real alternatives on the same eval set.

Finding 1: you don't have to teach it to read

Kodiak is an encoder, not a chatbot. It's built on ModernBERT from Answer.AI, a 150M-parameter model that already reads English because someone else paid to train it on about 2 trillion tokens. I "hired the reader" and taught it a new job.

The original plan had a second track: train an encoder from scratch. On day one the fine-tuned track produced a working model in 35 minutes, and the from-scratch track would have spent days reading about 200 times less text than ModernBERT already had, just to end up a worse reader. I shelved it. If Kodiak ever earns revenue, a custom encoder is on the table.

The clever part is how a request is packed. The document, each question and each question's options go into one sequence, with a custom attention mask:

• the document only sees itself;
• each question sees the document and its own options;
• each option sees the document and its own question, and nothing else.

That buys guarantees I verified with tests:

• Order doesn't matter. A question gets the same answer alone or alongside 30 others.
• Shuffling the options changes nothing. The chat-model habit of favoring "option A" is impossible here.

To control that mask, Kodiak re-implements ModernBERT's architecture in its own code. A parity test proves the copy is bit-identical to Hugging Face's reference: maximum difference 0.0.

Finding 2: 400x faster than a 27B model, and more honest about its confidence

First fight: Kodiak's first model against Qwen 27B running on the same Spark, same 200 test examples.

	Kodiak (150M)	Qwen 27B
Latency per request	8 ms	3,431 ms
Accuracy, familiar kinds of tasks	75.5%	76.5%
Accuracy, never-seen kinds of tasks	64.9%	86.0%
Calibration error (lower is better)	0.095	0.192
When it abstains, how often it's right	90%	68%

Roughly tied on familiar tasks at about 400 times the speed, and twice as trustworthy about its own confidence. But on task types it had never seen, the big model was 21 points better. That gap has been the main thing I've worked on since.

Making that comparison fair took three tries. The first prompt told Qwen to use only information in the text, and it refused 96% of judgment questions like "how toxic is this?" I could have published that and declared victory. Instead we fixed the prompt, fixed the output schema, and compared only once Qwen had a fair shot. Beating a handicapped opponent proves nothing.

Finding 3: training data for pennies, with a rule: don't grade your own homework

Public datasets got me 356,000 training examples from 20 sources, each license checked at its original source (share-alike and non-commercial data excluded). Four datasets are held out entirely, to measure how Kodiak does on tasks it has never seen.

For more variety I built a synthetic data factory. An open-weight model writes realistic documents and typed questions, including some that deliberately can't be answered. Two rules:

• Only openly licensed teachers. Closed-model terms generally forbid using outputs to build competing models, and Kodiak's data will be public. So: gpt-oss-120b (Apache-2.0) and DeepSeek V3.2 (MIT), rented through DigitalOcean's serverless inference.
• The writer doesn't check its own work. I ran a bake-off against my own hand-graded answers. A model from a different company, answering blind, agreed with me 97% of the time; the writer checking itself, 92%, and ten times slower.

The first 9,700 examples cost about $8.50 of cloud inference (plus 3,300 made on local models for free). A cloud pilot of 50 jobs cost four cents.

Finding 4: more data didn't help, but showing it less did

I trained three identical models on 0, 3,300 and 9,400 synthetic examples. More data helped overall accuracy and calibration, but on never-seen tasks the line was flat (63.9%, 61.7%, 62.5%).

Digging in found two things:

• It had learned to refuse too much. The synthetic data taught "if the fact isn't written down, say you can't tell," and Kodiak started refusing questions that needed a little inference: a biography that never literally says "attorney," or "why hasn't my card arrived?", which obviously means card delivery.
• It was cramming. The training mix showed a few small datasets 20 to 30 times per run (the prompt-injection set 29 times). It was memorizing.

The fix was to cap every dataset at three passes and let training finish its full schedule. Overnight, accuracy on never-seen tasks went from 62.5% to 66.4%, and jailbreak detection rose almost 9 points. The cost was about 1.6 points on the small datasets it had memorized. For a model whose whole job is handling new questions, that's an easy trade.

The lesson I keep relearning: decide what result would change your plan before you run the experiment. I had written down "if more data doesn't move the never-seen number, stop buying more of the same," so a disappointing result turned into a clear next step.

Finding 5: a smarter data factory, and agreement isn't correctness

So I rebuilt the factory for better data instead of more:

• Breadth: 16 sectors expanded into 320 domains and 971 document types (tenant complaints, CI logs, payment webhooks, soil test reports), for about a penny.
• Real text: about 45% of examples use real web passages (FineWeb-Edu), and the model only writes the questions.
• Inference: every question is labeled stated, inferred or unanswerable, so Kodiak learns the difference between "not written down" and "not knowable."

Reading the pilot's rejects turned up a sneaky one: the writer kept adding options like "Not known" to the answer list. That's a second way to say "I don't know," which would have muddled Kodiak's calibrated abstention. It's now banned, and filtered.

Then I hand-graded 165 of the new answers: 92.7% correct, and every single "unanswerable" was right. The misses had a pattern: questions with two defensible answers, where two models from two different companies agreed on the wrong one. Agreement isn't correctness. The fix was stricter rules against ambiguous questions, plus two AI critics whose only job is to attack each answer. The full batch of about 9,400 examples cost around $25.

Finding 6: the real competition, and where Kodiak is still behind

Beating a chatbot on speed doesn't tell you whether you're the best of your kind. The fair rivals are the open models built for zero-shot classification: NLI-based classifiers and GLiClass, each about three times Kodiak's size. Same eval set, choice questions:

	Kodiak (150M)	Best rival (GLiClass-instruct, 439M)
Accuracy, all choice questions	78.0%	54.8%
Never-seen tasks, forced to pick an answer	69.1%	70.5%
Calibration error	0.049	0.156
Latency	8 ms	27 ms

Overall it's a blowout, and Kodiak also handles scores and abstention, which these models can't do properly. On never-seen tasks it's close, and Kodiak is slightly behind. It wins big on jailbreak detection (68% vs. about 50%) and loses on guessing someone's occupation from a biography.

One more catch worth sharing. A rival looked strong on banking intents, one of my held-out tasks, and its model card showed it had trained on that dataset. Its clean variant dropped 12 points there. A "zero-shot" score only means something if the model never saw the test.

Before running any of this, I wrote down what "best in its class" has to mean: beat every open zero-shot classifier on never-seen tasks, come within about 10 points of an 8B chat model at 100 times its speed, and have the best calibration. Kodiak doesn't clear that bar yet. That's the point of writing it down first.

It runs on a CPU

The research preview is on Hugging Face. On the Spark it's about 8 ms per request on the GPU and about 80 ms on the CPU alone, so you don't need a GPU to use it. The same package includes a handler for one-click Hugging Face Inference Endpoints, and the demo runs on a free CPU Space.

Even the upload had a catch-our-own-mistake moment. The first version was missing its calibration settings, because my eval code had been applying them separately. We caught it before anyone downloaded it. Test the thing users actually download, not just the pipeline you evaluate.

It's a preview, and it makes mistakes. Tell it a box arrived crushed with a broken lamp and it thinks the customer wants "delivery status" (they want a refund). I left those misses in the demo on purpose. They're the before-and-after test for the next round of training.

Where this is going

The pitch isn't "a smaller ChatGPT." It's System 1 in front of System 2: Kodiak answers every decision in milliseconds, confident answers get used directly, and only the uncertain ones go to a big model or a human. Because the confidence is calibrated, you can set that dial and know what you'll get, and you only pay big-model prices on the hard cases.

Next up:

• Tonight: the head-to-head of the old training data against the new, at equal size. Does better data finally move the never-seen number?
• Hard-example mining: Kodiak screens candidate training examples in milliseconds, and we keep mostly the ones it gets wrong.
• Minimal pairs: twin examples where one word flips the answer, to teach exactly which fact matters.
• A bigger backbone: ModernBERT-large for a quality tier.
• Later: a hosted Kodiak API from Cortex Agent, and fine-tuning on your own labels.

What I'd tell someone who wants to try this

• Start from open pretrained weights. Fine-tuning a decision layer takes about an hour; learning to read takes months. Someone already paid for reading.
• Check licenses at the source, not the tag on a model hub, and use only openly licensed teachers for training data.
• Don't let a model grade its own work. Use a checker from a different family, and grade a sample by hand.
• Read the data, not just the metrics. Most of the real bugs in this project were found by reading examples, not by tests.
• Give your baselines their best shot, and check whether they trained on your test set.
• Write down what would change your mind before you run the experiment.

---

All figures came from runs on my DGX Spark and are documented in the repo, including the dead ends: STORY, DECISIONS and STRATEGY. Kodiak is by Cortex Agent LLC, Apache-2.0. It's inspired by Jev but built independently, and makes no claims about Jev's internals. Feedback and failure cases are welcome as GitHub issues.