Jev vs. Kodiak: I Made Two "System One" Decision Models Play Zork

If you've used Jev, TypeSafe AI's "System One" decision model, you know the idea: instead of generating text, the model picks from options you give it, with a confidence you can put a threshold on. Yesterday I wrote about Kodiak, my open-weights take on the same idea: 150M parameters, Apache-2.0, runs on a CPU.

So I made them both play Zork.

Not because I expected either to win. Zork is hard even for the biggest chat models, and neither of these models was built for text adventures. I wanted a test where the thing System One models are actually for, fast, on-menu decisions with an honest confidence, has to hold up over hundreds of decisions in a row. Same harness, same game, same random seeds, one model swapped for the other. Jev played better. Then it walked into a grue, and the reason why turned out to be the most useful thing in the whole project.

• Code: github.com/grizzlypeaksoftware/kodiak-plays-zork (Apache-2.0)
• Model: cortex-agent-llc/kodiak-small-r1-preview and the new v2 preview

As with the other projects, I wrote the spec and worked through it with Claude Code on my DGX Spark. Every number below came off that machine, on the CPU.

Playing a text adventure without typing

A text adventure normally needs you to type commands. Kodiak can't type, so it plays by choosing. Microsoft Research's Jericho framework runs the game and lists the commands that actually do something at each turn ("open mailbox", "north", "take leaflet"). Each turn, the harness:

1. builds a short state: the room, the last few moves and what happened, the inventory, the score;
2. asks Kodiak, "Which command best makes progress?", with the valid commands as the options;
3. takes Kodiak's pick if its confidence is at least 0.5. Otherwise it's a System 2 turn, and a fallback decides: plain exploration (try something you haven't tried here), or a local LLM.

That third step is the whole product idea in miniature: a fast model handles the moves it's sure about, and the rest escalate.

The first promise holds absolutely. Across more than 200 games and about 20,000 moves, not one command was invalid. Neither Kodiak nor Jev can answer outside the options it's given, so neither can hallucinate a command, misspell one, or invent an object. That's not a tuning result; it's how these models work.

Finding 1: the first thing to build was a way to lose

My first runs looked encouraging. Kodiak made about half the moves on its own, at around 200 ms each on the CPU. Then I asked the obvious question: compared to what?

So I added a baseline with no model at all: the same harness, where every turn is an exploration turn. Five games, five random seeds each, 100 moves per game. The result was humbling:

Points gained, first harness	Exploration only	Kodiak cascade
Zork I	6.0	1.0
Detective	96	46
All Quiet on the Library Front	13.0	5.6

Kodiak wasn't just failing to help. Where it made the most moves, it made the game worse. Its confident picks were worse than trying something new at random. A share-of-moves number would have hidden that completely: in one early run Kodiak made 61% of the moves, about 20 of them spent picking up and putting down the same bottle.

Lesson: before you measure your model, build the dumbest thing that could possibly work and measure that.

Finding 2: one sentence was worth 35x

I tried three wordings of the question, tuning only on practice games and never on Zork:

Question wording	Points per 100 of Kodiak's own moves
"Which command best makes progress in this adventure?" (the original)	1.2
"Which command should the player try next?"	26.8
"Which command best makes progress in this text adventure: exploring new places, getting useful items or solving a puzzle?"	42.3

Same model, same game states, a 35x difference from the question alone. Kodiak reads the question as part of the input, so spelling out what "progress" means gives it something concrete to match the options against. For anyone using a decision model: the question is a prompt, and it deserves the same care as one.

Finding 3: the harness mattered more than the model

The second version of the harness added four things, none of them specific to any game, and none of them allowed to peek ahead at what a move would do:

• the better question wording;
• a rule against immediately undoing the last move (open then close, take then drop);
• the score, and when points last came in, shown in the state;
• a list of exits not yet tried from the current room.

Points gained, second harness	Exploration only	Kodiak cascade
Zork I	13.0	6.0
Detective	92	178
All Quiet on the Library Front	12.4	12.8
Balances	10	9

Kodiak's own moves went from earning 14 points per 100 moves to 68. On Detective, a game with frequent rewards where the obvious move is usually right, its confident picks beat exploration's. On the other games it ties, and on Zork it still loses: its Zork moves earned nothing. Behind the white house, with the kitchen window open, it wants to close the window, every single time.

Finding 4: the game engine was the flakiest part

The benchmark hung overnight: ten hours stuck at move 88 of one Zork game. The culprit was a single command. When the bird's nest ends up inside the egg you're carrying, merely testing "drop all down nest" crashes the game emulator. Jericho can't recover in any of its modes, and its parallel mode waits forever instead.

The fix: run each turn's valid-move search in a throwaway child process. If it crashes, find the command that did it, block it for the rest of the game (a command that crashes the game isn't a valid move), and carry on. It costs about 0.1 seconds per turn. I also gave the benchmark a timeout per game, so one bad game can't stall a whole night again. The details and a reproduction are in the README.

Finding 5: the new model is more confident, not more right

While I was benchmarking, the Kodiak v2 preview went up on Hugging Face, so I ran it through the frozen harness: same games, same seeds, only the model changed.

	Kodiak r1	Kodiak v2
Zork I, points gained	6.0	21.4
Detective, points gained	178	10
Share of moves the model made	25%	53%
Points per 100 of the model's own moves	68	2

v2's Zork number looks great until you read the transcripts. The points came from exploration, in two lucky games that reached the cellar; v2's own Zork moves earned two points. On Detective it found a trap: after taking the paper in the chief's office (+10), it walked outside and then repeated "north" (which fails), "put paper down" and "take paper" for 95 moves. The only real exit, "west", was on the list every turn, and the state even listed it as an untried exit. v2 never picked it, and because it was confident, the fallback never got a turn.

That's the failure mode a System 1 / System 2 cascade is most exposed to: confidence without judgment. An unsure model is cheap, because the fallback catches it. A confidently wrong one gets to keep going. Those transcripts are going straight into the next round of training data.

Finding 6: Kodiak vs. Jev

Jev has an API, so I added it as an optional backend: the same states, options and questions, sent to TypeSafe's endpoint. Jev has no "I can't tell" answer, so its cascade runs on confidence alone, against the same 0.5 threshold. Five Zork I games, 100 moves each:

Zork I, 5 games	Exploration only	Kodiak r1	Kodiak v2	Jev
Points gained, average	13.0	6.0	21.4	11.0
Rooms visited	10.2	10.0	11.2	12.2
Share of moves the model made	–	18%	31%	50%
Points from the model's own moves	–	0	2	45
Deaths	0	1	1	3
"Is the player in danger?" says yes	–	13%	60%	4%
Time per decision	–	275 ms, local CPU	301 ms, local CPU	133 ms, over the network
Invalid moves	0	0	0	0

Jev plays Zork noticeably better, move by move. It opens the mailbox on move one; both Kodiak models walk west. It climbs in through the kitchen window on its own in four of five games. Its own moves earned 45 points where Kodiak's earned almost none. Its danger answers are far better calibrated too. I also replayed all 498 of Jev's decisions through both Kodiak models offline: they agree with Jev's pick only about one time in five.

And yet it didn't beat the no-model baseline, because of three deaths. Each time, Jev climbed into the dark attic, the room where the game tells you "you are likely to be eaten by a grue," and walked north into one. The detail I keep coming back to: in every one of those deaths, Jev's own answer to "Is the player in danger?" was yes, at about 0.95, in the same call that chose the fatal move.

The model knew. The harness didn't listen. My harness logs the danger answer but doesn't act on it, and that's on me, not Jev. It's also the most useful thing I learned in this whole project: when a decision model can answer several questions at once, the second question can guard the first. "In danger, so don't walk into the dark; escalate instead" is a one-line rule. That's the next version of the harness, for every model.

For context on the wider field: agents trained with reinforcement learning on Zork I itself reach 30 to 45 points, and LLM agents on the same framework about 50. Frontier chat models that were handed the manual, a map and a walkthrough score from about 40 to over 200 in 200 moves. None of that is a like-for-like comparison, so I've kept it out of the headline. The fair comparison is the one inside the repo: same harness, same seeds.

Where this leaves Kodiak

Honestly: the r1 preview is a cautious, fast, free model that doesn't know how to play text adventures, and v2 is a more confident one that doesn't either. Jev is clearly ahead on this task. What Kodiak does deliver is what it was built for: every move valid, local, no per-call cost, a few hundred milliseconds on a CPU, and a clean way to hand the hard turns to something bigger.

The demo also gave me something I didn't have before: a reproducible, long-horizon test that shows how a decision model fails, not just how often. Confidence that doesn't track quality, and a danger signal that's right but ignored, are both things a single-turn eval would never have caught.

What I'd tell someone building one of these

• Build the no-model baseline first. Without it, "the model made 60% of the moves" sounds like a result.
• Measure the model's own decisions, not just the final score. A cascade can score well while the model contributes nothing.
• Treat the question like a prompt. One sentence was worth 35x.
• Freeze the harness before comparing models, tune on practice games only, and keep the test game out of it.
• Use every answer you pay for. If the model tells you it's in danger, don't let it walk into the dark.
• Run several seeds and show the spread. One game is an anecdote; the ± column is where most of these tables stop looking exciting.

---

Kodiak and the games ran on my DGX Spark's CPU; Jev ran on TypeSafe's servers. Results use 5 seeds × 100 moves per game, and the Jev comparison covers Zork I only, so treat small differences as ties. Game files aren't included; Zork I is a commercial Infocom title, and I used my own copy. Kodiak is by Cortex Agent LLC, my company, under Apache-2.0. Jev was called through TypeSafe AI's public API under my own account; this article makes no claims about Jev's internals. The harness, benchmark and full results are in the repo.