What a Weekend of Benchmarking Taught Me About My DGX Spark

I have an NVIDIA DGX Spark on my desk running Ollama as an always-on LLM server. My agents hit it over Tailscale from my laptop and my phone. I'd been running qwen3:30b-a3b as the default for weeks and had a nagging suspicion it was stale, and that my Ollama install was old enough to be costing me throughput.

I was half right, in the least useful way possible. Here's what I actually measured.

Every number below came off my own machine. Nothing is from a vendor chart.

The setup

The Spark is a GB10 Grace Blackwell part with 128GB of unified LPDDR5X and roughly 273 GB/s of memory bandwidth. That bandwidth figure is the single most important number about this machine, and I'll come back to it.

Baseline state: Ollama 0.32.5, driver 580.173.02, CUDA 13.0, DGX OS 7.2.3, 3.4TB free.

I wrote a harness that hits /api/generate with stream:false and records Ollama's own instrumentation — the same prompt_eval_duration and eval_duration fields that ollama run --verbose prints, except machine-readable. Every prompt gets a unique nonce prepended so the KV prefix cache can't serve a cached prefill.

Three context depths, same prompts for every model: 635 tokens, 17.5K, and 34.5K. Three runs each, reporting the spread.

The three depths matter more than anything else I did. I'll show you why shortly.

Finding 1: the Ollama upgrade bought me nothing (almost)

My first hypothesis was that a stale Ollama was leaving throughput on the table. To test it I ran both versions side by side — 0.32.5 on port 11434, 0.32.13 on port 11500 — and alternated between them run by run, so any drift in machine state hit both arms equally.

depth	metric	0.32.5	0.32.13	delta
short	decode	90.7 t/s	90.9 t/s	+0.1%
medium	decode	52.2 t/s	52.4 t/s	+0.2%
deep	decode	36.7 t/s	36.9 t/s	+0.5%
medium	prefill	2,506 t/s	2,545 t/s	+1.5%
deep	prefill	1,926 t/s	1,956 t/s	+1.6%

Decode gains are noise. Prefill is up a consistent ~1.5%, and the sign held in every paired run so I believe it's real, but it is not "a large fraction of available throughput."

The upgrade's value turned out to be access, not speed. Ollama 0.32.5 physically refuses to pull two of the three models I wanted to test:

$ ollama pull nemotron-3.5-lightning:30b
Error: pull model manifest: 412:
The model you are attempting to pull requires a newer version of Ollama.

That's the real cost of running stale. Not slow — locked out. If you take one operational thing from this piece: an HTTP 412 on pull means your runtime is too old for the model, and no amount of retrying fixes it.

Two smaller notes for anyone reproducing this. The release assets renamed from .tgz to .tar.zst, so the download URL you have bookmarked probably 404s. And upgrade with a rollback path — mine kept a copy of the old binary and preserved the systemd drop-ins, which is what made me willing to run it at all on a box my phone depends on.

Finding 2: benchmarking a machine while you use it destroys both

Partway through, my re-baseline produced deep-context decode of 17.6 t/s against a baseline of 36.2. A 51% collapse, reproducible across three runs.

I formed a hypothesis — memory bandwidth contention from having two models resident — and I was wrong. Then I formed a second one, page residency after a cold load, and I was wrong again. Then a third: that my own harness's nvidia-smi polling was perturbing the measurement. Wrong a third time; that costs 0.3%.

The answer was in the journal. During those runs, the production server was decoding a 3,660-token generation for one of my own agents. Two llama-servers generating at once, and both got roughly halved.

So I measured it deliberately:

condition	deep decode	vs solo
solo	35.77 t/s	—
second 22GB model resident, idle	35.76 t/s	−0.1%
second model actively decoding	17.26 t/s	−51.8%

Residency is free. Concurrent inference is not. Keeping a second model warm costs essentially nothing in speed. Two models generating simultaneously split the bandwidth roughly in half, because on unified memory they're pulling from the same 273 GB/s.

This is the Spark's defining characteristic and you can't configure around it. It also means any benchmark you run on a box that's serving live traffic is garbage unless you check.

I'll note the process here honestly: I published three wrong explanations to my own results file before finding the right one. The interleaved A/B design is what saved the version comparison — because it alternates arms, the interference that destroyed my naive sweep hit both sides equally and cancelled out.

Finding 3: the depth curve reorders everything

Here's why three depths mattered.

model	short (635)	medium (17.5K)	deep (34.5K)	short→deep
qwen3:30b-a3b (incumbent)	88.5	51.2	36.2	−59%
qwen3.6:35b-a3b	75.4	66.4	59.8	−21%
nemotron-3.5-lightning:30b	99.1	73.5	71.2	−28%
qwen3.8:27b (dense)	30.5	31.2	~27	−11%

Look at qwen3.6 versus my incumbent. At short prompts it's worse — 75.4 against 88.5. At deep context it's 65% better. If I had benchmarked with a short prompt, which is what most quick comparisons do, I would have concluded qwen3.6 was a downgrade and moved on.

Real agent contexts are not 600 tokens. Mine run to tens of thousands. The short-prompt number is the one everybody quotes and the one that matters least.

Nemotron is the standout: fastest at every depth, roughly double the incumbent at 34.5K tokens, and the flattest curve. Two things contribute, both visible in the process arguments rather than guessed at. It's the only model I tested that gets speculative decoding on the CUDA path:

--spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-backend-sampling

And its architecture is nemotron_h_moe, a hybrid whose non-attention layers don't grow KV state linearly with context.

I'd initially concluded, from release notes, that MTP speculative decoding was Apple/MLX only. That was too broad, and inspecting the live process corrected me: it's enabled per model, for models that ship an MTP head. qwen3 and qwen3.6 get no such flags. Upgrading your runtime for speculative decoding does nothing unless the model exposes the head.

Finding 4: the dense model is uniformly slow, not slow at length

qwen3.8:27b is a dense ~27.8B multimodal model. I expected it to be slower. What I didn't expect was the shape.

Its decode is nearly flat — 30.5, 31.2, ~27 — because it's weight-bandwidth-bound at every depth rather than KV-bound. It isn't "slow at long context." It's uniformly slow.

The number that actually hurts is prefill: ~730 t/s against nemotron's ~2,800. Nearly 4x slower just to read a prompt. On a 35K-token agent context that's about 48 seconds of prefill per call versus 13.

This is the bandwidth ceiling doing exactly what physics says it will. A dense model must stream every parameter for every token. A sparse MoE with ~3B active parameters streams a fraction. On a box where bandwidth is the binding constraint, sparsity isn't an optimization — it's the whole ballgame. No configuration gets around 273 GB/s.

Finding 5: tokens per second is the wrong metric, and I can prove it

This is the part I actually care about, because I run agents, not chatbots.

I built a harness that scores what my agents really do: multi-step tool calling, multi-hop chains, instruction adherence under a large system prompt, and recovery when a tool errors. Deterministic Python scoring, no LLM judge.

My incumbent failed the multi-step tool test 2 out of 2 — while scoring a perfect 1.00 on tool-call cleanliness.

Here's what it did. Asked to read a file, sum the numbers, and file a report, it emitted all three calls in a single turn, before the file contents existed:

CALL file_read  {"path": "/costs/2026-q3.txt"}
CALL calculator {"expression": "100 + 200 + 300"}   <- invented
CALL ship_report {"summary": "Total cost: 600"}     <- real answer is 1618

Every call is schema-valid. A malformed-JSON metric scores this flawless. The model invented its inputs and reported a confident wrong answer.

This is the failure that convinced me tok/s is the wrong headline number. I'd been running this model for weeks. It never threw an error. It just occasionally made things up, in well-formed JSON, and I had no instrumentation that would have caught it.

Then I found something more actionable. My error-recovery test passed on the same model, and the two tests differed in two ways, not one — so I couldn't attribute the difference. I ran the disambiguation:

system prompt	correct	fabricated
control	0/2	yes
+ "Never invent data you did not read from a tool."	2/2	no
+ "Call one tool at a time and wait for its result."	2/2	no

One sentence moves this from 0% to 100%. Same weights, same temperature. Turn count goes from 2 to 4, which confirms the mechanism: the guarded model waits for each result instead of batching.

If your agent system prompts don't contain a line like that, you may be losing correctness you will never see in a benchmark.

Finding 6: when both candidates score perfect, your test is broken

Both new models scored 100% on everything in that first harness, including the unguarded control that breaks my incumbent. That's not a result — it's a ceiling. "Both perfect, pick on speed" would have been a measurement failure dressed up as a finding.

So I built a harder tier: a 12-hop chain seeded with decoys, two sensors returning contradictory values, an unrecoverable missing file, a 25K-token system prompt with contradictory rules plus a precedence rule, and a tool whose enum forbids the value the prompt demands.

The long chain and the contradiction test still saturated — all four models handled them. Three tests discriminated.

On the unrecoverable file, my incumbent and the dense model both invented a total anyway (2/2 each). qwen3.6 and nemotron refused and reported it missing.

The forbidden-enum test was the most useful thing I built, because all four failed differently. I demanded sev1 when the tool only allows sev3–sev5:

• nemotron called it with sev1 anyway, got rejected, then explained the constraint clearly and asked which value to use. Honest — but it broke the contract and then stalled waiting for a human. In an unattended loop that's a hang.
• qwen3.6 called sev3 — valid, closest to "catastrophic" — and never mentioned that sev1 was impossible. Contract kept, silent substitution.
• my incumbent called sev5 — the lowest severity, for a catastrophic outage — then filed a report saying it was handled. Confidently backwards and self-certifying.
• qwen3.8 made no call and explained the limit. Safe but inert.

There is no clean winner. There are three failure modes and one refusal, and which one you want depends entirely on whether a human is watching.

And one genuine surprise: on the precedence rule buried in 25K tokens, only the slow dense model got it right, 2/2. All three MoE models failed. That's the one place dense clearly earns its keep, and it's a reasoning-depth result, not a speed one. Worth remembering before writing dense models off on throughput alone.

What I actually run now

Two models pinned hot, agents choose per task, harder reasoning delegated to cloud models.

• qwen3.6:35b-a3b as the agent workhorse — best failure mode, never fabricated, 65% faster than my incumbent at depth.
• nemotron-3.5-lightning:30b for long-context and throughput work — fastest at every depth, flattest curve.

The two-model setup is viable, and I measured why:

scenario	measured
cold swap (evict each time)	6.07–6.65 s
warm swap (both pinned)	0.41–0.63 s
RAM for both resident	44.9 GiB of 121

Keeping both hot makes swapping 20x cheaper, and an idle second model costs 0.1% speed. What you cannot do is let both generate at once.

I tried pinning a third model. Don't. At 61 GiB resident, Ollama started evicting and reloading, and my "warm" swaps ballooned from 0.4s to 8–14s. Two is comfortable on this box; three thrashes.

I kept qwen3:30b-a3b on disk as a fallback. It's the model that fabricated data on me, but it also has months of real use behind it, and both replacements are between two days and two weeks old.

What I'd tell someone deciding whether to buy one

The Spark is a genuinely good box for local agent work, and 273 GB/s is a hard physical ceiling that shapes everything. Sparse MoE models fly. Dense models of the same nominal size run at roughly a third the speed and take 4x as long to read a prompt. That's not a tuning problem.

Buy it if your workload is sparse models with long contexts and you value the 128GB of unified memory — being able to keep two 20GB+ models permanently hot with room to spare is genuinely useful. Don't buy it expecting dense models to be fast, and don't buy it expecting to run two models concurrently at full speed.

And whatever you run: measure the failure modes, not just the throughput. A model at 70 t/s that fabricates its tool inputs is worse than one at 40 t/s that doesn't. I ran the fast one for weeks without noticing.

---

Raw data, harness code, and the full running log — including the three wrong hypotheses — are in my spark-bench directory. Every figure here came from a command run on that machine.