How Do You Know Your AI Feature Works? A Practical Guide to Evals

I've written a lot of "build an agent in N lines" posts. They're fun to write and people like them. But there's a question that sits directly between one of those demos and anything you'd actually put in front of a customer, and I've never really answered it:

How do you know it works?

Not "did it work when I tried it." That's the demo. I mean: you changed the system prompt on Tuesday, does it still work? You swapped to a cheaper model to cut the bill, did quality hold? A user reports a bad output — is that a new bug or has it always done that?

If your answer to any of those is "I'll try a few prompts and see how it feels," you don't have a product. You have a demo that hasn't broken in front of you yet.

This is the single biggest reason AI features die between prototype and production, and it's the least glamorous thing in the field. Nobody's writing threads about assertion libraries. So let me write the boring one.

Why normal testing doesn't work here, and what changes

Traditional tests assert equality. Same input, same output, forever. Models are non-deterministic — the same prompt can produce different wording every time — so assertEquals is useless on the raw text.

That leads people to one of two bad conclusions: "you can't test this," or "we'll have a human review outputs." The first is wrong and the second doesn't scale past about twenty examples.

The actual shift is this: you stop asserting on the exact output and start asserting on properties of the output. Not "the response equals this string" but "the response contains a valid ISO date," "the JSON parses and has these five keys," "the answer cites at least one source from the provided documents," "the refund amount is under the policy cap."

Once you frame it that way, most of what you want to check is ordinary code.

Start with a JSON file

Your eval suite starts as a file. Not a framework, not a platform, not a subscription. A file.

[
  {
    "id": "refund-within-window",
    "input": "I bought this 10 days ago and want to return it.",
    "context": { "purchase_date": "2026-07-27", "today": "2026-08-06" },
    "expect": {
      "decision": "approve",
      "must_mention": ["30-day"],
      "must_not_mention": ["restocking fee"]
    }
  },
  {
    "id": "refund-outside-window",
    "input": "I bought this back in February, can I return it?",
    "context": { "purchase_date": "2026-02-14", "today": "2026-08-06" },
    "expect": {
      "decision": "deny",
      "must_mention": ["30-day"]
    }
  }
]

Where do the cases come from? In order of value:

1. Every bug anyone has ever reported. This is the highest-value source and it's free. Somebody complains, you fix it, the case goes in the file. Now it can never regress silently again. This is exactly how regression suites work everywhere else.
2. The edge cases you worried about at 2 a.m. Empty input. Someone being rude. A question in another language. A request that should be refused. The ambiguous case where two policies collide.
3. Boring happy paths. You need these, because a change that fixes an edge case sometimes breaks the common one.

Twenty cases is a genuinely useful suite. Fifty is strong. You do not need a thousand. The failure mode is having zero, not having too few.

Tier one: assertion checks

Start here, and stay here as long as you can. These are free, instant, and completely deterministic.

const CHECKS = {
  isJson: (out) => {
    try { JSON.parse(out); return { pass: true }; }
    catch (e) { return { pass: false, reason: `Invalid JSON: ${e.message}` }; }
  },

  hasKeys: (out, keys) => {
    const obj = JSON.parse(out);
    const missing = keys.filter((k) => !(k in obj));
    return missing.length
      ? { pass: false, reason: `Missing keys: ${missing.join(', ')}` }
      : { pass: true };
  },

  mentions: (out, phrases) => {
    const lower = out.toLowerCase();
    const absent = phrases.filter((p) => !lower.includes(p.toLowerCase()));
    return absent.length
      ? { pass: false, reason: `Never mentioned: ${absent.join(', ')}` }
      : { pass: true };
  },

  doesNotMention: (out, phrases) => {
    const lower = out.toLowerCase();
    const present = phrases.filter((p) => lower.includes(p.toLowerCase()));
    return present.length
      ? { pass: false, reason: `Should not have mentioned: ${present.join(', ')}` }
      : { pass: true };
  },

  matches: (out, pattern) =>
    new RegExp(pattern).test(out)
      ? { pass: true }
      : { pass: false, reason: `No match for /${pattern}/` },
};

A surprising share of what you care about is checkable this way. Does the classifier return one of the five allowed labels? Does the extracted invoice total parse as a number? Does the SQL the model generated actually run against a test database? Does the generated code pass node --check? Does the summary stay under the length the UI can render?

That last category — run the output and see if it works — is the strongest signal available and people consistently under-use it. If the model produces something executable, execute it. That's not a heuristic, it's ground truth.

If you can use structured outputs to constrain the response to a schema at the API level, do that too. It doesn't replace evals — a schema-valid answer can still be wrong — but it eliminates a whole class of parse failures before they happen.

Tier two: LLM-as-judge, used carefully

Some things genuinely resist assertion. "Is this summary faithful to the source?" "Is this reply appropriately empathetic?" "Did it actually answer the question?" For those, you use a model to grade a model.

This works, but it's easy to do badly. Four rules:

Grade one narrow property at a time. A judge asked "is this response good?" returns noise. A judge asked "does the response contain any factual claim not supported by the source document? Answer yes or no" returns something you can use.

Force structured output. You want a parseable verdict, not a paragraph.

Use a different context, and ideally a different call, from the one that generated the output. A model asked to grade its own work in the same conversation says yes. Of course it does — it's grading its own homework with its own rubric while holding its own red pen. The failure isn't dishonesty, it's shared blind spots: whatever assumption produced the bad output also produces the approval.

Validate the judge itself. Hand-label twenty outputs yourself, run the judge on them, and see how often it agrees with you. If your judge is only 70% aligned with your own judgment, its scores are barely more informative than a coin flip, and you need a better rubric before you trust it on anything.

async function judge(output, criterion, sourceDoc) {
  const res = await client.messages.create({
    model: 'claude-haiku-4-5',
    max_tokens: 512,
    system:
      'You are a strict evaluator. Judge only the single criterion given. ' +
      'Default to failing when uncertain. Return JSON only.',
    messages: [{
      role: 'user',
      content: [
        `<source>\n${sourceDoc}\n</source>`,
        `<output>\n${output}\n</output>`,
        `<criterion>${criterion}</criterion>`,
        'Return: {"pass": boolean, "reason": "one sentence"}',
      ].join('\n\n'),
    }],
  });

  return JSON.parse(res.content.find((b) => b.type === 'text').text);
}

Note the model choice. Judges run on every case on every run, so use a cheap fast model. Note also "default to failing when uncertain" — judges skew generous, and you want the bias pointing toward catching problems.

The runner

import evals from './evals.json' with { type: 'json' };

async function run(feature) {
  const results = [];

  for (const testCase of evals) {
    const output = await feature(testCase.input, testCase.context);
    const failures = [];

    for (const [check, arg] of Object.entries(testCase.expect)) {
      if (!CHECKS[check]) continue;
      const result = CHECKS[check](output, arg);
      if (!result.pass) failures.push(`${check}: ${result.reason}`);
    }

    results.push({ id: testCase.id, passed: failures.length === 0, failures, output });
  }

  const passed = results.filter((r) => r.passed).length;
  console.log(`\n${passed}/${results.length} passed\n`);

  for (const r of results.filter((r) => !r.passed)) {
    console.log(`✗ ${r.id}`);
    r.failures.forEach((f) => console.log(`    ${f}`));
  }

  return results;
}

Save every run to a timestamped file. The score alone tells you less than the diff — you want to see which cases flipped, and in which direction.

The thing evals are actually for: change

Here's where this stops being a chore and starts paying rent.

Swapping models. You want to drop from a big model to a small one and cut your bill by 80%. Without evals that's a leap of faith you'll never take. With evals it's a twenty-minute experiment: run the suite on both, compare. Maybe the small model scores 47/50 against the big one's 49/50, and the three failures are all in a category you can route separately. Now you have a decision instead of a hunch.

Editing prompts. Prompt changes are the least reviewable code in your repo. Nobody can look at a diff of a system prompt and tell you what it broke. The eval suite can, in ninety seconds.

Provider drift. Models get updated. Behavior shifts under you without a version bump. Running the suite on a schedule turns that from a mystery bug report into an alert.

Cheaply reproducing bugs. "Customer says it gave a wrong refund amount" becomes a test case, then a fix, then permanent protection.

Then wire it into CI so it runs on every change to the prompts or the model config:

{
  "scripts": {
    "eval": "node evals/run.js",
    "eval:ci": "node evals/run.js --threshold 0.9"
  }
}

Set a threshold rather than demanding 100%. A hard requirement of every case passing means people start deleting inconvenient cases, and a suite people game is worse than no suite at all. Fail the build below your bar, and make the report readable enough that a drop from 96% to 92% gets investigated rather than re-run.

Practical notes from doing this wrong first

Run each case more than once if the variance matters. Non-determinism means a case can pass on one run and fail on the next. If a case is flaky, that's information — either your prompt is underspecified or your check is too strict. Three runs and a majority vote costs little and stops you chasing ghosts.

Set a temperature of zero where you can. Some newer models don't accept sampling parameters at all anymore, so this is less available than it used to be — but where you can reduce variance, do it, so eval failures point at real regressions instead of dice rolls.

Watch the cost of the suite itself. Fifty cases with a three-lens judge on each is 200 model calls per run. Use cheap models for judges, batch the runs where latency doesn't matter, and don't run the full suite on every commit if a subset covers the change.

Keep failing cases in the file. The temptation to delete a case you can't fix is enormous. Mark it known_failure: true and let it sit there as a visible acknowledgment of a limitation. A suite that only contains problems you've already solved measures nothing.

Start today, with five cases

You don't need to build all of this before it's useful. Here's the version that fits in an afternoon:

1. Make evals.json with five cases — three ordinary, two edge.
2. Write three assertion checks that matter for your feature.
3. Write a twenty-line runner that prints pass/fail.
4. Add a case every time someone reports something broken.

That's it. That's the whole discipline. It's not sophisticated and it doesn't need to be.

The gap between a demo and a product isn't a better model or a cleverer prompt. It's the ability to change something and know within two minutes whether you made it better or worse. Everything else in this field is downstream of that.