Build Once, Run on Any Provider: A Portable LLM Abstraction Layer

If you're the kind of developer who reads free-tier comparisons — and a lot of you are, because those are the articles of mine that people actually find — then you are going to switch providers. Not maybe. The free tier tightens, a better model ships, the one you picked has a bad month, or the bill finally arrives.

The question is whether switching costs you an afternoon or a fortnight.

I once spent two hours migrating a service off a platform that was charging me $300 a month. Two hours, because the provider-specific code lived behind one interface. If it had been scattered through forty files, that's a sprint, and I'd probably have just kept paying.

So here's the layer. But first, an honest caveat, because there's a real cost to this pattern and most articles skip it.

When not to do this

If you call one model in one place and you're fine with that, do not build an abstraction layer. Use the official SDK directly. You'll get better types, better errors, and access to provider-specific features the day they ship instead of whenever you get around to plumbing them through.

The abstraction earns its keep when you have several call sites, when you genuinely expect to switch, or when you want a fallback chain for reliability. Below that bar it's ceremony.

There's also a real cost even when it is justified: an abstraction is a lowest-common-denominator surface. Prompt caching, extended thinking, structured outputs, server-side tools — those are exactly the features that differentiate providers and exactly the features a naive adapter flattens away. I'll come back to that, because the answer isn't "give them up."

The interface

Keep it small. Every method you add is a method you must implement N times.

/**
 * @typedef {Object} Message
 * @property {'user'|'assistant'} role
 * @property {string} content
 *
 * @typedef {Object} CompletionRequest
 * @property {Message[]} messages
 * @property {string}   [system]
 * @property {number}   [maxTokens]
 * @property {Tool[]}   [tools]
 *
 * @typedef {Object} CompletionResponse
 * @property {string} text
 * @property {ToolCall[]} toolCalls
 * @property {'stop'|'length'|'tool_use'|'refusal'} finishReason
 * @property {{ inputTokens: number, outputTokens: number }} usage
 * @property {string} model
 * @property {object} raw          // always keep the escape hatch
 */

class Provider {
  async complete(request) { throw new Error('not implemented'); }
  async *stream(request)  { throw new Error('not implemented'); }
}

That raw field is the most important line in the file. It's your escape hatch: when you need something the abstraction doesn't model, you reach into raw at one call site instead of redesigning the interface. Without it, every unmodeled feature becomes a refactor.

Two adapters

import Anthropic from '@anthropic-ai/sdk';

class AnthropicProvider extends Provider {
  constructor({ model = 'claude-opus-5' } = {}) {
    super();
    this.model = model;
    this.client = new Anthropic();
  }

  async complete({ messages, system, maxTokens = 4096, tools }) {
    const res = await this.client.messages.create({
      model: this.model,
      max_tokens: maxTokens,
      ...(system ? { system } : {}),
      ...(tools ? { tools: tools.map(toAnthropicTool) } : {}),
      messages: messages.map((m) => ({ role: m.role, content: m.content })),
    });

    return {
      text: res.content.filter((b) => b.type === 'text').map((b) => b.text).join(''),
      toolCalls: res.content
        .filter((b) => b.type === 'tool_use')
        .map((b) => ({ id: b.id, name: b.name, args: b.input })),
      finishReason: mapAnthropicStop(res.stop_reason),
      usage: {
        inputTokens: res.usage.input_tokens,
        outputTokens: res.usage.output_tokens,
      },
      model: res.model,
      raw: res,
    };
  }
}

const mapAnthropicStop = (r) => ({
  end_turn: 'stop',
  max_tokens: 'length',
  tool_use: 'tool_use',
  refusal: 'refusal',
}[r] ?? 'stop');

And an adapter for anything speaking the OpenAI-compatible chat-completions shape, which covers a large share of the ecosystem — Groq, Mistral, local runtimes like Ollama and vLLM, and various hosted gateways:

class OpenAICompatProvider extends Provider {
  constructor({ baseUrl, apiKey, model }) {
    super();
    Object.assign(this, { baseUrl, apiKey, model });
  }

  async complete({ messages, system, maxTokens = 4096, tools }) {
    const body = {
      model: this.model,
      max_tokens: maxTokens,
      messages: system
        ? [{ role: 'system', content: system }, ...messages]
        : messages,
      ...(tools ? { tools: tools.map(toOpenAITool) } : {}),
    };

    const res = await fetch(`${this.baseUrl}/chat/completions`, {
      method: 'POST',
      headers: {
        'content-type': 'application/json',
        authorization: `Bearer ${this.apiKey}`,
      },
      body: JSON.stringify(body),
    });

    if (!res.ok) throw await toProviderError(res);

    const json = await res.json();
    const choice = json.choices[0];

    return {
      text: choice.message.content ?? '',
      toolCalls: (choice.message.tool_calls ?? []).map((t) => ({
        id: t.id,
        name: t.function.name,
        args: JSON.parse(t.function.arguments),
      })),
      finishReason: { stop: 'stop', length: 'length', tool_calls: 'tool_use' }[choice.finish_reason] ?? 'stop',
      usage: {
        inputTokens: json.usage.prompt_tokens,
        outputTokens: json.usage.completion_tokens,
      },
      model: json.model,
      raw: json,
    };
  }
}

Now switching is configuration:

const PROVIDERS = {
  anthropic: () => new AnthropicProvider({ model: 'claude-opus-5' }),
  groq:      () => new OpenAICompatProvider({ baseUrl: 'https://api.groq.com/openai/v1', apiKey: process.env.GROQ_API_KEY, model: process.env.GROQ_MODEL }),
  local:     () => new OpenAICompatProvider({ baseUrl: 'http://localhost:11434/v1', apiKey: 'ollama', model: 'llama3.1' }),
};

const llm = PROVIDERS[process.env.LLM_PROVIDER || 'anthropic']();

The parts that genuinely don't normalize

This is the section other posts leave out, and it's where you'll spend your time.

Tool-call shapes differ structurally. Anthropic returns tool calls as content blocks with input already parsed into an object. The OpenAI-compatible shape returns them under message.tool_calls with function.arguments as a JSON string you must parse. Worse, that string is model-generated, so it can be malformed. Always wrap the parse:

const parseArgs = (s) => {
  try { return JSON.parse(s); }
  catch { return { _parseError: true, _raw: s }; }
};

Feeding results back differs too: one expects a tool_result content block inside a user message, the other expects messages with role: "tool" and a tool_call_id. Your adapter has to own both directions, not just the outbound half.

System prompts sit in different places. Anthropic has a top-level system parameter. The OpenAI shape puts it as the first message in the array. Straightforward, but if you get it wrong the prompt silently becomes an ordinary user turn.

Sampling parameters are not universal anymore. This one bites. Anthropic's newest models — Opus 5, Sonnet 5, Fable 5 — reject temperature, top_p, and top_k outright with a 400. If your abstraction dutifully forwards temperature: 0.7 to every provider, you've built something that hard-fails on the newest models in the ecosystem.

The right handling is per-adapter capability, not a universal passthrough:

class AnthropicProvider extends Provider {
  static REJECTS_SAMPLING = new Set(['claude-opus-5', 'claude-sonnet-5', 'claude-fable-5']);

  #sampling(request) {
    if (AnthropicProvider.REJECTS_SAMPLING.has(this.model)) return {};
    return request.temperature != null ? { temperature: request.temperature } : {};
  }
}

The general lesson: an adapter's job is to translate intent, not to forward fields. "I want low variance" is intent. temperature: 0 is one provider's expression of it, and on some models the expression is now "use a lower effort setting" or nothing at all.

Token counts mean different things. Different tokenizers, so the same text yields different counts. Any cost math has to be per-provider, and any context-window budgeting you do against one provider's numbers is wrong on another.

Errors have nothing in common. Normalize them at the boundary or you'll be string-matching error messages in business logic forever:

class ProviderError extends Error {
  constructor(message, { status, kind, retryAfterMs, provider }) {
    super(message);
    Object.assign(this, { status, kind, retryAfterMs, provider });
  }
}

// kind: 'rate_limit' | 'auth' | 'bad_request' | 'server' | 'timeout' | 'refusal'

Your retry logic branches on kind. It should never see a raw provider payload.

Streaming

Streaming is where a sloppy abstraction leaks worst, because event shapes differ substantially. Normalize to an async generator of simple events:

async function* stream(request) {
  const s = this.client.messages.stream({ /* ... */ });

  for await (const event of s) {
    if (event.type === 'content_block_delta' && event.delta.type === 'text_delta') {
      yield { type: 'text', text: event.delta.text };
    }
  }

  const final = await s.finalMessage();
  yield { type: 'done', usage: { inputTokens: final.usage.input_tokens, outputTokens: final.usage.output_tokens }, raw: final };
}

Three event types — text, tool_call, done — covers the overwhelming majority of consumers, and your UI code stops caring which provider is behind it.

Preserving the good stuff

Here's the resolution to the lowest-common-denominator problem: make advanced features optional capabilities rather than required interface methods.

class Provider {
  supports(feature) { return false; }
}

class AnthropicProvider extends Provider {
  supports(feature) {
    return ['prompt_caching', 'batch', 'thinking', 'structured_output'].includes(feature);
  }

  async complete(request) {
    const system = request.system && this.supports('prompt_caching') && request.cacheSystem
      ? [{ type: 'text', text: request.system, cache_control: { type: 'ephemeral' } }]
      : request.system;
    // ...
  }
}

Now your calling code can opt in without breaking portability:

const res = await llm.complete({
  messages,
  system: LARGE_STABLE_PROMPT,
  cacheSystem: true,   // honored where supported, ignored where not
});

Providers that support caching get 90% off the repeated prefix. Providers that don't just… don't, and nothing breaks. You keep the differentiating feature and the portability, which is the whole point.

The fallback chain

Once every provider is behind one interface, resilience is nearly free:

class FallbackProvider extends Provider {
  constructor(providers) { super(); this.providers = providers; }

  async complete(request) {
    let last;
    for (const p of this.providers) {
      if (p.openUntil > Date.now()) continue;
      try {
        return await p.complete(request);
      } catch (err) {
        last = err;
        if (err.kind === 'rate_limit' || err.kind === 'server') {
          p.openUntil = Date.now() + (err.retryAfterMs ?? 60_000);
          continue;
        }
        throw err;  // bad_request and auth fail identically everywhere
      }
    }
    throw last;
  }
}

const llm = new FallbackProvider([primary, secondary, local]);

The circuit breaker matters — without it you pay full latency on the dead provider before every single call. And note that a bad_request rethrows immediately: a malformed payload isn't going to succeed on provider two, so failing over just wastes quota to produce the same error.

Prove it works, then switch

The last piece: run your eval suite against each provider. That's what turns "I could switch" into "I know exactly what switching costs me."

for (const [name, make] of Object.entries(PROVIDERS)) {
  const results = await runEvals(make());
  const passed = results.filter((r) => r.passed).length;
  console.log(`${name}: ${passed}/${results.length}`);
}

That output is a decision-support table. Provider A scores 48/50 at ten times the price; provider B scores 46/50 and is free. Now the tradeoff is a number instead of an argument.

What you actually bought

You didn't build this to be clever. You built it so that the next time a provider triples its price, changes its free tier, or ships a model that makes your current one look silly, the response is editing one environment variable and running the eval suite.

Providers are a commodity you rent. Your prompts, your evals, your tool definitions, and your product are the assets you own. Keep the boundary between them sharp enough that the commodity stays swappable, and the whole free-tier shopping habit stops being a liability and starts being leverage.