Engineering Around Free-Tier Rate Limits: Queues, Backoff, and Failover in Node

Most people find out what a rate limit is at the worst possible moment. The demo works. You show someone. It works again. Then you loop over 200 records and everything turns into 429 Too Many Requests and your carefully built pipeline dies eleven items in with half its work done and no record of which half.

I've written a lot about which providers give you what for free. That's the easy half. The hard half is what happens after you take them up on it, because a free tier isn't just cheaper — it's a much tighter box. Where a paid tier might give you thousands of requests per minute, a free tier gives you tens, plus a daily ceiling, plus a token-per-minute cap that you can blow through with a single large document even when your request count is nowhere near the limit.

The fix is not "call it less." The fix is a client that understands the limits and shapes traffic to fit them. That's maybe 150 lines of Node, and once you have it you can stop thinking about rate limits entirely.

The three limits you're actually up against

Almost every provider enforces some combination of:

• RPM — requests per minute. The one everybody knows about.
• TPM — tokens per minute, usually counted as input plus output. This is the one that gets you. Ten requests carrying 30K tokens of context each will trip a TPM ceiling while your RPM counter sits at 10.
• RPD / TPD — daily ceilings. These don't recover in seconds. If you burn your day's quota at 9 a.m. by retrying a bug in a loop, you are done until the reset.

The important consequence: you cannot solve this with retries alone. Retrying a TPM violation with the same 30K-token payload just re-spends the tokens you don't have. You need to slow down before you send, not only recover after you fail.

Step one: read the headers

Nearly every provider returns its current limit state on every response, whether it succeeded or not. The header names vary but the shape is consistent — a limit, a remaining count, and a reset time. On a 429 you'll usually also get retry-after in seconds.

Stop guessing and start reading them:

function readLimits(headers) {
  const num = (k) => {
    const v = headers.get(k);
    return v == null ? null : Number(v);
  };

  return {
    requestsRemaining: num('x-ratelimit-remaining-requests'),
    tokensRemaining: num('x-ratelimit-remaining-tokens'),
    // retry-after is seconds; some providers send an HTTP-date instead
    retryAfterMs: parseRetryAfter(headers.get('retry-after')),
  };
}

function parseRetryAfter(value) {
  if (!value) return null;
  const seconds = Number(value);
  if (!Number.isNaN(seconds)) return seconds * 1000;
  const when = Date.parse(value);
  return Number.isNaN(when) ? null : Math.max(0, when - Date.now());
}

That retry-after value is not a suggestion. It's the provider telling you exactly how long until you're welcome back. Honoring it is both faster and more polite than any backoff curve you'd invent.

Step two: a token bucket that counts tokens

A plain request-per-second limiter isn't enough, because it doesn't know that this particular call is carrying a 40K-token context. You want a bucket that refills over time and that you draw from by estimated cost, not by request count.

class TokenBucket {
  constructor({ capacity, refillPerSecond }) {
    this.capacity = capacity;
    this.refillPerSecond = refillPerSecond;
    this.tokens = capacity;
    this.last = Date.now();
  }

  #refill() {
    const now = Date.now();
    const gained = ((now - this.last) / 1000) * this.refillPerSecond;
    this.tokens = Math.min(this.capacity, this.tokens + gained);
    this.last = now;
  }

  // Wait until `cost` units are available, then spend them.
  async take(cost) {
    // A single request larger than the bucket can never be satisfied.
    if (cost > this.capacity) {
      throw new Error(`Request costs ${cost}, bucket holds ${this.capacity}. Split it.`);
    }
    for (;;) {
      this.#refill();
      if (this.tokens >= cost) {
        this.tokens -= cost;
        return;
      }
      const deficit = cost - this.tokens;
      const waitMs = Math.ceil((deficit / this.refillPerSecond) * 1000);
      await sleep(waitMs);
    }
  }

  // Called after a 429: assume we know less than the server does.
  drain() {
    this.tokens = 0;
    this.last = Date.now();
  }
}

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

Run two of these: one for requests, one for tokens. If a provider gives you 30 RPM and 6,000 TPM, that's refillPerSecond of 0.5 and 100 respectively.

The interesting question is how you estimate cost before the call. Some providers offer a token-counting endpoint; use it if the accuracy matters. Otherwise, characters divided by four is a rough English-text approximation that runs low on code and non-English input — so pad it. Add your max_tokens to the estimate, because output counts too:

const estimateTokens = (text) => Math.ceil(text.length / 3.5);

const cost = estimateTokens(prompt) + maxTokens;
await tokenBucket.take(cost);
await requestBucket.take(1);

Note that I used 3.5, not 4. Deliberately pessimistic. Guessing high costs you a little throughput; guessing low costs you a 429 and a retry, which is strictly worse.

Step three: backoff with jitter, and know when to quit

When you do get a 429 anyway — and you will, because your estimate is an estimate — back off. But back off with jitter, or every worker in your pool will wake up at the same instant and hammer the provider in a synchronized wave. Synchronized retries are how a brief hiccup becomes a sustained outage.

async function withRetry(fn, { maxAttempts = 5, baseMs = 500, capMs = 60_000 } = {}) {
  let attempt = 0;

  for (;;) {
    try {
      return await fn();
    } catch (err) {
      attempt += 1;

      if (!isRetryable(err) || attempt >= maxAttempts) throw err;

      // Honor the server first. Fall back to decorrelated jitter.
      const exponential = Math.min(capMs, baseMs * 2 ** (attempt - 1));
      const waitMs = err.retryAfterMs ?? Math.random() * exponential;

      await sleep(waitMs);
    }
  }
}

function isRetryable(err) {
  // 429 and 5xx are worth another shot. 400/401/403/404 are not —
  // your key is wrong or your payload is malformed, and waiting won't fix it.
  return err.status === 429 || (err.status >= 500 && err.status < 600) || err.code === 'ECONNRESET';
}

Two things people get wrong here.

The first is retrying non-retryable errors. A 400 means your request is malformed. Retrying it five times with exponential backoff means you wait 15 seconds to fail in exactly the same way. Check the status code.

The second is unbounded retries. Somewhere in your code there needs to be a point where you stop, record the failure, and move on to the next item. A pipeline that retries forever on item 11 never reaches item 12.

Step four: a queue, so concurrency is a number you chose

If you're doing batch work, the naive Promise.all(items.map(process)) fires all 200 requests simultaneously and immediately buries you. You want bounded concurrency, and you want failures recorded rather than thrown away.

async function pooled(items, worker, { concurrency = 3 } = {}) {
  const results = new Array(items.length);
  let cursor = 0;

  async function runner() {
    for (;;) {
      const i = cursor++;
      if (i >= items.length) return;
      try {
        results[i] = { ok: true, value: await worker(items[i], i) };
      } catch (err) {
        results[i] = { ok: false, error: err, item: items[i] };
      }
    }
  }

  await Promise.all(
    Array.from({ length: Math.min(concurrency, items.length) }, runner)
  );
  return results;
}

Note that the worker never rethrows. Every item ends up in the results array as either a success or a labeled failure, so a bad record in the middle doesn't take the run down with it — and you finish with an exact list of what to reprocess.

On a free tier, concurrency: 3 is usually plenty. The buckets are doing the actual pacing; concurrency just controls how many calls can be in flight while you wait.

Putting it together

const requests = new TokenBucket({ capacity: 30, refillPerSecond: 0.5 });
const tokens   = new TokenBucket({ capacity: 6000, refillPerSecond: 100 });

async function callModel(prompt, { maxTokens = 1024 } = {}) {
  await tokens.take(estimateTokens(prompt) + maxTokens);
  await requests.take(1);

  return withRetry(async () => {
    const res = await fetch(ENDPOINT, {
      method: 'POST',
      headers: { 'content-type': 'application/json', ...auth() },
      body: JSON.stringify({ model: MODEL, max_tokens: maxTokens, messages: [{ role: 'user', content: prompt }] }),
    });

    if (!res.ok) {
      const err = new Error(`${res.status} ${res.statusText}`);
      err.status = res.status;
      err.retryAfterMs = parseRetryAfter(res.headers.get('retry-after'));
      if (res.status === 429) {
        tokens.drain();   // we were wrong about how much room we had
        requests.drain();
      }
      throw err;
    }

    return res.json();
  });
}

const results = await pooled(records, (r) => callModel(buildPrompt(r)), { concurrency: 3 });
const failed = results.filter((r) => !r.ok);

That drain() on a 429 matters. A 429 is proof your accounting was optimistic — maybe another process shares the key, maybe your estimator undercounted. Emptying the bucket forces a full refill cycle before you try again, which is the conservative response to being demonstrably wrong.

Step five: failover, and the honest part about keys

Once you have a client that shapes traffic, adding a second provider as a fallback is easy:

async function callWithFailover(prompt, opts) {
  for (const provider of PROVIDERS) {
    if (provider.circuitOpenUntil > Date.now()) continue;
    try {
      return await provider.call(prompt, opts);
    } catch (err) {
      if (err.status === 429 || err.status >= 500) {
        provider.circuitOpenUntil = Date.now() + (err.retryAfterMs ?? 60_000);
        continue;  // try the next one
      }
      throw err;   // a 400 will fail identically everywhere
    }
  }
  throw new Error('All providers exhausted');
}

The circuit breaker is the part worth keeping. Without it, every request pays the full latency of trying the dead provider first. With it, a provider that just told you to go away is skipped entirely until its stated reset time.

Now the part you were maybe hoping I'd cover differently: rotating multiple free-tier keys to multiply your quota is a terms-of-service violation at essentially every provider I've read. Creating several accounts to get around a limit is the specific thing free-tier terms prohibit, and providers do detect it. I'm not going to write you that code.

What is legitimate: separate keys for separate real environments (your dev key and your production key are not a quota-multiplication scheme), and keys belonging to different actual organizations in a multi-tenant product where each customer brings their own. The failover pattern above is the right shape for both — you're just choosing among credentials you're entitled to use.

If you genuinely need more throughput than one free tier provides, the honest options are: use several providers legitimately, batch your work into off-peak windows, cache aggressively so you stop asking the same question, or pay. A paid tier on a cheap model is often less than a month of coffee, and it costs a lot less than an afternoon of debugging a ban.

What this buys you

None of this is clever. It's a bucket, a sleep, a loop, and a fallback list. But the difference between a script that dies at record 11 and a pipeline that grinds through 10,000 records overnight on a free tier is exactly this machinery and nothing else.

The provider's rate limit isn't an obstacle you route around. It's a contract you can read, in headers they send you on every single response. Build the client that reads it, and the wall stops being a wall.