Rate limits & retries

Inference has no requests-per-minute limit, no tokens-per-minute limit, and no usage tiers. A request waits for a GPU card rather than being refused, within the limits below. Each refusal carries a code that names it, and most carry a Retry-After header.

Your balance and concurrency

On a catalog model, each request reserves the most it could cost until it finishes. A new request starts when its reservation fits beside the ones already running, so the more credit you hold, the more requests you can run at once. One request always starts while your balance is positive.

A reservation is the model's price for:

Output: your max_tokens (or max_completion_tokens, or max_output_tokens). Without one, the model's whole context length.
Prompt: at most one token per byte of the request body. With images, up to the whole context.

Prompt and output together never count for more than the context.

When you send too many at once, the request is refused with 429 balance_concurrency_exceeded. Nothing is billed. Retry when a request finishes, or add credits. Requests to your own deployments carry no per-token price, so they reserve nothing; any positive balance admits them.

Model capacity

A request waits at the gateway for a GPU card instead of being refused. It takes the first capacity that appears: a card already serving the model with room, a free card on a host that holds the model’s copy, or a card freed when another hold ends. Waiting requests are served in order of arrival. A caller that disconnects while waiting leaves the line, and waiting is never billed.

A wait is bounded:

A request to your deployment that waits three minutes ends with 503 capacity_wait_exceeded and a Retry-After header.
A request to a catalog model that waits three minutes ends with 429 model_at_capacity and a Retry-After header.
Your organization may hold a limited number of cards at once (8 by default), and have four times that many requests waiting. Past either, a request gets 429 card_limit_reached at once. We raise the card limit on request.
If the cards a request is sent to turn it away several times in a row, it gets 429 model_at_capacity.

Set your client timeout above three minutes for requests that can wait for a card.

Status codes

Branch on code. Errors follow the error shape of the API you call.

Status	Code	Meaning
402	`insufficient_credits`	Your balance is zero. Add credits.
403	`org_inactive`	Your organization is suspended. Contact us.
429	`balance_concurrency_exceeded`	Running requests have reserved your balance. Retry when one finishes, or add credits.
429	`model_at_capacity`	No card became free for a catalog model within three minutes, or the cards tried turned the request away. Retry after the delay.
429	`card_limit_reached`	Your organization holds, or is waiting for, as many cards as its limit allows. Retry shortly, or ask us to raise the limit.
429	`rate_limit_exceeded`	Too many requests a minute, or from one IP address. Retry after the delay.
502	`upstream_error`	The engine failed. Retry with backoff.
503	`model_unavailable`	The model couldn’t be reached. Retry with backoff.
503	`capacity_wait_exceeded`	No card became free for your deployment within three minutes. Retry after the delay.

Failed requests are never billed, so retrying is always safe.

Retrying

Wait at least Retry-After seconds, add a little random delay so your clients do not retry in step, and stop at a deadline you choose. The OpenAI and Anthropic SDKs already retry 429 and 5xx twice and honor Retry-After, which covers a short refusal. Retry in your own code for workloads that can wait longer:

import os
import random
import time

from openai import APIStatusError, OpenAI

# Retry here, not in the SDK, so a busy model gets more than the SDK's two attempts.
client = OpenAI(
    base_url="https://api.boardwalk.cloud/v1",
    api_key=os.environ["BOARDWALK_API_KEY"],
    max_retries=0,
)


def create(deadline_s: float = 300, **params):
    give_up_at = time.monotonic() + deadline_s
    attempt = 0
    while True:
        try:
            return client.chat.completions.create(**params)
        except APIStatusError as error:
            # A failed request is never billed; 4xx other than 429 needs a change first.
            if error.status_code not in (429, 502, 503) or time.monotonic() >= give_up_at:
                raise
            wait = float(error.response.headers.get("retry-after", "1"))
            time.sleep(wait + random.uniform(0, min(2**attempt, 8)))
            attempt += 1

There are no idempotency keys. Each retry is a new request with its own x-request-id, and a receipt once it completes.

Streams

A refusal usually arrives as an HTTP status, before any stream starts. A request waiting for a card sends nothing, so its refusal keeps its status. Once a request is admitted, a stream that has sent nothing for ten seconds is opened with a keep-alive comment, so a failure after that arrives as an error event with the same code, before any content. Retry it the same way.

An error event after content has arrived means the completion is incomplete. It is recorded at zero cost like any failure, but your application may already have acted on part of it. Do not replay the stream automatically, and do not run a tool call from it twice.

Other endpoints

Tokenization and management reads allow at least 60 requests a minute per API key. Management reads also count all of an organization’s keys together at the same rate, with a separate budget from inference. These responses include:

x-ratelimit-limit: requests allowed per minute.
x-ratelimit-remaining: requests left before you are refused.
x-ratelimit-reset: seconds until another request is allowed. This is not a timestamp.

Inference responses do not include these headers, because inference has no rate budget.

Running more at once

On a catalog model, hold more credit, or set max_tokens so each request reserves less. On your own deployments, ask us to raise your organization’s card limit, or reserve capacity to keep GPUs allocated to your organization.