Latency & cold starts

A cold start on Boardwalk is a swap from host memory onto a free card, not a boot. It takes seconds at any model size.

Cold starts

A deploy leaves the model warm, and a model called in the last 30 minutes stays warm on at least two hosts. Warmth is free and is not a setting. The console shows every cold start.

Stay hot

• Hold. A model keeps its card for 1 to 300 seconds after its last request (default 5). A longer hold covers gaps between requests. It bills as GPU time.
• Reserved capacity. Reserve GPUs for your organization to share across its models. Reserved GPUs remain allocated even while idle.
• Conversation key. Send prompt_cache_key so each turn lands on the card holding its cache.

client.chat.completions.create(
    model="support-agent",
    messages=messages,
    prompt_cache_key=session_id,
)

Client timeouts

Use streaming for interactive applications. A request waits up to three minutes for a card before it is refused (see model capacity), so set your client timeout above three minutes when your workflow can wait.

client = OpenAI(base_url=BASE_URL, api_key=KEY, timeout=240)

Time to first token

Time to first token is the wait for a card, any swap, and prompt processing. A long prompt whose prefix is cached skips most of the processing. After the first token, generation speed is your engine’s, set by your engine settings.

Long context

A deployment serves its model’s whole window by default. The max context covers the prompt and the output together, including reasoning. Use tokenize to count a message list with the served chat template.

Measure your workload

Each receipt records the wait for a card, the cold start, first-token latency, and total latency. The console charts first-token p50 to p99, waits, and cold starts per model.

{
  "ttft_ms": 412,
  "latency_ms": 2380,
  "gpu": {
    "wait_ms": 0,
    "interval": { "cold_start_ms": 310, "serving_ms": 2070, "hold_ms": 5000 }
  }
}