Serving settings

Review these settings when deploying or redeploying a model. Start with the proposed configuration, then change the controls that affect your workload. A revision pins both your weights and settings.

Choose a configuration

Control	When to change it
Engine and version	Choose vLLM or SGLang for the model’s supported features. An omitted version selects and pins the newest qualified release.
GPU and replica size	Review memory and price. A replica uses 1, 2, 4, or 8 GPUs; available classes and slices depend on capacity.
Max context	Sets the combined prompt and output limit, including reasoning. Defaults to the model’s full window. A smaller window can reduce memory needs.
KV cache precision	Use FP8 to fit more context when supported, or keep the checkpoint’s precision. This does not change your model’s weight precision.
Idle hold	How long GPUs remain allocated after the last request: 1–300 seconds, default 5. A longer hold can reduce repeated cold starts and costs more.

If the model and one full-context request do not fit, deployment is refused with an explanation. Review the suggested context limit, GPU size, or a quantized checkpoint. Boardwalk does not silently reduce context or quantize your weights.

Control scaling

Replicas scale with traffic within your organization’s GPU and spending limits, and release their GPUs after the idle hold. Optional concurrency controls apply per replica:

Setting	Behavior
`target_concurrency`	Guides scale-up. Requests can exceed the target while more capacity becomes available.
`max_concurrency`	Hard admission limit. Excess requests wait for capacity or receive an error when the wait expires.

Counts include preparation and streaming. Engine memory and batch limits can restrict concurrency further. Configure these values in serving.autoscaling through the deployment API.

Reserve capacity

Reserved GPUs remain allocated to your organization and are billed while idle. They are shared across your models; a reservation does not keep every model loaded. Traffic beyond the reservation uses on-demand capacity when available.

Review reservation pricing before changing capacity. For timeouts and caching, see latency and cold starts.

Advanced engine settings

Use these when you need a specific engine feature. Settings are checked during deployment; unsupported values fail with a reason. Compare configurations by calling each pinned revision before promoting it.

Engine flags, extended context, and speculative decoding

Enter flags for your selected engine release, one name=value per line without leading dashes. For example:

max-num-seqs=64
max-num-batched-tokens=8192
enable-prefix-caching=true

Flags control scheduling, caching, kernels, parallelism, and parsers. Boardwalk rejects flags that conflict with its managed controls or require an unsupported layout. Engine defaults apply, with a 2,048-token prefill chunk.

YaRN extends context beyond the shipped window when supported by the model; follow the publisher’s settings. Speculative decoding uses a configured method and, when needed, a draft-model repository. Both are off by default.