Batch inference

Run Chat Completions asynchronously on a catalog model or your own deployment. Upload a JSONL file, submit a batch, then download its results. Work uses spare capacity, with a 24-hour completion window; unfinished requests can expire. Batch work runs separately from interactive traffic.

Catalog batches for the same serving revision can share an engine across organizations. Private batches share only within your deployment. Files, results, and cancellation remain separate for each batch.

Use an organization API key with inference:write and available credit. A key with either inference:write or requests:read can read that organization’s batches and files.

For a complete Python or TypeScript workflow, follow the batch inference guide.

Prepare the input

Save one request per line in requests.jsonl. Every line must target the same model revision. Use your deployment’s name, or a catalog model whose catalog entry has a non-null batchfield.

{"custom_id":"task-1","method":"POST","url":"/v1/chat/completions","body":{"model":"support-agent","messages":[{"role":"user","content":"Summarize this support request: I cannot sign in."}],"max_completion_tokens":256}}

Each custom_id must be unique and at most 128 UTF-8 bytes. Include max_tokens or max_completion_tokens. Requests cannot stream. Images (inline or https URLs, fetched when the row runs) and audio (inline) are accepted where the model publishes them; a catalog model accepts them in batch once its batch qualification covers media. Other fields follow the Chat Completions reference.

BatchRouter sectioned JSONL (batchrouter.sectioned.v1) is also accepted. Sections supply shared parameters and prefix messages; request fields override shared parameters. Results retain each request’s exact ID and use the standard result format.

Upload and submit

Send multipart fields purpose=batch and file to POST /v1/files. Uploading validates the whole file before returning a file object; it does not start inference. These shell examples use jq.

BASE_URL=https://api.boardwalk.cloud/v1
FILE_ID=$(curl --fail --silent --show-error "$BASE_URL/files" \
  -H "Authorization: Bearer $BOARDWALK_API_KEY" \
  -F purpose=batch -F file=@requests.jsonl | jq -r .id)

curl --fail --silent --show-error "$BASE_URL/batches" \
  -H "Authorization: Bearer $BOARDWALK_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: support-run-1" \
  --data-binary @- <<JSON
{
  "input_file_id": "$FILE_ID",
  "endpoint": "/v1/chat/completions",
  "completion_window": "24h"
}
JSON

Idempotency-Keyheader · stringrequired

1–256 bytes. Required for batch creation; scoped to the organization.

input_file_idstringrequired

ID from a validated input upload in the same organization.

endpoint"/v1/chat/completions"required

The only supported batch endpoint.

completion_window"24h"required

The only supported completion window.

metadataobject

Optional BatchRouter Standard assignment fields, described below.

Unknown fields are rejected. Upload and creation return 200 on success.

BatchRouter metadata accepts its work-unit ID, RFC 3339 deadline, assignment window, standard tier and role, optional payment rates, and batchrouter_input_format: "sectioned.v1". The deadline is the earlier of the supplied deadline and assignment window, with a 24-hour maximum. Payment rates must match the catalog revision’s published batch prices; metadata cannot override prices. Other metadata, tiers and roles are unsupported. See the BatchRouter field definitions.

Save the returned id as BATCH_ID. Reusing the idempotency key with the same submission returns the original batch; changing the submission returns 409. Use a new key for a new batch. Acceptance pins the model revision and checks available credit; insufficient credit returns 402, and capacity or queue limits return 429.

Read the results

curl "$BASE_URL/batches/$BATCH_ID" \
  -H "Authorization: Bearer $BOARDWALK_API_KEY"

status is in_progress while waiting or running, then may become finalizing or cancelling. Stop polling at completed, cancelled, failed, or expired. Check request_counts for total, completed, and failed items. Batch timestamps use Unix seconds.

idstring

Batch ID. Save it to resume polling.

object"batch"

Resource type.

input_file_idstring

Validated input file ID.

endpoint"/v1/chat/completions"

Endpoint for every item.

completion_window"24h"

Maximum window; expires_at is the effective deadline.

statusstring

in_progress (queued or running), finalizing, cancelling, completed, cancelled, failed, or expired. completed does not mean every item succeeded.

request_countsobject

Integer total, completed, and failed counts. Inspect item results as well as batch status.

output_file_id / error_file_idstring | null

Published result file IDs, hidden until terminal. Either file can be empty; IDs can be null when publication fails.

model_config_idstring

Model revision pinned at acceptance.

billing"catalog" | "gpu_time"

Catalog token billing or private deployment GPU-time billing.

pricingobject | null

Null for GPU-time billing. Catalog fields: unit is microusd_per_million_tokens; input, cached_input, and output are integer decimal strings fixed at acceptance.

created_at / expires_atinteger

Acceptance time and completion deadline, in Unix seconds.

completed_at / cancelled_at / failed_at / expired_atinteger | null

Unix seconds for the matching terminal state; other timestamps are null.

errorsobject | null

Batch-level failure: {object: "list", data: [{code, message}]}. Per-item errors are in the error file.

metadataobject | null

Accepted BatchRouter assignment fields, or null for ordinary batches.

Terminal batches expose output_file_id and error_file_id. Download both: one contains successful results, the other contains item failures. Match rows using custom_id. A row includes response.status_code, response.request_id and response.body, or an error.

curl "$BASE_URL/files/$OUTPUT_FILE_ID/content" \
  -H "Authorization: Bearer $BOARDWALK_API_KEY" \
  --output results.jsonl

A row also includes its id. Successful rows have error: null; failures can have response: null. Work that never started may have no row, so reconcile the original IDs with the downloaded results. Files are stable across repeated downloads. Use a row’s response.request_id to retrieve its receipt.

Manage batches and files

There is no file-list endpoint. Save file IDs from uploads and batch responses. File metadata has the following fields:

idstring

File ID used by batch and download endpoints.

object"file"

Resource type.

bytesinteger

File size in bytes.

filenamestring

Server-assigned filename; the original upload filename is not retained.

purpose"batch" | "batch_output"

batch for input; batch_output for result and error files.

status"processed" | "uploaded" | "deleted"

Successfully validated uploads return processed.

created_at / expires_atinteger

Creation and content expiry times in Unix seconds.

Download content

GET /v1/files/{id}/content returns JSONL, with Content-Type: application/jsonl, ETag, and Accept-Ranges: bytes. To resume a download, send one Range: bytes=OFFSET- and the saved ETag as If-Range. A matching range returns 206; an ETag mismatch returns the full file with 200, so replace rather than append in that case. Delete returns {id, object: "file", deleted: true}.

Billing

• Your deployments: normal GPU-time billing, including work before cancellation or failure. Shared GPU time is charged once when your batches overlap. No token charge. The batch returns billing: gpu_timeand pricing: null.
• Catalog models: separate batch token rates, fixed at acceptance and returned in pricing. Only successful results durably published for download are charged. The batch returns billing: catalog.

Waiting in the batch queue has no GPU charge. Downloading results does not add another inference charge.

Limits and retention

• Input: 256 MiB, 1,000 items, and 20 MiB per line. Private batches currently accept at most 95 items; backlog, output size, and credit can lower the accepted size further.
• Output and error files: at most 20 MiB each.
• Per organization: four unfinished batches and 1 GiB retained input.

Batch uploads explicitly opt into private, encrypted content storage. Unused input expires after 24 hours; accepted input is removed after completion and expires within 48 hours of acceptance. Terminal results expire after seven days. Deleting a result file also deletes its recovery copies. Content never enters request history, billing records, logs, or analytics. See privacy and access.

Errors