Batch inference

Summarize, classify, or extract information from a dataset without keeping a request open. Batch inference processes Chat Completions asynchronously within a 24-hour window. Unfinished work can expire; use synchronous requests for interactive responses.

PrepareOne request per JSONL line.
SubmitUpload once and save the batch ID.
CollectDownload results and match by ID.

Before you start

Use a key with inference:write and available credits. Choose your deployed model or a catalog model with a non-null batch field in the catalog API. All rows must use the same model revision. Rows may include the images and audio the model accepts; batch does not stream.

Install the OpenAI SDK with pip install openai or npm install openai. Replace your-api-key with your app’s server-side key and your-model-name with your model ID. Steps 1–4 share the same client and variables; use them together in your app’s background job.

1. Prepare requests

This example summarizes two support tickets. Each row needs a unique custom_idand an output-token limit. Keep the IDs so you can join results back to your dataset.

import json
import time
from pathlib import Path
from openai import OpenAI

client = OpenAI(
    base_url="https://api.boardwalk.cloud/v1",
    api_key="your-api-key",
)
model = "your-model-name"
tickets = {
    "ticket-1": "I cannot sign in.",
    "ticket-2": "Please send a copy of my invoice.",
}
requests = [
    {
        "custom_id": ticket_id,
        "method": "POST",
        "url": "/v1/chat/completions",
        "body": {
            "model": model,
            "messages": [{"role": "user", "content": "Summarize: " + text}],
            "max_completion_tokens": 256,
        },
    }
    for ticket_id, text in tickets.items()
]
Path("requests.jsonl").write_text(
    "".join(json.dumps(row) + "\n" for row in requests), encoding="utf-8"
)

2. Upload and submit

Uploading validates the file; submission pins the model revision and returns a batch ID. Save both IDs. Change support-run-1 for each new batch.

with open("requests.jsonl", "rb") as source:
    uploaded = client.files.create(file=source, purpose="batch")
Path("input-file-id.txt").write_text(uploaded.id, encoding="utf-8")

batch = client.batches.create(
    input_file_id=uploaded.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
    extra_headers={"Idempotency-Key": "support-run-1"},
)
Path("batch-id.txt").write_text(batch.id, encoding="utf-8")
print("Batch:", batch.id)

If submission times out, retry only the create call using the saved input file ID and the same idempotency key. Do not upload again for that retry: a different file ID with the same key returns 409. A 402 needs more credit; a 429 can mean the batch is too large or current capacity is full. Check the error before retrying.

3. Wait and download

Poll until the batch is completed, cancelled, failed, or expired. Download both result files when present; either may be empty. You can resume polling using batch-id.txtwithout submitting again.

batch_id = Path("batch-id.txt").read_text(encoding="utf-8").strip()
terminal = {"completed", "cancelled", "failed", "expired"}
while True:
    finished = client.batches.retrieve(batch_id)
    print(finished.status, finished.request_counts)
    if finished.status in terminal:
        break
    time.sleep(30)

for file_id, filename in [
    (finished.output_file_id, "results.jsonl"),
    (finished.error_file_id, "errors.jsonl"),
]:
    text = client.files.content(file_id).text if file_id else ""
    Path(filename).write_text(text, encoding="utf-8")

4. Match results and review failures

Match by custom_id, not line order. A completed batch can contain failed requests. Check errors.jsonl and compare with your original IDs: work that never started may have no result row. Also check each answer’s finish reason before using it.

successful = set()
for line in Path("results.jsonl").read_text(encoding="utf-8").splitlines():
    row = json.loads(line)
    response = row.get("response")
    if row.get("error") or not response or response["status_code"] != 200:
        continue
    successful.add(row["custom_id"])
    answer = response["body"]["choices"][0]["message"]["content"]
    print(row["custom_id"], tickets.get(row["custom_id"]), answer)

# Review errors and missing results before submitting this file as a new batch.
remaining = [row for row in requests if row["custom_id"] not in successful]
Path("retry.jsonl").write_text(
    "".join(json.dumps(row) + "\n" for row in remaining), encoding="utf-8"
)
print("Items to review:", len(remaining))

Correct invalid requests before retrying. Submit only the remaining items with a new idempotency key; skip submission when retry.jsonl is empty. Retried work can incur new charges. See result fields and statuses.

Cancel a batch

client.batches.cancel(batch_id)

Cancellation stops unfinished work. Continue polling and download any completed results.

Costs, limits, and retention

Model	What you pay
Catalog	Published batch token rates, fixed at acceptance. Only successful results published for download are charged.
Your deployment	Normal GPU time, including work before cancellation or failure. There is no separate token charge.

Queue time has no GPU charge. Inputs are limited to 256 MiB and 1,000 requests; private batches currently accept at most 95 requests. Capacity, output size, and credit can reduce the accepted size.

Unlike synchronous inference, batch files use private, encrypted storage. Unused input expires after 24 hours; accepted input is removed after completion and expires within 48 hours of acceptance. Download results within seven days of completion. See all limits and deletion options.