AI integration16 min read

OpenAI API and Batch API: choosing the right execution path

A production-oriented guide to synchronous Responses API calls and asynchronous Batch API workloads, including boundaries, JSONL input, correlation, partial failure and operational handling.

Begin with the product latency requirement

An interactive request normally needs a response while the user is waiting. The Responses API supports that direct path and can return text or structured output from text, image or file input, with tools where the application needs them.

Batch is for work that does not require an immediate answer: offline classification, enrichment, evaluation or scheduled processing. It exchanges latency for asynchronous throughput and discounted processing rather than acting as a faster interactive endpoint.

Keep model selection in configurationtypescript / OpenAI

import OpenAI from "openai"

const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY })

const response = await client.responses.create({
  model: process.env.OPENAI_MODEL!,
  instructions: "Return a concise technical summary.",
  input: sourceText,
})

const summary = response.output_text

Put the API behind a server-owned boundary

Never expose the API key through browser code or public Nuxt runtime configuration. A server route should authenticate the caller, validate input, apply request limits and translate provider responses into the application contract.

Store stable identifiers for users and jobs rather than sending unnecessary personal data. Log provider request identifiers and local correlation IDs, but avoid logging full prompts and outputs by default when they may contain customer data.

A batch begins as JSONL

Each line in the input file is one request with a unique custom_id, POST method, supported relative URL and request body. The file is uploaded with purpose batch, then its file ID is used to create the batch.

The official API currently supports a 24-hour completion window. A batch input file can contain up to 50,000 requests and be up to 200 MB; account and model queue limits still apply. Validate the JSONL locally before upload because one formatting mistake can invalidate work.

Two independent Responses API requestsjsonl

{"custom_id":"account-17","method":"POST","url":"/v1/responses","body":{"model":"YOUR_MODEL","input":"Summarise account 17"}}
{"custom_id":"account-29","method":"POST","url":"/v1/responses","body":{"model":"YOUR_MODEL","input":"Summarise account 29"}}

Upload, create and observe the job

Upload the input file, create a batch for the same endpoint used by every line, then persist the returned batch ID. Polling can work for a small internal process; a production workflow should have explicit job state, bounded retry and a recovery path when the worker restarts.

Statuses include validation, in-progress, finalising and terminal outcomes such as completed, failed, expired or cancelled. A completed batch can still contain individual failed requests, so completion is not equivalent to every item succeeding.

Create the asynchronous batchshell / HTTP

curl https://api.openai.com/v1/files \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -F purpose="batch" \
  -F file="@requests.jsonl"

curl https://api.openai.com/v1/batches \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"input_file_id\":\"file_INPUT\",\"endpoint\":\"/v1/responses\",\"completion_window\":\"24h\"}"

Correlate by custom_id, never by line order

Output order may differ from input order. Join every result to application state using custom_id and make that identifier unique within the batch. Keep it opaque enough that exported output files do not disclose sensitive domain information.

Successful results and request errors are available through output and error files. Record each item independently so a retry batch contains only failed or expired work. Idempotent result handling prevents a restarted importer from applying the same enrichment twice.

Index output by the application identifiertypescript

type BatchLine = {
  custom_id: string
  response?: { status_code: number; body: unknown }
  error?: { code: string; message: string }
}

for (const line of outputLines as BatchLine[]) {
  if (line.response?.status_code === 200) {
    await results.saveOnce(line.custom_id, line.response.body)
  } else {
    await failures.record(line.custom_id, line.error ?? line.response)
  }
}

Production quality lives around the model call

Set timeouts and retry only transient failures for direct requests. Validate structured results before they enter the domain, monitor token usage and latency, and evaluate behaviour against representative examples whenever prompts or models change.

For batches, monitor queued volume, age, item failure rate and import completion. Keep prompts and model configuration versioned with the job so an output can be explained after the application has moved on.