Batch

Total latency = slowest single request. Batch 20 prompts, wait as long as 1.

POST /api/v1/batch

{
  "requests": [
    {"model": "gpt-5", "messages": [{"role": "user", "content": "Summarize: React hooks enable state..."}]},
    {"model": "deepseek-v3", "messages": [{"role": "user", "content": "Fibonacci in Rust"}]},
    {"model": "auto", "messages": [{"role": "user", "content": "Explain gradient descent"}]}
  ]
}

Response

{
  "results": [
    {"index": 0, "success": true, "model": "gpt-5",       "content": "React hooks provide...", "cost_cents": 0.42, "latency_ms": 950},
    {"index": 1, "success": true, "model": "deepseek-v3",  "content": "fn fibonacci(n: u64)...", "cost_cents": 0.11, "latency_ms": 680},
    {"index": 2, "success": true, "model": "o3-mini", "requested_model": "auto", "content": "Gradient descent minimizes...", "cost_cents": 0.19, "latency_ms": 1400}
  ],
  "succeeded": 3,
  "failed": 0,
  "total_cost": "$0.0072",
  "metadata": {"total_latency_ms": 1420, "batch_size": 3, "parallelism": 3}
}

cost_cents on each result is that item’s real metered charge for its actual token usage — not a fixed per-model number, so the same prompt on the same model can bill slightly differently between runs. Results come back in request order, tagged by index. Each also carries tokens and (for auto) a routing object.

Parameters

Parameter	Default	Description
`requests`	required	1–20 request objects — each takes `model`, `messages` (1–30), `temperature`, `max_tokens` (≤8,192). `auto*` works per-request.
`fail_on_any_error`	`false`	`true` = all-or-nothing; default returns partial results.
`stream`	`false`	Return results as they complete over SSE instead of waiting for the whole batch — see Streaming results below.

Billing: each successful request bills its exact metered token usage at that model’s $/MTok rate; failures aren’t charged. An estimated maximum for the whole batch is held against your balance up front and the unused portion refunds once every item completes.

Fallback chains ("a>b>c") aren’t valid model values here and are rejected up front with a validation_error. Ensemble models (ensemble, ensemble-quality) do pass validation — they’re valid model IDs elsewhere in the API — but batch has no ensemble execution path, so each one fails at runtime as a per-item error (success: false) rather than a clean validation rejection. You aren’t charged for it, but don’t rely on ensembles working inside /batch.

Streaming results

Set stream: true to get each result the moment it finishes instead of waiting for the slowest one — useful when a couple of the 20 prompts are much slower than the rest:

POST /api/v1/batch (stream)

{
  "requests": [
    {"model": "gpt-5", "messages": [{"role": "user", "content": "Summarize: React hooks enable state..."}]},
    {"model": "deepseek-v3", "messages": [{"role": "user", "content": "Fibonacci in Rust"}]}
  ],
  "stream": true
}

The response is text/event-stream: one type: "result" event per completed request (in whatever order they finish, tagged by index), followed by one final type: "summary" event and a [DONE] sentinel:

data: {"type":"result","index":1,"success":true,"model":"deepseek-v3","content":"fn fibonacci(n: u64)...","cost_cents":0.11,"latency_ms":680,"tokens":{...}}

data: {"type":"result","index":0,"success":true,"model":"gpt-5","content":"React hooks provide...","cost_cents":0.42,"latency_ms":950,"tokens":{...}}

data: {"type":"summary","succeeded":2,"failed":0,"total_cost_cents":0.53,"total_cost":"$0.0053","request_id":"..."}

data: [DONE]

Billing works the same as non-streaming: the full estimate is held up front and the unused portion is refunded once the summary event fires.

The classic use case — classification at scale

import requests, os

texts = ["I love this product!", "Terrible experience.", "It's okay.", "10/10!", "Waste of money."]

r = requests.post("https://www.ninjachat.ai/api/v1/batch",
    headers={"Authorization": f"Bearer {os.environ['NINJACHAT_API_KEY']}"},
    json={"requests": [
        {
            "model": "gpt-5-mini",
            "messages": [
                {"role": "system", "content": "Classify as POSITIVE, NEGATIVE, or NEUTRAL. One word."},
                {"role": "user", "content": text},
            ],
            "max_tokens": 5,
        }
        for text in texts
    ]}
)
data = r.json()
for text, result in zip(texts, data["results"]):
    if result["success"]:
        print(f"{text!r:30} → {result['content'].strip()}")
print(f"Total: {data['total_cost']} in {data['metadata']['total_latency_ms']}ms")

Handle partial failures per item:

for result in data["results"]:
    if not result["success"]:
        print(f"#{result['index']} failed: {result['error']}")

Limits

Limit	Value
Requests per batch	20
Messages per request	30
Content per message	50,000 chars
Tokens per request	8,192

Batching identical work at the lowest price? Pair with auto-cheap per request, or estimate the total first — estimates are free.

Get Started

Chat

Superpowers

Media & Search

Reference

Parameters

Streaming results

The classic use case — classification at scale

Limits

​Parameters

​Streaming results

​The classic use case — classification at scale

​Limits

Parameters

Streaming results

The classic use case — classification at scale

Limits