Model Compare

Stop guessing which model is best for your use case — race them on your actual prompt:

POST /api/v1/compare

{
  "messages": [{"role": "user", "content": "Explain quantum entanglement in one paragraph"}],
  "models": ["gpt-5", "claude-sonnet-4.6", "gemini-3.1-pro", "deepseek-v3"],
  "rank_by": "balanced"
}

The response is a ranked leaderboard:

{
  "winner": {
    "model": "claude-sonnet-4.6",
    "name": "Claude Sonnet 4.6",
    "reason": "Best balance of quality (0.97), speed (1180ms), and cost ($0.0091)"
  },
  "results": [
    {"rank": 1, "model": "claude-sonnet-4.6", "content": "...", "quality": {"confidence": 0.97}, "latency_ms": 1180, "cost_cents": 0.91},
    {"rank": 2, "model": "gpt-5",             "content": "...", "quality": {"confidence": 0.95}, "latency_ms": 920,  "cost_cents": 0.69},
    {"rank": 3, "model": "gemini-3.1-pro",    "content": "...", "quality": {"confidence": 0.93}, "latency_ms": 1050, "cost_cents": 0.67},
    {"rank": 4, "model": "deepseek-v3",       "content": "...", "quality": {"confidence": 0.89}, "latency_ms": 780,  "cost_cents": 0.17}
  ],
  "summary": {
    "fastest":         {"model": "deepseek-v3", "latency_ms": 780},
    "highest_quality": {"model": "claude-sonnet-4.6", "confidence": 0.97},
    "cheapest":        {"model": "deepseek-v3", "cost_cents": 0.17},
    "best_value":      {"model": "gpt-5"}
  },
  "total_cost": "$0.0244"
}

cost_cents on each result is that model’s real metered charge for its actual response length in this race — not a fixed per-model number, so re-running the same compare can shift these slightly even with the same models.

Full response shape

Only the top-level winner object carries a display name (resolved from the model catalog) — individual entries in results[] and failed[] identify the model by its model slug only. Each result also carries tokens (prompt / completion / total), success, and quality.flags / quality.suggested_retry. Top level adds failed[], ranked_by, models_compared, succeeded, total_cost_cents, balance, and compared_at.

Parameters

Parameter	Default	Description
`messages`	required	1–20 messages of `{role, content}`. Unlike `/chat`, `content` here is plain text only (no vision content-parts) and capped at 50,000 chars per message instead of 100,000.
`models`	top 5 across tiers	2–8 model IDs. No `auto` or `ensemble`.
`rank_by`	`balanced`	`quality`, `speed`, `cost`, or `balanced` (50% quality + 30% speed + 20% cost).
`max_tokens`	`1024`	Per-model response cap (1–8,192).
`temperature`	`0.7`	0 = deterministic, 2 = creative. Applied to every model in the race.
`include_full_responses`	`true`	`false` returns 200-char previews — cheaper to read, same billing.
`stream`	`false`	Stream results over SSE as each model finishes instead of waiting for all of them.

Billing: the sum of each successful model’s real metered token usage — a 4-model compare here cost $0.0244. Failed models aren’t charged, and the whole request is rejected up front if your balance can’t cover the estimated maximum.

The workflow it’s built for

Race on real prompts

Take 5 prompts from your actual workload, run them through /compare with the models you’re considering.

Read the summary, not the vibes

summary.best_value is the quality-per-dollar pick; winner is the balanced pick.

Commit and monitor

Hard-code the winner. Re-run the same compare monthly — model updates shift results.

r = requests.post("https://www.ninjachat.ai/api/v1/compare",
    headers={"Authorization": f"Bearer {os.environ['NINJACHAT_API_KEY']}"},
    json={
        "messages": [{"role": "user", "content": "Write a haiku about machine learning"}],
        "models": ["gpt-5", "claude-sonnet-4.6", "gemini-3.1-pro"],
        "rank_by": "quality",
    }
)
data = r.json()
print(f"Winner: {data['winner']['model']} — {data['winner']['reason']}")
for res in data["results"]:
    print(f"  #{res['rank']} {res['model']}: {res['quality']['confidence']:.2f} quality, {res['latency_ms']}ms")

The same idea exists for images — agents race image models with compare_models on the MCP server.

Get Started

Chat

Superpowers

Media & Search

Reference

Parameters

The workflow it’s built for

​Parameters

​The workflow it’s built for

Parameters

The workflow it’s built for