Call LLMs Through the Gateway

The Core Proxy exposes an OpenAI-compatible LLM surface plus a native Anthropic Messages API. Existing SDKs work unmodified — you change the base URL and the API key.

OpenAI-compatible base URL	http://localhost:8100/v1/proxy/llm
Anthropic native	http://localhost:8100/v1 → POST /v1/messages (no /proxy/ prefix)
Auth	Authorization: Bearer sk_brutor_api_... or X-API-Key: sk_brutor_api_... — or a portal JWT

Full request/response schemas live in the LLM Proxy API reference. This page is the working guide.

Endpoints at a glance

Endpoint	Path
Chat Completions	POST /v1/proxy/llm/chat/completions (SSE streaming with "stream": true)
Responses API	POST /v1/proxy/llm/responses
Legacy Completions	POST /v1/proxy/llm/completions
Embeddings	POST /v1/proxy/llm/embeddings
Models	GET /v1/proxy/llm/models, GET /v1/proxy/llm/models/{model_id}
Images	POST /v1/proxy/llm/images/generations
Audio TTS	POST /v1/proxy/llm/audio/speech
Audio STT	POST /v1/proxy/llm/audio/transcriptions
Video (async)	POST /v1/proxy/llm/video/generations → GET /v1/proxy/llm/video/jobs/{job_id}
Anthropic Messages	POST /v1/messages (+ POST /v1/messages/count_tokens)

Addressing a model: model vs model_id

Request bodies accept either field:

• model — the provider model name as configured in the gateway (e.g. "gpt-4o", "claude-haiku-4-5-20251001"), or the name of a routing group’s virtual model.
• model_id — the Brutor model UUID (as returned by GET /v1/proxy/llm/models or GET /v1/portal/models). Unambiguous when two configured models share a provider name — this is what the shipped User Portal sends.

Which models a credential can address is determined by its resource group. GET /v1/proxy/llm/models returns exactly the models your key or JWT may use:

curl -s http://localhost:8100/v1/proxy/llm/models \
  -H "Authorization: Bearer sk_brutor_api_..."

Chat completions

curl

curl -s http://localhost:8100/v1/proxy/llm/chat/completions \
  -H "Authorization: Bearer sk_brutor_api_..." \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user", "content": "What is an AI gateway?"}
    ],
    "temperature": 0.7,
    "max_tokens": 200
  }'

Python

from openai import OpenAI

client = OpenAI(
    api_key="sk_brutor_api_...",
    base_url="http://localhost:8100/v1/proxy/llm",
)

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": "You are a concise assistant."},
        {"role": "user", "content": "What is an AI gateway?"},
    ],
    temperature=0.7,
    max_tokens=200,
)
print(response.choices[0].message.content)
print(response.usage.total_tokens, "tokens")

Node.js

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: "sk_brutor_api_...",
  baseURL: "http://localhost:8100/v1/proxy/llm",
});

const response = await client.chat.completions.create({
  model: "gpt-4o",
  messages: [
    { role: "system", content: "You are a concise assistant." },
    { role: "user", content: "What is an AI gateway?" },
  ],
  temperature: 0.7,
  max_tokens: 200,
});
console.log(response.choices[0].message.content);

Anthropic SDK

import anthropic

# The Anthropic SDK hits the native /v1/messages endpoint —
# base_url is the gateway root, NOT /v1/proxy/llm.
client = anthropic.Anthropic(
    api_key="sk_brutor_api_...",          # sent as x-api-key — the gateway accepts it
    base_url="http://localhost:8100",
)

message = client.messages.create(
    model="claude-haiku-4-5-20251001",
    max_tokens=200,
    messages=[{"role": "user", "content": "What is an AI gateway?"}],
)
print(message.content[0].text)

Response (standard OpenAI shape):

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "created": 1751450000,
  "model": "gpt-4o",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "An AI gateway is a proxy layer that sits between your applications and AI providers..."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 24,
    "completion_tokens": 87,
    "total_tokens": 111
  }
}

Success moment: the request now appears in Mission Control → Usage with the caller, model, token counts, and cost attributed to the API key’s resource group.

Streaming (SSE)

Set "stream": true. The gateway relays Server-Sent Events — data: frames with OpenAI chunk objects, terminated by data: [DONE]:

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","choices":[{"delta":{"role":"assistant"},"index":0}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","choices":[{"delta":{"content":"An AI"},"index":0}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","choices":[{"delta":{"content":" gateway"},"index":0}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","choices":[{"delta":{},"index":0,"finish_reason":"stop"}]}

data: [DONE]

curl

curl -sN http://localhost:8100/v1/proxy/llm/chat/completions \
  -H "Authorization: Bearer sk_brutor_api_..." \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-4o", "messages": [{"role": "user", "content": "Count to five"}], "stream": true}'

Python

stream = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Count to five"}],
    stream=True,
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

Node.js

const stream = await client.chat.completions.create({
  model: "gpt-4o",
  messages: [{ role: "user", content: "Count to five" }],
  stream: true,
});
for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}

Output guardrails on streams

When a resource group has block- or redact-mode output guardrails, the gateway may buffer-and-release the stream (redaction) or interrupt it mid-flight (block), depending on the guardrail config’s streaming mode. Your SSE parser should tolerate a stream that ends without a finish_reason: "stop" chunk. See Guardrails.

Tool calling

Tool definitions and tool_calls responses pass through unchanged — the OpenAI wire format works end to end. The gateway inspects them (argument policies can deny a generated tool call, e.g. “no SQL writes”), logs them, and passes them on:

{
  "model": "gpt-4o",
  "messages": [{"role": "user", "content": "What's the weather in Berlin?"}],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_current_weather",
        "description": "Get current weather for a city",
        "parameters": {
          "type": "object",
          "properties": {"city": {"type": "string"}},
          "required": ["city"]
        }
      }
    }
  ],
  "tool_choice": "auto"
}

The assistant’s tool_calls come back exactly as the provider produced them; you execute the tool (ideally through the MCP proxy) and send the role: "tool" result message back. See Build an agent for the full loop.

Embeddings

curl -s http://localhost:8100/v1/proxy/llm/embeddings \
  -H "Authorization: Bearer sk_brutor_api_..." \
  -H "Content-Type: application/json" \
  -d '{"model": "text-embedding-3-small", "input": "AI governance for agents"}'

{
  "object": "list",
  "data": [
    {"object": "embedding", "index": 0, "embedding": [0.0123, -0.0456, "..."]}
  ],
  "model": "text-embedding-3-small",
  "usage": {"prompt_tokens": 5, "total_tokens": 5}
}

input accepts a string or an array of strings; encoding_format and dimensions are supported where the provider supports them.

Images

curl -s http://localhost:8100/v1/proxy/llm/images/generations \
  -H "Authorization: Bearer sk_brutor_api_..." \
  -H "Content-Type: application/json" \
  -d '{
    "model": "dall-e-3",
    "prompt": "A friendly robot guarding a gateway, watercolor",
    "n": 1,
    "size": "1024x1024"
  }'

{
  "created": 1751450000,
  "data": [
    {"url": "http://localhost:8100/...signed-media-url...", "revised_prompt": "..."}
  ]
}

Image generation works across configured providers (DALL-E 3, SDXL, fal, Replicate, Stability). Generated media is stored in the gateway’s media store and served via short-lived signed URLs. Prompt-level guardrails (image_gen_input/image_gen_output surfaces) apply.

Audio and video

• TTS — POST /v1/proxy/llm/audio/speech with {"model": "...", "input": "text", "voice": "..."} returns audio bytes (OpenAI, ElevenLabs, Google voices depending on configured models).
• STT — POST /v1/proxy/llm/audio/transcriptions as multipart/form-data with a file field (Whisper, Deepgram, AssemblyAI).
• Video — asynchronous: POST /v1/proxy/llm/video/generations returns a job; poll GET /v1/proxy/llm/video/jobs/{job_id} until it completes.

See the LLM Proxy API reference for full multimodal schemas.

Response headers

Header	Example	Meaning
X-Usage-Warning	Daily budget at 85%: $4.25 / $5.00	A budget/quota warning threshold has been crossed — surface it to users, or use it to back off proactively
Retry-After	27	On 429 responses — seconds until the limiting window resets

Error handling

429 — limits

Throughput caps (RPM, tokens/min, $/min, concurrency) and quota caps (e.g. daily tokens) return 429 with a Retry-After header. Rate-limit shape:

{
  "error": {
    "type": "rate_limit_exceeded",
    "message": "Requests-per-minute limit reached for group 'sales'",
    "scope": "group",
    "limit_type": "requests_per_minute"
  }
}

Usage-limit (quota) shape:

{
  "error": "Usage limit exceeded",
  "usage_limit": {
    "group_id": "rg-01ABC...",
    "group_name": "sales",
    "limit_type": "daily_tokens",
    "current_usage": 100241,
    "max_limit": 100000
  }
}

Honor Retry-After. Token and request caps always block; dollar-budget caps block only when the operator sets hard_stop (otherwise they warn via headers and alerts). See Limits.

403 — guardrail blocked

When an input or output guardrail blocks the request:

{
  "error": {
    "code": "guardrail_blocked",
    "message": "PII detected in input: credit_card",
    "guardrail": "Sales — PII Egress Block (input)",
    "check": "pii",
    "direction": "input"
  }
}

On the native Anthropic surface the same information arrives Anthropic-shaped:

{
  "type": "error",
  "error": {
    "type": "invalid_request_error",
    "code": "guardrail_blocked",
    "message": "PII detected in input: credit_card",
    "guardrail": "Sales — PII Egress Block (input)",
    "check": "pii",
    "direction": "input"
  }
}

Treat guardrail_blocked as non-retryable — retrying the same content will be blocked again. Surface the message to the user.

Don't parse blocks as provider errors

A 403 guardrail_blocked or 429 from the gateway never reached the provider. If your code maps all non-200s to “OpenAI is down”, governance blocks will look like outages. Branch on error.code / error.type first.

Other statuses

Status	Meaning
400	Validation error (bad body, unknown parameters)
401	Missing/invalid credential
403	Guardrail block, argument-policy denial, or no access to the model
404	Unknown model / model not in your groups
5xx	Upstream provider failure (after the gateway’s retry/failover policy is exhausted)

Related

• Build an agent — put chat + tools together in a loop
• Coding agents — Claude Code and CLIs via base-URL env vars
• LLM Proxy API reference — every field, every endpoint