Skip to content
Brutor AI Docs

Metrics & Grafana

The Core Proxy exposes Prometheus metrics at GET /metrics on port 8100 — no auth, standard exposition format, scrape-ready.

Terminal window
curl -s http://localhost:8100/metrics | grep proxy_requests_total | head

Key metric families

Section titled “Key metric families”

All families are prefixed proxy_:

Traffic & latency

Metric Type Labels Meaning
proxy_requests_total counter proxy_type, method, status, provider Every completed proxy request
proxy_request_duration_seconds histogram proxy_type, method, provider End-to-end request latency
proxy_active_streams gauge — SSE streams currently open
proxy_slow_requests_total counter route, outcome Requests over the slow threshold

LLM economics

Metric Type Labels Meaning
proxy_llm_tokens_total counter direction, provider, model Prompt/completion tokens
proxy_llm_estimated_cost_microdollars counter provider, model Cost in micro-dollars (divide by 1,000,000 for USD)
proxy_prompt_cache_tokens_total counter direction, mechanism, provider Prompt-cache read/write tokens
proxy_prompt_cache_savings_microdollars_total counter mechanism, provider USD saved by prompt caching
proxy_prompt_cache_decisions_total counter mechanism, applied, reason Cache-injection decisions

Governance & alerts

Metric Type Labels Meaning
proxy_guardrail_blocks_total counter surface, check Requests blocked by guardrails
proxy_system_alerts_raised_total counter category, severity, resource_type System alerts raised (first occurrence)
proxy_system_alerts_deduped_total counter category, severity, resource_type Alert re-raises collapsed by dedup
proxy_system_alerts_auto_resolved_total counter category Alerts that self-resolved

Internals

Metric Type Meaning
proxy_db_pool_size / proxy_db_pool_idle gauge DB connection pool state
proxy_db_pool_acquire_wait_seconds histogram Connection checkout latency — the first thing to alert on when the proxy slows down

The dev observability stack

Section titled “The dev observability stack”

brutor-gateway-core/docker-compose.yml ships a complete local observability stack alongside the proxy:

Service Port(s) Role
brutor-gateway-core 8100 Gateway + /metrics
otel-collector 4317 (OTLP/gRPC), 4318 (OTLP/HTTP), 8889 (Prometheus export) Receives the proxy’s OTel traces/metrics
prometheus 9090 Scrapes the proxy and collector (15s interval, 30d retention)
loki 3100 Log aggregation
grafana 3010 Dashboards (default login admin / admin)

Data flows two ways into Prometheus: the proxy pushes OTel to the collector (:4317), which re-exports on :8889; Prometheus also scrapes the proxy’s /metrics directly. Grafana is pre-provisioned with Prometheus and Loki datasources (deploy/grafana/provisioning/); collector and Prometheus configs live under deploy/otel-collector/config.yaml and deploy/prometheus/prometheus.yml.

Terminal window
cd brutor-gateway-core
docker compose up -d
open http://localhost:3010   # Grafana

Production wiring

Section titled “Production wiring”

In production you bring your own Prometheus/Grafana (or a compatible vendor). The essentials:

  1. Scrape every core-proxy instance at /metrics on :8100. In Kubernetes/ECS, use your usual pod/task discovery; the endpoint is cheap to scrape at 15–30s intervals.
  2. /metrics is unauthenticated by design (like /health) — restrict it at the network layer so it isn’t internet-reachable.
  3. Metrics are per-instance and label-free with respect to tenants — use sum(...) across instances, and turn to the Control Plane metrics API or logs when you need per-tenant/per-group breakdowns.
  4. Suggested first alerts: 5xx error-rate per proxy_type, proxy_request_duration_seconds P95, proxy_db_pool_acquire_wait_seconds P95, and a delta alert on proxy_guardrail_blocks_total (a sudden spike usually means an attack or a misconfigured guardrail — either way you want to know).
  5. Point Loki/your log pipeline at the containers’ JSON logs (USE_JSON_LOGGING=true, RUST_LOG=brutor_gateway_core=info).