User guide

Vocabulary #

Choosing a model #

Callers put one of four things in the model field. The gateway does not guess how hard a request is — that was measured and it did not work, so the decision stays where the knowledge is, at the call site.

Whatever the caller asks for, the response says what actually ran and why:

x-iag-model-routed: openai/gpt-oss-20b
x-iag-route-reason: alias iag/fast → cheapest allowed small-tier model
                    (excluded, upstream unavailable: local/qwen3-4b)

Policy #

Policy is per workspace and versioned. The cache key includes the policy version, so publishing a policy cannot serve an answer that an older policy allowed.

PUT /admin/v1/policy/ws-pilot
{
  "allowed_models": ["openai/gpt-oss-20b", "openai/gpt-oss-120b"],
  "features": {"routing": true, "cache": true, "trim": true,
               "verify": true, "tool_gating": true},
  "tool_gates": {"lookup_order": "read", "issue_refund": "financial"},
  "tool_memo":  {"lookup_order": {"ttl": "10m", "scope": "principal"}},
  "baseline_model": "openai/gpt-oss-120b",
  "budget_posture": "fail-closed"
}

A request naming a model outside allowed_models is refused at the gate, before it costs anything. The allowlist is intersected with the catalog, so a policy cannot introduce a model the operator did not configure.

Feature toggles #

Budgets #

PUT /admin/v1/budgets
{"workspace":"ws-pilot","principal":"alice","period":"monthly","cap_usd":250}

Caps are per workspace, or per principal within one. budget_posture decides what happens when the budget cannot be read: fail-closed refuses the request, fail-open serves it and reconciles afterwards. A breach returns a structured error naming the budget and what to do about it — never a silently truncated answer.

Verdicts and approvals #

Reading a verdict #

Verification is a set of named structural checks, not a truth oracle. It catches the failures that have a shape: an empty completion, a response truncated by the token limit, invalid JSON where the request demanded JSON, a refusal template, hedging density, repetition loops, a language mismatch, a tool call that violates its schema or a business rule.

An optional judge can run as a second layer through your own small tier, spend-bounded per check. It is off by default, because it costs money on every request and the structural checks catch the shaped failures for free.

Tool gates #

A tool is classified by what it does, and the class decides whether a human sees it first. Read-class calls flow. Financial and external-communication classes are held.

"tool_gates": {
  "lookup_order":  "read",              // flows
  "send_email":    "external-comms",    // held for approval
  "issue_refund":  "financial"          // held for approval
}

A held call is stripped from the response before the client sees it, and the caller re-requests after the approval lands. That is deliberate: handing a client a tool call and then revoking it is not something the client can act on.

The approval inbox #

curl -s -H "$A" "$B/approvals?workspace=ws-pilot&status=pending" | jq .
curl -s -H "$A" -X POST "$B/approvals/$ID/approve"
curl -s -H "$A" -X POST "$B/approvals/$ID/deny" -d '{"reason":"customer not verified"}'

Every decision is persisted with the actor, the reason and the time, and written into a hash-chained evidence ledger. The point is not the queue; it is that six months later you can reconstruct who allowed what, and prove the record has not been edited since.

Spending less #

In the order that actually matters, measured rather than assumed.

Keep the prefix stable #

This is the single biggest lever, and it is mostly about what you stop doing. Providers cache prompt prefixes; a stable system prompt comes back almost entirely from cache on every call after the first. One volatile byte at the front destroys that.

The gateway prices the provider's cached prompt tokens at your contract's cached rate when you have configured one, and reports the share on every response and on the overview.

Tool-call memoisation #

Agent loops re-issue the same read-only call across turns — the same lookup on turn three and again on turn eleven. The gateway never executes tools, but it sees both the call and its result in the conversation, so it can finish the round-trip itself.

Learning is passive and costs nothing: the client already paid for those executions. When a model proposes calls that are all memo hits, the gateway appends the remembered results and re-invokes, and the client gets the continuation. A mixed proposal is returned untouched, because the client has to execute the miss anyway.

Admin API #

Prefix /admin/v1 on the admin listener, bearer authentication. Lists return {data, next}; errors return {error: {code, message}}.

The console #

Served at / on the admin listener, on the same origin as the API. Every number on screen comes from the endpoints above. There is no fixture mode and no demo data.

Day to day #