Architecture

Where it sits #

The gateway is consumer-side. It is deployed inside the enterprise blast radius, not in front of it: private subnets, an internal load balancer, adjacent to the applications and agent runtimes it governs. The Token Factory is reached over a private interconnect. The gateway never serves inference itself, and it holds no model weights.

Z1 apps, agents, CI ─┐
                     ├─► Z3  iag-lite  ─── private interconnect ───► Z5  Token Factory
Z2 people (OIDC)  ───┘         │                                          │
                               └─► Z3 state: Postgres (metadata only)     └─► model serving

One workspace maps to one upstream project, and tenants are never multiplexed onto a single upstream credential. Prompt and completion bodies are not stored by default; body capture is a per-workspace opt-in with an explicit retention window.

The request path #

Both ingress dialects converge on one middleware chain. An Anthropic request is translated into an OpenAI-shaped body on the way in and translated back on the way out, so every stage downstream sees one object regardless of which SDK called.

HTTP ingress   OpenAI /v1/chat/completions  |  Anthropic /v1/messages
  └─ dialect adapter
  └─ chain
       1 wire hygiene    body cap 8 MiB, strip inbound identity headers, traceparent
       2 gate            virtual key → principal, model allowlist, budget preflight
       3 exact cache     key scoped by workspace + principal + policy + catalog version
       4 trim            tool pruner → dedup → history window
       5 route           alias or cascade over the workspace's allowed models
       6 invoke          upstream client; streaming tee reassembles the full response
       7 verify          structural checks, tool-call interception, approvals
       8 meter           usage block × catalog rates → cost, baseline, savings
       9 record          one metadata-only row per inference, live-feed cursor

The nine stages #

Streaming #

Request-side stages run before the upstream call, so a streamed request is gated, cached, trimmed and routed exactly like a buffered one. Frames are then streamed to the client as they arrive, with one documented exception: when a request carries tools and the workspace has tool gating on, the response is buffered so that held calls can be stripped before the client sees them. Delivering a tool call and then revoking it is not a thing a client can act on.

In both cases the invoke stage reassembles the full response — content, tool calls and the usage block — so verify, meter and record see the same object for streamed and non-streamed calls. On OpenAI streams the gateway injects stream_options.include_usage=true so the usage block actually arrives; when it does not, cost is billed at the abort floor and flagged estimated rather than silently recorded as zero.

Fail posture #

The rule is one sentence: a cache outage must never become an inference outage, and an optimiser must never become a security control.

Hot-path rules #

These come from verified upstream behaviour. Breaking one causes silent billing or attribution corruption rather than a test failure, which is why they live in the chain and in the upstream client rather than in a proxy config.

Two listeners #

The split is a security control, not a convenience.

The admin surface mints virtual keys, moves spend caps and rewrites policy. An application that can reach inference must not reach it merely because the two once shared a port, so config validation refuses a boot where the two addresses are equal. Every unmatched path on the application listener returns a JSON error, never the console HTML.

Routing #

Routing is never a black box: every response carries the model that ran and the reason it was chosen, and the full decision record is persisted per request.

Virtual model names #

Routine-ness is declared at the call site, not guessed per request. Three candidate classifiers were measured and none beat sending everything to the small tier; the measurements are on the evidence page.

Declared upstreams #

A catalog model may declare the endpoint that serves it, so a self-hosted small model can sit on the small tier and take iag/fast traffic:

- id: local/qwen3-4b
  tier: small
  input_per_1m: 0            # allowed only because the model declares an upstream
  output_per_1m: 0
  upstream:
    base_url: "http://model-host:8090"                   # origin only
    model: "mlx-community/Qwen3-4B-Instruct-2507-4bit"   # its id on that server

Every hot-path rule above still applies, because they live in the chain and the client rather than in the destination. The upstream credential is per endpoint and the Token Factory key is never sent to a declared upstream — config refuses an entry that names it, and refuses one whose origin is the Factory itself.

The catalog id is the only name that escapes: the endpoint's own id for the model is substituted on the way out and translated back on the way in, on both the JSON and the streaming paths. That is not cosmetic, because the meter prices what the response body names.

Availability is the optimiser's fail-open. Each declared endpoint is probed at boot and on an interval; one that stops answering has its models excluded from routing with the exclusion named in the route reason, while a request that pinned an unavailable model is refused with a clear 503 model_unavailable rather than a failure blamed on the Factory.

Trust boundaries #

The claim the deployment has to support is that prompt content traverses exactly two hops: inside the VPC, and a private interconnect to your own Token Factory region. Corporate identity never leaves the enterprise, and no inference path touches the public internet.

There is one deliberate exception, off by default. A declared upstream is a third content-bearing flow, and it exists only where your config names it. It is shown on the catalog and in the console as local:<host>, because an operator has to be able to see that prompt content reaches somewhere other than the Factory. The two-hop claim continues to hold while that endpoint sits inside your VPC, and the deployment documentation states plainly that this is operator discipline rather than something the gateway enforces.

What is stored #

All gateway state lives in your Postgres: workspaces, virtual keys, policy versions, budgets, the request record, held tool calls and approvals, and the evidence ledger. The request record is metadata only — identity, model, tokens, cost, verdict, routing decision, timings. Prompt and completion text are not written unless body capture is switched on for that workspace with a retention window.

Out of scope, named rather than hidden #

The single-container profile does not include: listener TLS, an OIDC session login for the console, the hosted Envoy form factor, regional edge steering, semantic caching with a real embedder, or upstream catalog synchronisation. Each is a real gap with a reason, and each is listed in the design document rather than implied to exist.