Building a Private LLM Gateway for Multi-Cloud Enterprise Environments

# Building a Private LLM Gateway for Multi-Cloud Enterprise Environments

*A Security Platform Engineer’s blueprint for breaking model-vendor lock-in, containing data exfiltration, and routing inference across Bedrock, OpenAI, and Azure from a single hardened control plane.*


—

## 1. Introduction: Two Risks Nobody Signs Up For

When an enterprise adopts its first LLM feature, the decision looks tactical: pick a provider, get a key, ship the demo. Eighteen months later that decision has metastasized into architecture. Every service calls the vendor SDK directly, prompts are tuned to one model’s quirks, token limits are baked into business logic, and the contract renewal is now a hostage negotiation. That is **model-vendor lock-in** — and it is rarely the result of a deliberate bet. It is the residue of a thousand convenience imports.

The second risk is quieter and far more expensive: **data exfiltration through the inference path**. An LLM call is an outbound POST containing your most sensitive context — customer PII, internal identifiers, regulated records, sometimes raw database rows. If every team holds its own provider credential, your security team has no chokepoint, no redaction, no audit trail, and no way to answer the only question that matters after an incident: *what data left the perimeter, to which processor, under whose authorization?*

A **private LLM gateway** solves both. It is a self-hosted control plane that terminates every model call inside your trust boundary, normalizes provider APIs behind one contract, scrubs sensitive data before it egresses, attributes every token to an owner, and makes the downstream model a swappable implementation detail. This post covers the architecture, the security controls, the latency discipline, a working Go implementation, and the procurement leverage it creates.

—

## 2. Gateway Architecture: One API in Front of Bedrock, OpenAI, and Azure

The core design principle is **terminate, don’t proxy blindly**. A gateway that forwards raw vendor JSON is a router; a gateway that parses, normalizes, and re-issues requests is a control plane. The normalized shape is the OpenAI Chat Completions contract — not because it is the best schema, but because it is the de facto interoperability standard and every SDK your teams already use can target it.

“`
┌──────────────────────────────────────────────┐
│ Private LLM Gateway │
Apps / Agents ───►│ authn → authz → quota → PII scrub → route │
│ → provider adapter → meter → audit log │
└───────┬──────────────┬──────────────┬────────┘
│ │ │
┌───────▼──────┐ ┌─────▼──────┐ ┌─────▼─────────┐
│ Bedrock │ │ OpenAI │ │ Azure OpenAI │
│ (SigV4/IAM) │ │ (Bearer) │ │ (Entra ID) │
└──────────────┘ └────────────┘ └───────────────┘
“`

Three architectural commitments make this work:

**Provider adapters behind a stable interface.** Each upstream is an implementation of one interface — `Chat(ctx, Request) (Response, error)` — responsible for its own auth, request translation, response normalization, and error mapping. Bedrock signs with SigV4 and speaks the Converse API; OpenAI uses Bearer auth; Azure OpenAI keys off a deployment name and API version and authenticates with Entra ID managed identities. The application layer never imports a provider SDK again, and adding a fourth provider is a new adapter, not a migration.

**Capability negotiation, not assumption.** Providers differ in ways that silently break prompts: max context, tool-calling fidelity, JSON-mode support, streaming semantics, and tokenizer. The gateway keeps a **capability registry** and either downgrades gracefully (strip unsupported tools, chunk an oversized context) or returns a typed `unsupported_capability` error — never a corrupted response. Token counting uses the correct tokenizer per model, because a 4,000-token prompt is not 4,000 tokens everywhere, and your quota math depends on getting that right.

**Stateless edge, stateful control plane.** The request path is stateless so it scales horizontally; policy, quotas, keys, and cost ledgers live in a control plane with its own storage. That separation is what lets you rotate a provider credential or tighten a rate limit without redeploying the data plane.

—

## 3. Security: PII Scrubbing and Cost Attribution at the Chokepoint

The gateway’s security value comes entirely from being the *only* egress path. Two controls justify the entire build.

### 3.1 Deterministic PII scrubbing before egress

Redaction must happen in the request pipeline, after auth and before the provider adapter — the last point at which you still control the bytes. The layer combines three detectors, in increasing cost and decreasing determinism:

1. **Regex and checksum rules** for high-precision formats: emails, phone numbers, national IDs, credit cards (Luhn-validated), IBANs, IP addresses, AWS keys. Fast, explainable, auditable.
2. **Named-entity recognition** (a small local transformer, e.g. a DistilBERT-class NER model) for names, organizations, and addresses that no pattern catches. Runs in-process on CPU — no model call leaves your perimeter to sanitize a model call.
3. **Structured-field policy** for JSON payloads: any key matching a configured allow/deny schema (`patient_id`, `ssn`, `account_number`) is redacted by path, not by guessing at free text.

Redaction uses **consistent, reversible tokenization**: `jane.doe@corp.com` becomes `[EMAIL_7f3a]`, and the same value maps to the same placeholder within a session. That preserves the model’s ability to reason about “the same customer” without ever seeing the identity, and a protected vault held outside the model context lets you re-hydrate placeholders in the final response for authorized callers. Everything else stays masked.

Critically, the scrubber is **fail-closed and logged**. If the NER model fails to load or a rule errors, the request is rejected, not passed through. Every redaction event is written to an immutable audit record: request ID, actor, provider, rule matched, count, and the decision — answering the post-incident question with evidence instead of memory. Prompt and response bodies are never logged by default; only hashes, metadata, and redaction counts.

### 3.2 Cost attribution at the token boundary

The gateway is also the only place where every model call, across every provider and every team, is observable in one numerator. That makes it the natural home for **per-request cost attribution**:

– Each call is tagged with `team`, `app`, `user`, `environment`, and `cost_center` — propagated from the authenticated principal, not trusted from client headers.
– Token counts come from the provider’s usage object, validated against the gateway’s own tokenizer estimate to catch drift.
– Rates are applied from a versioned price book so a procurement renegotiation doesn’t rewrite history; you can reprice past windows without losing the original ledger.
– A `cost_per_request` record and Prometheus counter are emitted on every call, enabling chargeback, anomaly detection, and a live answer to “which team spent $40k last month and on what?”

The compounding benefit: because the gateway already knows the cost and capability of each adapter, it can enforce **budget-aware routing** — serving a request from the cheaper equivalent model when a team’s monthly ceiling is near, and escalating to the frontier tier only when policy allows.

—

## 4. Performance: Holding p99 Across Cross-Cloud Routing

Multi-cloud routing introduces three latency taxes, and a gateway that ignores them turns a 300ms feature into a 3-second one.

**Tax 1 — the extra hop.** The gateway adds a network round trip. Mitigate by colocating the data plane in the same region as the dominant provider and by keeping the hot path free of synchronous control-plane reads (policy is cached with short TTLs and invalidated by event, not by polling).

**Tax 2 — provider variance.** Bedrock, OpenAI, and Azure have different TTFT and tail behaviors under load. You cannot fix an upstream’s p99, but you can *insulate* your callers from it:

– **Hedged requests for idempotent calls.** If a request hasn’t produced a first token within a threshold (e.g. p50 × 2), launch the same request against a fallback provider and take the first stream to emit. Costs a duplicate call only on the slow tail; collapses p99 dramatically.
– **Deadline propagation.** Every call carries a context deadline derived from the caller’s SLA. When it expires, the stream is cancelled upstream — no zombie generations burning tokens.
– **Circuit breakers per provider.** Consecutive timeouts or 5xx bursts open the breaker and route around the degraded provider until half-open probes succeed.

**Tax 3 — streaming semantics.** Streaming must be byte-transparent. The gateway proxies SSE chunks with flush discipline (flush after every event) so added latency is bounded by processing, not buffering. Any transformation — PII scrubbing — runs on the *request* and on *complete response segments*, never on the token stream mid-flight.

The discipline that ties it together is **measuring the gateway’s own overhead** as a first-class metric: `gateway_added_latency_us` (arrival to upstream dispatch) and `provider_ttfb_ms{provider}`, both exported as histograms. If the gateway’s own p99 contribution ever exceeds a few milliseconds, that is a bug with a budget, not an acceptable cost of doing business.

| Routing strategy | p50 | p99 | Cost |
|—|—|—|—|
| Single provider, direct | 420 ms | 3,100 ms | Baseline |
| Gateway, no hedging | 428 ms | 3,180 ms | Baseline |
| Gateway + hedge + breakers | 430 ms | **1,450 ms** | +~2% tokens |

—

## 5. Implementation: A Go Proxy with Auth and Rate Limiting

Go is the right substrate: single static binary, no runtime to patch, goroutine-per-request concurrency for streaming, and a GC you can bound with `GOMEMLIMIT` for predictable tail latency.

The middleware chain is the whole security story, expressed as ordered `http.Handler` wrappers:

“`go
func Chain(h http.Handler, mw …func(http.Handler) http.Handler) http.Handler {
for i := len(mw) – 1; i >= 0; i– {
h = mw[i](h)
}
return h
}

func main() {
mux := http.NewServeMux()
mux.Handle(“/v1/chat/completions”, Chain(
http.HandlerFunc(handleChat),
Recoverer, // panic → 500, no stack leak
RequestID, // correlation for audit
Authn, // verify API key / OIDC, resolve tenant
Authorize, // tenant → allowed models & providers
RateLimit, // per-key token bucket
QuotaCheck, // monthly budget enforcement
PIIScrub, // fail-closed redaction
Meter, // token + cost accounting
))
srv := &http.Server{
Addr: “:8443”,
Handler: mux,
ReadHeaderTimeout: 5 * time.Second,
WriteTimeout: 0, // streaming: never time out the body
MaxHeaderBytes: 1 << 20, } log.Fatal(srv.ListenAndServeTLS("tls.crt", "tls.key")) } ``` **Authentication** resolves a credential to a principal with a tenant, scopes, and cost center. Keys are stored as hashes; OIDC tokens are validated against the issuer's JWKS with audience and expiry checks. Authorization is a small policy function — this key may call `gpt-4o` and `claude-3-5-sonnet`, not `o1-pro` — enforced *before* any upstream call is made. ```go func Authn(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { key := strings.TrimPrefix(r.Header.Get("Authorization"), "Bearer ") p, err := auth.Lookup(r.Context(), key) // hash → principal, cached if err != nil { http.Error(w, `{"error":{"type":"unauthorized"}}`, http.StatusUnauthorized) return } next.ServeHTTP(w, r.WithContext(withPrincipal(r.Context(), p))) }) } ``` **Rate limiting** needs to bound two different resources: requests per second and tokens per minute. A single counter is insufficient because a 200k-token request is not one unit of load. The limiter takes both costs: ```go type Limiter struct { mu sync.Mutex buckets map[string]*tokenBucket // keyed by tenant:model } // Allow consumes the request cost and the estimated token cost atomically. func (l *Limiter) Allow(tenant string, tokens int) (bool, time.Duration) { l.mu.Lock() defer l.mu.Unlock() b := l.buckets[tenant] if !b.tryTake(1) { return false, b.retryAfter() } if !b.tryTakeTokens(tokens) { b.refund(1) // atomic: don't leak a request slot on a token denial return false, b.retryAfter() } return true, 0 } ``` For a single-node deployment an in-process bucket is correct. The moment you run more than one replica you need a **shared limiter** — Redis with a Lua script, or a consistent-hash lease per tenant so buckets aren't split across nodes. The failure mode to avoid is the classic: three replicas each enforcing "100 rpm" and letting 300 through. Finally, the **provider adapter interface** keeps the routing decision clean, and the handler stays a thin dispatcher: ```go type Adapter interface { Name() string Chat(ctx context.Context, req *ChatRequest) (*ChatResponse, error) Stream(ctx context.Context, req *ChatRequest) (io.ReadCloser, error) CountTokens(model string, msgs []Message) int } func handleChat(w http.ResponseWriter, r *http.Request) { p := principalFrom(r.Context()) req := decode(r) // normalized shape req = pii.Scrub(r.Context(), req) // fail-closed a, model := router.Pick(r.Context(), p, req) // capability + budget aware ctx, cancel := context.WithTimeout(r.Context(), policy.Deadline(p, model)) defer cancel() resp, err := a.Chat(ctx, req) if err != nil { metrics.ProviderError(a.Name(), err) writeErr(w, err) return } meter.Record(p, a.Name(), model, resp.Usage, req.Redactions) writeJSON(w, resp) } ``` --- ## 6. ROI: Why Model-Agnosticism Is Procurement Leverage The business case is not "we save tokens." It is that **switching cost collapses to a configuration change**, and that changes your negotiating position in every renewal. Three compounding effects: **Pricing leverage.** When a competing provider undercuts your incumbent, the migration is a price-book update and a capability-registry entry — days, not quarters. Providers price against the credible threat of being swapped. A gateway that makes switching real is worth more than the discount it unlocks. **Capacity arbitrage.** Gateways enable portfolio routing: steady-state extraction to the cheapest capable tier, reasoning to the frontier, failover to whichever provider is healthy. Worked honestly, teams on this pattern report 40–70% inference cost reduction *without* an accuracy regression, because the bulk of traffic never needed the premium model. **Risk transfer.** A provider outage, a deprecation, or a sanctions or compliance change no longer becomes your incident. You route around it. Regulated workloads can be pinned to an in-VPC or region-resident provider by policy, while everything else uses the cheapest compliant path — a control that is impossible to express when every team holds its own key. | Lever | Without gateway | With gateway | |---|---|---| | Switch provider | 1–2 quarters of refactor | Config change + eval run | | Cost per request | Flat, single-vendor list price | Routed to cheapest capable tier | | Provider outage | Customer-visible downtime | Automatic failover | | Data leaving perimeter | Unknowable | Enforced at one chokepoint | The one non-negotiable: **you must own an eval suite.** Model-agnosticism is only safe when you can prove, on your own traffic, that the cheaper equivalent model holds quality. That eval set is the asset that turns optionality from a slide into a lever. --- ## Build Your Gateway with OnlineCloud A private LLM gateway is the difference between "we use AI" and "we operate AI under control." It is the layer where lock-in ends, exfiltration gets contained, cost becomes attributable, and procurement regains the upper hand — but it has to be built with the rigor of a security control plane, not a convenience proxy. That is exactly what we do at **OnlineCloud**, an MCP-native engineering studio building evaluated, observable AI infrastructure — Bedrock-versus-self-hosted decisions, VPC isolation for regulated data, gateway and MCP server architecture, and the FinOps instrumentation to prove the savings are real. **Have a project in mind? [Talk to the OnlineCloud architects](https://www.onlinecloud.dev/)** — email **contact@onlinecloud.dev** for a free initial consultation, or start at [onlinecloud.dev](https://www.onlinecloud.dev/). US-based senior engineers, no offshore handoffs, shipped to production in 6–10 weeks.