# Securing the LLM Request Path: Implementing a Forward-Proxy for Data Privacy
*A Security Architect’s field guide to terminating every model call inside your own VPC — with real-time PII redaction, multi-provider routing, and token-level audit logging that survives an audit.*
—
## 1. Introduction: The Vulnerability of Raw Model-Vendor API Integrations
The fastest path to an LLM feature is also the least defensible. A developer imports the vendor SDK, drops an API key into an environment variable, and calls the endpoint. The demo works. Nobody stops to ask what actually traversed the wire.
What leaves your perimeter on every call is not “just a prompt.” It is the accumulated context of your business: customer names and emails embedded in support transcripts, account numbers pasted into a summarization request, internal hostnames, database IDs, source-code snippets, and the occasional secret that a well-meaning engineer forgot to scrub. From a security standpoint, a raw SDK integration is an **unfirewalled, unlogged, unmanaged exfiltration channel** that happens to be carrying regulated data.
Three properties make this worse than a typical outbound API call:
**There is no chokepoint.** If ten services each hold their own provider credential, you have ten egress paths with ten different code paths, none of which your security team can inspect. There is no place to enforce policy because there is no single place *where the request lives*.
**There is no data-loss prevention.** Vendor SDKs transmit exactly what the application passes them. No redaction, no classification, no fail-closed behavior. The provider’s data-retention policy is a contract, not a control — it governs what the processor *does* with the data after it has already crossed your boundary.
**There is no attributable audit trail.** When legal asks *”did any PHI leave the environment in Q2, to which processor, under whose credentials?”* a sprawl of SDK call sites cannot answer. Prompt and response bodies are not logged, tokens are not attributed per team, and there is no immutable record of the decision to send.
The mistake is architectural, not moral. Direct SDK integration optimizes for the first deploy and silently mortgages the next three years of compliance, cost control, and incident response. The fix is equally architectural: stop letting applications talk to model vendors, and insert a control plane that every model call must pass through.
—
## 2. The Privacy Proxy Pattern: Terminating Every Model Call Inside Your VPC
A **privacy forward-proxy** is a self-hosted service that sits between your workloads and every model vendor. Applications no longer resolve `api.openai.com` or `bedrock-runtime.
The pattern rests on a single enforcement primitive: **the egress allowlist**. Your VPC’s security groups and route tables permit outbound traffic to model-vendor endpoints *only* from the proxy’s subnet. Application subnets have a default route that goes nowhere near the internet. This converts “we trust developers to redact” into a network-layer guarantee — a request that does not transit the proxy literally cannot reach a provider.
That inversion produces four properties you cannot get from SDK-level discipline:
– **Terminate, don’t tunnel.** The proxy parses the request into a normalized structure, which is the precondition for doing anything intelligent — redaction, routing, metering — before re-serializing it for the upstream. A blind TCP forwarder has no idea where the PII is; a terminating proxy does.
– **One policy surface.** Classification rules, redaction engines, quotas, and provider credentials live in one place. Tightening a rule is one deploy, not a cross-team migration.
– **Credential custody.** Provider secrets never reach application containers. The proxy holds them, ideally via a secrets manager or IAM role rather than static config, and rotates them independently.
– **Blast-radius reduction.** An application compromise yields a request to the proxy, not to a raw vendor key. The proxy still enforces policy on that traffic.
The mental model to hold onto: the proxy is not a convenience router bolted onto the side of your architecture. It **is** the trust boundary for AI traffic. Everything upstream of it is untrusted by design; everything downstream is a confined, inspected, attributable channel.
—
## 3. PII Redaction: Real-Time Regex and NER-Based Scrubbing
Redaction must run inside the request pipeline — after authentication, before the provider adapter — at the last moment you still control the bytes. A layered detector stack balances precision, recall, and latency.
**Layer 1 — Deterministic patterns.** Regular expressions with checksum validation catch high-precision identifiers cheaply and deterministically: emails, phone numbers, credit-card numbers (Luhn-validated), IBANs, US SSNs, IPv4/IPv6 addresses, AWS access keys, and JWT-shaped strings. These rules are explainable line-by-line, which matters when an auditor asks why a specific string was masked. Regex is fast enough to sit on the hot path with negligible cost, and it should run first because it removes the highest-risk tokens before any heavier processing.
**Layer 2 — Named-entity recognition.** Patterns cannot catch names, organizations, or street addresses that no fixed format describes. A small local transformer — a DistilBERT-class NER model or a quantized span tagger — running in-process on CPU handles free-text entities at a few milliseconds per request. The critical constraint: **the redaction model runs inside your VPC**. Using a hosted moderation or classification API to sanitize an API call would recreate the exact leak you are trying to close, so the NER stage must never make an outbound request.
**Layer 3 — Structured-field policy.** LLM payloads are often JSON, and when they are, path-based rules beat text scanning. A schema declares which keys are sensitive (`patient_id`, `ssn`, `account_number`, `date_of_birth`) and redacts by JSON path, so a field is treated as sensitive because *of where it is*, not because a pattern guessed it was. This is more reliable than free-text scanning for structured data and cheaper, since it skips the NER stage for known-safe fields.
**Consistent, reversible tokenization.** Naive redaction destroys the model’s ability to reason about “the same customer.” Replace every occurrence of a value with a *stable* placeholder — `jane.doe@corp.com` → `[EMAIL_7f3a]`, and the same value maps to the same token within a session. The model sees referential consistency without ever seeing the identity. A token vault, held outside model context and encrypted at rest, lets authorized callers re-hydrate placeholders in the final response; everything else stays masked.
Two non-negotiables govern the whole stage. First, **fail closed**: if the NER model fails to load or a rule errors, reject the request — never pass it through unscanned. Second, **log the decision, not the data**: every redaction event writes an immutable record of the request ID, actor, provider, rule matched, and match count. Prompt and response bodies are hashed or omitted by default; you audit the *behavior* of the scrubber, not the sensitive content it removed.
—
## 4. Forward-Proxy Architecture: A Go or Rust Multi-Provider Router
The proxy must be fast (it is on every request), safe under concurrency, and boring enough to operate. Go and Rust are the correct choices for opposite reasons: **Go** for operational simplicity, a mature HTTP/2 and streaming story, and a hiring pool; **Rust** when you want zero-cost abstractions, no GC pauses on latency-sensitive streaming, and the strongest memory-safety guarantees in the egress path.
### 4.1 Control plane vs. data plane
Split the system in two. The **data plane** is stateless — it holds no mutable state between requests, so it scales horizontally behind a load balancer and can be redeployed without draining a database. The **control plane** owns configuration: provider credentials, routing rules, capability registry, quotas, and the cost ledger. That separation is what lets you rotate a provider key or tighten a redaction rule without touching the request path’s availability.
### 4.2 The provider adapter interface
Each upstream is an implementation of one interface. The application layer never imports a provider SDK again.
“`go
type Request struct {
Model string
Messages []Message
Tools []Tool
MaxTokens int
Stream bool
Actor string // for attribution
Redactions []RedactionEvent // filled by the scrubber
RequestID string
}
type Response struct {
Content string
ToolCalls []ToolCall
Usage Usage // prompt/completion tokens
Provider string
Model string
}
type Provider interface {
Name() string
Chat(ctx context.Context, r Request) (Response, error)
Capabilities() Capabilities // context window, tool fidelity, JSON mode
}
“`
Bedrock implements `Chat` by signing with **SigV4** (credentials from an IAM role, never a static key) and translating to the Converse API. OpenAI uses Bearer auth. Azure OpenAI keys off a deployment name and API version and authenticates with Entra ID managed identities. Adding a fourth provider is a new adapter file, not a migration.
### 4.3 Capability negotiation, not assumption
Providers differ in ways that silently corrupt prompts: maximum context length, tool-calling fidelity, JSON-mode support, and tokenizer. The proxy keeps a **capability registry** and either degrades gracefully — strip unsupported tools, chunk an oversized context — or returns a typed `unsupported_capability` error. It never silently truncates. Token counting uses the model-specific tokenizer, because a 4,000-token prompt is not 4,000 tokens everywhere, and your quota and cost math depend on being right.
### 4.4 The request pipeline
Every request walks the same ordered chain, and any stage may reject it:
“`
authn → authz → quota → classify → PII scrub → provider route
→ upstream call → normalize response → re-hydrate → meter → audit
“`
Routing selects a provider by policy: a **capability match** (does this model support the tools the caller requested?), a **data-residency rule** (regulated workloads pinned to a region-resident provider), and a **cost tier** (steady-state extraction to the cheapest capable model, reasoning to the frontier). Failover is a health-checked second choice, so a provider outage becomes a routing decision rather than a customer-visible incident.
### 4.5 Hardening the egress path
The proxy is now your highest-value target, so treat it like one. Enforce **mTLS** between applications and the proxy. Run it with a minimal base image and no shell, drop the outbound path to an explicit allowlist of provider endpoints, and put per-actor **rate limits** in front of the whole chain so a runaway agent cannot exhaust a quota or your budget. Rotate provider credentials on a schedule, and log every administrative config change to the same immutable store as request metadata.
—
## 5. Observability: Centralizing Cost Attribution and Token Audit Logs
Because the proxy is the chokepoint, it is the natural and *only* place to build a complete financial and security record of AI usage. Two ledgers, one stream.
**Cost attribution.** Every normalized response carries provider-reported token usage. The proxy multiplies it by the current per-model price book and attributes the cost to an owner derived from the authenticated identity — team, service, tenant, or agent. Now you can answer the questions that otherwise arrive as a surprise at renewal: *which team drove the bill, which endpoint is the token hog, and what does a retired feature actually cost to keep?* This is the instrumentation that makes routing decisions accountable rather than aspirational.
**Token audit logs.** Each request emits a structured, append-only record: request ID, actor, provider, model, prompt and completion token counts, redaction count and matched rule classes, latency, and the routing decision. Bodies are never logged by default — only metadata and hashes. Written to append-only storage with retention tied to your compliance window, this log is the evidence that satisfies legal, audit, and incident response without turning your logging stack into a secondary data reservoir.
**Operational telemetry.** Beyond finance and compliance, the same layer feeds p50/p99 latency, upstream error rates, fallback frequency, and per-provider availability into your existing observability stack. Security and reliability share a chokepoint.
The design rule: **observability is a byproduct of architecture, not a retrofit.** Because every call already transits one control plane, you get a complete ledger for free. Teams that skipped the proxy spend their next two quarters reconstructing usage from vendor dashboards they do not control.
—
## 6. Reference Implementation: A VPC-Isolated Forward-Proxy for Bedrock and OpenAI
A minimal but production-shaped data-plane handler in Go shows how the pieces compose.
“`go
func (p *Proxy) handleChat(w http.ResponseWriter, r *http.Request) {
actor, err := p.authn(r) // mTLS + identity
if err != nil { http.Error(w, “unauthorized”, 401); return }
req, err := decode(r) // normalize to internal Request
if err != nil { http.Error(w, “bad_request”, 400); return }
req.Actor = actor
req.RequestID = newID()
if !p.quota.Allow(actor, req) { // fail closed on budget
http.Error(w, “quota_exceeded”, 429); return
}
req, events := p.scrub.Scrub(req) // regex → NER → field policy
req.Redactions = events // never log bodies, only events
provider, err := p.registry.Select(req) // capability + residency + cost
if err != nil { http.Error(w, “unsupported_capability”, 422); return }
resp, err := provider.Chat(r.Context(), req)
if err != nil {
resp, err = p.failover(r.Context(), req, provider) // health-checked
if err != nil { http.Error(w, “upstream_unavailable”, 502); return }
}
resp.Content = p.vault.Rehydrate(resp.Content, actor) // authorized only
p.meter.Record(actor, provider.Name(), req.Model, resp.Usage, events)
p.audit.Append(actor, req.RequestID, provider.Name(), resp.Usage, events)
writeJSON(w, resp)
}
“`
Deployment shape: the proxy runs in a dedicated private subnet with **no inbound internet and a tightly scoped egress allowlist** to Bedrock and OpenAI endpoints (via VPC endpoints for Bedrock where available). Provider credentials come from IAM roles and a secrets manager, never environment files. Applications reach it over mTLS on an internal load balancer. The whole component deploys as an immutable container, logs to append-only storage, and is monitored like any other security-critical service.
—
## Build Your Privacy Proxy with OnlineCloud
A forward-proxy for the LLM request path is the difference between *”we use AI”* and *”we operate AI under control.”* It is the layer where raw exfiltration becomes an enforced boundary, PII stops leaving the perimeter, token cost becomes attributable, and the model vendor becomes a swappable implementation detail — but it only delivers those properties when it is built with the rigor of a security control plane, not a convenience router.
That is exactly what we do at **OnlineCloud**, an MCP-native engineering studio building evaluated, observable AI infrastructure — VPC isolation for regulated data, Bedrock-versus-self-hosted decisions, gateway and forward-proxy architecture, and the FinOps instrumentation to prove the savings are real.
**Have a project in mind? [Talk to the OnlineCloud architects](https://www.onlinecloud.dev/)** — email **contact@onlinecloud.dev** for a free initial consultation, or start at [onlinecloud.dev](https://www.onlinecloud.dev/). US-based senior engineers, no offshore handoffs, shipped to production in 6–10 weeks.
Leave a Reply