Building a Custom Retrieval-Interleaved Generator (RIG) Architecture

# Building a Custom Retrieval-Interleaved Generator (RIG) Architecture


*By the engineering team at OnlineCloud — a US-based Agentic AI & MCP engineering studio*

—

## 1. Introduction: Moving Beyond Basic RAG to Dynamic, Interleaved Retrieval

Classic retrieval-augmented generation makes a single, early bet. Embed the question, pull the top-k chunks, stuff them into the prompt, and let the model write one uninterrupted answer against that frozen context. For single-hop lookups over a semi-static corpus, that bet usually pays. The moment questions become multi-hop — reconciling ledgers, comparing contracts against amendments, explaining variances from live ERP rows — the bet is made before the model knows what it needs, and retrieval recall becomes a ceiling no prompt engineering can lift.

The architectural response is to treat retrieval not as a prologue but as a *conversation the generator conducts with its data while it writes*. In a retrieval-interleaved generator (RIG), the model writes what it can defend, emits a control token when its internal evidence runs thin, pauses while a retrieval tool executes against the live system of record, ingests the results, and resumes — possibly several times before the final token. Retrieval becomes a loop the generator drives mid-generation, conditioned on the state of its own output.

This is a different problem than RAG, and different again than the outer agentic loop of plan-act-reflect tool calls, where an orchestrator bounces whole LLM turns back and forth. RIG lives in between: tool-calling granularity *inside* a single response, at sub-turn latency. Done well, it produces answers that are dynamically grounded, numerically verifiable, and traceable to the exact rows behind each sentence. Done badly, it is a latency and cost trap. This post lays out the architecture we use at OnlineCloud — the decision mechanism, the decoding tricks, the verification layer, and the latency budget that keeps it shippable.

## 2. The RIG Pattern: How Generators Decide When to Retrieve Mid-Generation

The core design question is one of control: **who decides that more evidence is needed, and on what signal?** Production systems fuse two decision families.

**Token-level learned decisions.** The model is fine-tuned so the retrieval decision lives in its own output distribution. Extend the vocabulary with control tokens (``, ``, ``) and supervise trajectories with retrieval calls placed where ground truth shows they were needed — before a number only the ledger can supply, before a clause only the contract text contains. Supervised fine-tuning teaches *placement*; a reinforcement pass over outcome rewards teaches *judgment* — when the expected value of another lookup beats the cost of guessing. The result is a generator that retrieves opportunistically from internal state, not from an external planner’s guess.

**Heuristic interrupt gates.** Separate from (or layered on top of) learned placement, an external monitor watches the decoding stream. The cheapest reliable signal is token-level predictive uncertainty: when the softmax over the next token is flat, or the top candidates disagree structurally, the model is improvising — exactly the moment an architect wants a retrieval checkpoint rather than a confident hallucination. A classifier head or a calibrated threshold on sequence log-probability fires an interrupt that forces retrieval before the model commits to a fact-bearing span.

What distinguishes RIG from the outer agentic loop is *coupling*. An agentic workflow decomposes work into whole-turn tool calls — orchestrator asks, tool answers, fresh turn — while RIG keeps the generator’s state (its KV cache, attention pattern, narrative thread) alive across retrieval. The interruption lands at a token boundary, not a turn boundary, and generation resumes from its own hidden state rather than a synthesized transcript. One document can thus hold five interleaved lookups without the coherence loss of five handoffs.

The failure mode to design against is RAG’s mirror image. RAG fails by retrieving too little, too early. RIG fails by retrieving too eagerly — a model that treats `` as a procrastination device, burning round-trips to postpone committing to an answer. The budget controls in Section 5 keep the learned policy honest.

## 3. Implementation: Using Control Tokens and Logit Bias to Trigger Retrieval Tools

Building a custom RIG means owning the decoding loop. Here is the skeleton, in the shape we ship against Oracle, MS SQL, Postgres, and Snowflake backends.

**Vocabulary and fine-tuning.** Reserve three tokens: `` signals a lookup; `` and `` delimit injected results so the model can tell retrieved fact from generated prose. During SFT, annotate trajectories with retrieval points wherever a fact’s source changes, inserting `Q` where Q is the query that would surface the needed rows. During RL, reward trajectories whose outputs pass the verifier from Section 4 and penalize premature or excessive retrieval.

**Decoding with an interruptible loop.** The naive decoder is replaced with a loop that owns the token stream:

“`python
def decode_with_retrieval(model, prompt, tools, gate, budget):
tokens, cache = model.encode(prompt)
for step in range(budget.max_decode_steps):
logits = model.forward(tokens, cache)
if gate.should_retrieve(logits): # uncertainty interrupt
logits[RETRIEVE_ID] += 2.5 # logit bias -> forced trigger
token = sample(logits)
if token == RETRIEVE_ID:
query = model.decode_until(tokens, RETRIEVE_END)
results = tools.execute(query) # typed MCP tool call
tokens += encode(f”{results}“)
budget.retrievals += 1
if budget.retrievals >= budget.max_retrievals: break
tokens.append(token)
“`

Three details decide whether this works in production. **First, the logit bias is a governor, not a crutch.** A +1.5 to +3.0 bias on `` makes the token sampleable exactly when the gate fires while leaving ordinary decoding untouched — the model never *must* retrieve, it is merely made willing. This is how a learned policy sits under a deterministic safety control. **Second, KV-cache continuity.** Pausing at the control token and resuming after evidence injection shifts position IDs mid-sequence, and models with relative position encodings (RoPE and friends) tolerate that poorly. We fine-tune on the exact `` continuation format so the model learns the shift, or recompute the cache from the last stable prefix on deep injections. Skip this and interleaving quietly degrades into confusion. **Third, retrieval is a typed tool surface.** The evidence channel is the output of an MCP-style tool with a JSON Schema contract: queries validate before executing, results carry provenance metadata, and the audit trail records every mid-generation lookup as a first-class event.

The gate needs its own calibration pass. On held-out generation traces, measure the fraction of gate firings that changed the downstream answer: gates that fire on trivia add latency without value; gates that fire too late let hallucinations through. We tune the uncertainty threshold so the *missed-retrieval rate* — fact-bearing spans written with no evidence behind them — stays below what the verification layer can absorb.

## 4. Data Integrity: Verifying Interleaved Facts Against the Source of Truth

Interleaved retrieval raises the bar on trust in both directions. More retrieval means more grounded content — but also more surface where a retrieved fact, a synthesized inference, and a hallucinated number can blur together in one document. A RIG answer is only as good as its ability to prove, sentence by sentence, which source of truth stands behind each claim.

**Claim decomposition.** Before an answer ships, a post-processor splits it into atomic claims — one verifiable proposition per unit: a balance, a date, a policy threshold. For financial content this is mechanical; every figure becomes a claim like *”the GL balance for cost center 4200 in June is $184,220.15.”*

**Two verifier classes.** Numeric and categorical claims verify *deterministically*: re-run the query behind the figure and compare against the live row — byte equality with the source of truth, not semantic similarity. Semantic claims (summaries, classifications, policy interpretations) go through a smaller entailment model that checks whether the claim is *entailed by the retrieved evidence block*. Passages carry provenance — table, row key, query, timestamp — so every verdict has a concrete referent:

“`json
{
“claim”: “GL balance for cost center 4200 is $184,220.15”,
“verdict”: “verified”,
“evidence”: {
“tool”: “ledger.balance”,
“query”: “SELECT SUM(amount) FROM gl WHERE cost_center=4200 AND period=’2026-06′”,
“row_key”: “gl|4200|2026-06”,
“retrieved_at”: “2026-09-06T09:14:22Z”
}
}
“`

Three verdicts, three dispositions: **verified** claims ship with their provenance attached; **unverified** claims trigger one re-retrieval before the system abstains rather than asserts; **contradicted** claims force the generator to revise the span — never to argue with the data. Row-level access controls from your MCP layer apply here too: a claim is only verifiable against rows the caller may see, keeping the integrity guarantee aligned with the security boundary.

This is where RIG earns its keep over RAG. A RAG citation can launder a hallucination — the model cites a chunk it never grounded its answer in. RIG verification is *executed*, not asserted: claim, query, and live row are all on record, and the whole interleaved trajectory replays from the audit log. Compliance teams do not need to trust the model; they need to re-run the math — and with claim-level provenance, they can.

## 5. Performance: Managing the Latency of Multiple Retrieval Round-Trips

Interleaving is a latency multiplier by construction: N retrieval round-trips add N tool executions and N model-resume steps to every response. The architecture only survives if you budget and hide that cost. Four levers, in the order we pull them:

**Cap the hops and make them count.** Every deployment carries a hard retrieval budget — typically 3 to 5 per answer — enforced in the decode loop so a runaway policy cannot inflate p95. Within it, prefer *fewer, wider* lookups to many narrow ones: one query fanning out across a date range beats four single-day queries, because round-trip latency tracks hop count, not bytes returned. Teach this preference in the RL pass: reward trajectories that satisfy the verifier in the minimum number of retrievals.

**Hide latency behind streaming.** Users need the *first* token fast, not the final one. Stream tokens as they decode, so a four-hop answer’s visible latency is the first hop plus the final segment, with intermediate stalls masked by what has already rendered. This turns a 6-second wall-clock answer into a roughly 1.5-second perceived one — the highest-ROI change in production RIG.

**Parallelize and prefetch.** When a segment needs three independent facts, fire the retrievals concurrently and join before resuming. Better: predict the next retrieval from current context — while the model writes the filler between evidence blocks, speculatively run a cheap embedding match on the probable next query and cache it. The decoder consumes the cache only if the trigger fires; otherwise the speculation is discarded. Classic prefetch for a stochastic consumer.

**Keep a semantic cache and a fast path.** Repeated reconciliations hit the same rows: key a semantic cache on the normalized retrieval query and skip the database on identical lookups. And route by query shape — an exact-identifier lookup (`invoice_id = 88132`) should never pay ANN latency when a primary-key hit on the indexed ledger costs under 10 ms versus 50–150 ms for embedding search. Expose both paths and let the generator’s query choose.

| Workload shape | Retrievals | Added p95 latency | Mitigation |
|—|—|—|—|
| Exact-ID lookups | 1–2 | +20–60 ms | PK-index path, semantic cache |
| Multi-hop analytical | 3–5 | +0.5–2.5 s | Parallel fan-out + streaming |
| Full reconciliation | 5–10 | +3–8 s | Budget caps, async verification, human-in-loop review |

The governing rule is the one for any distributed system with a nondeterministic core: *measure per-hop latency in production, set budgets from the tail, and degrade into a handoff, never into a hang.* When the budget is exhausted and the answer is still unverified, surface the partial result with its verification gaps labeled — an honest exception report beats a late, confident guess.

## 6. Case Study: RIG for Complex Financial Reconciliation Tasks

The pattern earns its complexity where questions are numeric, multi-source, and compliance-bound — which is precisely financial reconciliation. Consider the workload we have built RIG for repeatedly: closing the books when the general ledger, the subledger, and the bank statement disagree and every variance must be explained.

A classic RAG approach fails here structurally. The question is not “find the passage about reconciliation” — the answer lives in no document. It lives in millions of rows across three systems of record, and it is different every single day. With RIG, the generator drives the reconciliation like an analyst would, one verified step at a time:

1. **Open.** The generator emits its run summary and ``s the period’s unmatched transaction set — a parameterized query against the subledger’s exceptions view.
2. **Compare.** The model classifies each exception (timing difference, keying error, missing posting) and, where classification depends on policy thresholds, emits a second `` for the approval matrix and tolerance rules.
3. **Recompute.** For every candidate adjustment, the generator does not trust its arithmetic — it ``s the supporting rows and recomputes totals against the live GL, so each proposed entry ties to source rows byte-for-byte.
4. **Verify.** The Section 4 verifier re-checks every figure in the draft exception report against the rows that produced it; contradicted figures force a revision cycle before the report is released.
5. **Compose.** The final output is an exception report in which every number carries a row-level provenance pointer, and the full interleaved trajectory — every query, result set, and revision — is replayable from the audit log for the external auditor.

The measurable outcome is where errors occur. Pre-RIG, single-shot retrieval produced plausible-looking exception reports whose recomputed totals disagreed with the ledgers on a meaningful share of flagged items — the classic confidently-wrong profile. With RIG, the deterministic verifier catches numeric discrepancies at generation time rather than month-end review, and tolerance-policy lookups cut false-positive classifications. Across deployments, manual review time on the exception queue drops by roughly two-thirds, and every figure in the output is re-derivable from a logged query, not defensible in a meeting.

That last property is the real product. A reconciliation no one has to re-audit by hand is not a faster report; it is a different control. RIG turns the generator from a summarizer of whatever it retrieved into an analyst that retrieves on demand and verifies what it writes.

## Conclusion: From Retrieved Answers to Verified Decisions

RIG is the answer to a simple observation: generation is a process, not an event, and retrieval belongs inside it. A generator that can decide mid-sentence that its evidence has run thin, reach into the live system of record, and resume with verified facts produces answers that RAG — with its single early bet — structurally cannot. It costs a decoding loop, control-token fine-tuning, a verification layer, and a latency budget; it buys grounded, replayable, audit-ready output where being wrong is expensive.

If you are building grounded AI against real financial or operational data — Oracle, MS SQL, Postgres, Snowflake, your CRM — and you want an architecture where every number in an answer is traceable to a row, talk to the OnlineCloud architects. We map your data, pick the model class, define the eval that means “done,” and ship a vertical slice in weeks — evaluated, observable, and built by the same senior engineers who scope it, with no offshore handoff. [Contact the OnlineCloud team](https://www.onlinecloud.dev/) or reach us at contact@onlinecloud.dev to explore whether RIG is the right loop for your hardest questions.