Optimizing Tool-Calling for Large Toolsets: Discovery & Filtering Patterns

# Optimizing Tool-Calling for Large Toolsets: Discovery & Filtering Patterns


*An infrastructure architect’s field guide to keeping MCP servers fast, cheap, and secure as the tool count grows from ten to a thousand.*

—

## 1. Introduction: The Tool-Inflation Problem

Every MCP deployment starts the same way. A team wires ten tools into a server — `search_orders`, `get_invoice`, `create_ticket` — and the agent feels magical. Six months later the same server exposes forty tools across three business domains, and the agent has quietly gotten dumber.

This is **tool inflation**, and it is a structural problem, not a prompt-tuning problem. When you serialize every tool definition into the model’s context on every turn, three things degrade simultaneously:

– **Context pressure.** Each tool costs 80–400 tokens once you include its name, description, JSON Schema, parameter docs, and examples. Forty tools is 8–16K tokens of *baseline* tax before the user has said a single word. At 200 tools you have spent the entire context window on capabilities the request will never use.
– **Selection accuracy.** Models do not scan a flat list of 200 tools the way a human scans a menu. Research on tool-selection benchmarks consistently shows accuracy decaying as the candidate set grows — the model starts confusing `delete_record` with `archive_record`, or hallucinating parameters from a neighbouring schema. More options means more ways to be *almost* right.
– **Operational drag.** Every tool in the global namespace is a tool that must be versioned, authorized, rate-limited, and audited. A monolithic toolset couples unrelated domains (billing, HR, infrastructure) into a single blast radius.

The old workaround — “just fine-tune the model on your tools” — does not scale either. Tools are the most volatile part of any system; they change weekly. You cannot retrain on every schema bump.

The durable fix is architectural. Instead of presenting the *entire* toolset, present a **small, intent-relevant slice** of it, chosen at request time. That single idea decomposes into three disciplines: **semantic discovery**, **dynamic filtering**, and **cached dispatch**. Together they turn a 200-tool server into something the model sees as a 6-tool server — while retaining access to all 200.

This post walks through the pattern end to end, including a production Go reference implementation.

—

## 2. Semantic Tool Discovery: Two-Stage Retrieval for Tool Registration

The insight borrowed from information retrieval is simple: **do not stuff the catalog into the prompt; build an index over it.** Tool descriptions are natural language, so the same embedding-based retrieval that powers RAG over documents works over your tool registry.

But naive single-stage retrieval is not enough for toolsets, because tools have both *semantic* descriptions and *structural* constraints. The pattern that holds up in production is **two-stage retrieval**.

**Stage 1 — Recall (vector search).** At registration time, each tool is embedded from a normalized document: name (split on snake_case), description, parameter names, and the domain/capability tags attached by the author. These vectors live in an index (pgvector, Qdrant, or an in-process HNSW). At request time the user’s intent — or a compact rewrite of it — is embedded and used to pull the top-K candidates, typically K = 20–30. Embedding models are cheap and fast; this stage is about *recall*, not precision, so the threshold is deliberately loose.

**Stage 2 — Precision (rerank + structural filter).** The K candidates are reranked by a cross-encoder or a lightweight LLM call that reads the actual request and the candidate schemas. This is where near-misses are eliminated. Crucially, Stage 2 *also* applies non-semantic gates: tenant scopes, feature flags, and RBAC (see §5). The output is the final tool slice, usually 4–8 tools.

Why two stages and not one? Because a single embedding model conflates *similarity* with *suitability*. “Export all customer records” and “delete all customer records” embed nearly identically but have wildly different policy implications. The reranker sees the whole request, not just the query vector, and can separate them.

A subtle but high-leverage detail: **register tools with the vocabulary your users actually type.** If finance asks “who owes us money,” the tool `ar_aging_report` will never be retrieved unless its registration document includes synonyms like “outstanding invoices,” “receivables,” and “who owes.” Treat the tool description as a *search document*, not a docstring.

“`text
Stage 1: intent → embed → ANN search → top-30 tool IDs
Stage 2: (request, top-30 schemas) → rerank + RBAC filter → 6 tool schemas
Stage 3: 6 schemas → model context → tool call
“`

The registry itself should be **append-only and versioned**. Never mutate a tool document in place; publish a new version and let the index pick it up on the next refresh. This gives you rollback, A/B comparison of descriptions, and a clean audit trail of *what descriptions the model was actually shown* when it made a bad call — the single most useful debugging artifact in an agent system.

—

## 3. Dynamic Filtering: Pruning the Toolset by Intent and State

Semantic relevance answers *what the request is about*. It does not answer *what the session is allowed to do right now*. Dynamic filtering is the second axis: prune by **session state**, not just by topic.

Three filter classes do most of the work:

**Intent-class gating.** A cheap upstream classifier (or a router LLM, or even a keyword/regex fast path) tags the turn as *read*, *write*, *admin*, or *transactional*. Read-only turns get read-only tools. This is not merely a context optimization — it is a safety primitive. The model literally cannot call `wire_transfer` during a question-answering turn because that schema was never in its context.

**Stateful workflow filtering.** Agents execute workflows, and workflows are state machines. After `create_order`, the relevant tools are `add_line_item`, `apply_discount`, `submit_order` — not the other 197. Keep a per-conversation `allowed_tools` set that the orchestrator transitions as the workflow advances. This collapses the candidate space dramatically and eliminates an entire class of “the agent skipped a step” bugs.

**Capability and environment filtering.** Feature flags, model tier, deployment region, and cost ceilings all prune the set. A tool backed by an expensive GPU job should not appear when the session is running under a $0.05/request cost budget.

The result is a **layered predicate** over your registry:

“`text
visible = INDEX(intent)
∩ ALLOWED(session_state)
∩ PERMITTED(principal, RBAC)
∩ ENABLED(flags, region, budget)
“`

Two implementation rules matter more than the design itself. First, **filter before you serialize**, never after — the expensive part is tokens in context, and post-hoc instruction to “ignore tool X” does not remove it from the window. Second, **make the filter deterministic and loggable.** When the agent fails to find a tool it “should” have had, you want to diff the filter input (state, principal, flags) against the expected set, not guess.

A practical guardrail: cap the visible set at a hard ceiling (we use 12). If the layered predicate returns more, force the reranker to trim further. A stable, small tool menu is worth more than marginally better recall.

—

## 4. Performance: Latency in the Tool-Call Hot Path

Here is the trap. You added an embedding call and a reranker to the critical path between the user’s message and the model’s first token. If you are not careful, you have traded a context-cost problem for a latency problem.

The budget matters. For interactive agents, retrieval must fit in a **single-digit-millisecond to low-double-digit-millisecond** window, because the model’s own inference time (often 300–2000 ms) dwarfs everything else and the user perceives the sum.

Five techniques keep semantic search off the latency floor:

**Embed the intent once, reuse it.** Cache the query embedding keyed by a normalized-intent hash. Turns within a workflow often share intent; you should not re-embed “add another item” five times.

**Do ANN in-process, not over the network.** An HNSW index for 200–5,000 tools fits comfortably in memory (tens of MB). A network hop to a vector database adds 3–15 ms of pure RTT per call. For tool counts in the thousands, an in-process index with periodic refresh beats a managed vector DB on the hot path.

**Warm the index at startup.** Deserialize and build the HNSW graph before the server reports ready. Never build it lazily on the first request.

**Make the reranker optional and bounded.** A cross-encoder reranking 30 candidates is ~2–8 ms on CPU; an LLM reranker is 100 ms+. Default to a small cross-encoder, and fall back to pure vector ranking if the reranker’s p99 exceeds your budget. Degrade, don’t stall.

**Cache the whole slice.** The single biggest win: most requests in a session map to the same `(intent_class, session_state, principal)` tuple → the same tool slice. Cache the *resolved slice* keyed by that tuple and invalidate on state transition or registry version change. A cache hit is a map lookup — sub-microsecond — and it removes retrieval from the hot path entirely.

The mental model: **retrieval is an optimization pass, and optimizations have caches.** Instrument it like one. Track `slice_cache_hit_ratio`, `ann_p50/p99`, and `rerank_skipped_ratio`. If your cache hit ratio is under ~70% in a workflow-heavy agent, your state keying is too coarse or too fine, and that metric will tell you which.

—

## 5. Security: Tool-Level RBAC in Dynamic Discovery

Dynamic discovery introduces a security anti-pattern that is easy to miss: **the retrieval layer becomes a confused deputy.** The index knows about every tool. If the retrieval path does not carry the principal’s identity and enforce policy, a cleverly worded request can surface a tool the user was never meant to see — and once it is in context, the model may call it.

Four rules keep discovery safe:

**Enforce RBAC before serialization, not at execution only.** Execution-time authorization is necessary but insufficient. If an unauthorized schema enters the context, you have already leaked the existence, shape, and semantics of an internal capability. Filter the index query by principal scope as a *first-class* predicate, ideally at the vector-store level (metadata filter on tenant/role) so unauthorized tools never even become candidates.

**Treat the tool index as an authorization surface.** A vector index partitioned by tenant is not just a performance optimization; it is a security boundary. Cross-tenant bleed in a shared ANN index is a real risk — always filter by tenant metadata, and verify with a false-positive test (attempt to retrieve a foreign tool by name).

**Re-authorize at invocation.** Discovery decides *what the model sees*; the dispatcher decides *what actually runs*. Every call re-checks the principal against the tool’s policy at execution time, so a stale cache or a smuggled tool name cannot bypass policy. This is the “verify at the edge” principle — never trust the filter alone.

**Audit the slice, not just the call.** Because the slice is deterministic and logged, you can answer the question auditors actually ask: *what tools was this principal shown, and could they have invoked each one?* Keep the resolved slice ID on every trace span. This turns a fuzzy “the model did something odd” investigation into a precise replay.

The composed posture: **RBAC filters the index (prevention), the dispatcher re-checks (enforcement), and the slice log records (evidence).** Three independent controls; a failure in any one does not silently become a breach.

One more: prompt injection through tool *descriptions*. If tool metadata is authored or imported from semi-trusted sources, sanitize it before it enters the registration document. A description is model-readable text — treat it as untrusted input, because that is exactly what it is.

—

## 6. Reference Implementation: A Go MCP Tool Dispatcher with Caching

The following is a compact but production-shaped sketch of the pattern. It shows a registry, an in-memory vector-ish index (interface-based, so you can drop in HNSW or pgvector), RBAC filtering before serialization, and a slice cache keyed by the security-and-state tuple. It is deliberately dependency-light so the architecture is legible.

“`go
package dispatcher

import (
“context”
“crypto/sha256”
“encoding/binary”
“errors”
“math”
“sync”
“time”
)

// Tool is a registered MCP capability.
type Tool struct {
Name string
Description string
Tags []string
Schema []byte // raw JSON Schema, serialized to the model
Required []string // required RBAC scopes
Embedding []float32
}

// Principal is the authenticated caller.
type Principal struct {
ID string
Scopes map[string]bool
Tenant string
}

func (p Principal) Has(scope string) bool { return p.Scopes[scope] }

// Index is the pluggable similarity backend (HNSW, pgvector, brute force).
type Index interface {
Search(ctx context.Context, query []float32, k int, tenant string) ([]string, error)
}

// Registry holds the append-only, versioned catalog.
type Registry struct {
mu sync.RWMutex
version string
tools map[string]Tool
index Index
embed func(context.Context, string) ([]float32, error)
}

// Filter is one layer of the dynamic predicate.
type Filter func(Tool) bool

// maxVisible is the hard ceiling on tools serialized into the model context.
const maxVisible = 12

// Slice is the resolved, cacheable output of discovery.
type Slice struct {
Version string
Tools []Tool
}

// keyTuple is what the resolved slice depends on. If any of these change,
// the slice must be recomputed — nothing else invalidates it.
type keyTuple struct {
Intents uint64
State uint64
Principal string
Tenant string
Version string
}

// Dispatcher is the hot path: discovery -> filter -> cache -> serialize.
type Dispatcher struct {
reg *Registry
cache sync.Map // keyTuple -> *Slice
}

// Resolve returns the tool slice to place in the model’s context.
func (d *Dispatcher) Resolve(
ctx context.Context,
p Principal,
intent string,
stateHash uint64,
layerFilters []Filter,
) (*Slice, error) {
// 1. Normalize intent and derive the cache key.
q, err := d.reg.embed(ctx, intent)
if err != nil {
return nil, err
}
key := keyTuple{
Intents: hashFloats(q),
State: stateHash,
Principal: p.ID,
Tenant: p.Tenant,
Version: d.reg.version,
}
if v, ok := d.cache.Load(key); ok {
return v.(*Slice), nil // cache hit: sub-microsecond
}

// 2. Stage 1 — recall via ANN, tenant-partitioned at the index layer.
ids, err := d.reg.index.Search(ctx, q, 30, p.Tenant)
if err != nil {
return nil, err
}

// 3. Stage 2 — RBAC + dynamic filters applied BEFORE serialization.
// Unauthorized tools must never reach the model context.
candidates := make([]Tool, 0, len(ids))
for _, id := range ids {
t, ok := d.reg.lookup(id)
if !ok {
continue
}
if !authorized(p, t) {
continue // prevention: filter the index result, don’t just block at call time
}
if !all(layerFilters, t) {
continue
}
candidates = append(candidates, t)
}

// 4. Enforce the hard ceiling on visible tools.
if len(candidates) > maxVisible {
candidates = candidates[:maxVisible]
}

slice := &Slice{Version: d.reg.version, Tools: candidates}
d.cache.Store(key, slice)
return slice, nil
}

// Invoke re-authorizes at execution time. Discovery decides what the model
// sees; the dispatcher decides what actually runs. Never trust the filter alone.
func (d *Dispatcher) Invoke(
ctx context.Context,
p Principal,
name string,
args []byte,
) ([]byte, error) {
t, ok := d.reg.lookup(name)
if !ok {
return nil, errors.New(“unknown tool”)
}
if !authorized(p, t) {
return nil, errors.New(“forbidden: principal lacks required scopes”)
}
return d.call(ctx, t, args) // rate limits, audit span, execution
}

func authorized(p Principal, t Tool) bool {
for _, s := range t.Required {
if !p.Has(s) {
return false
}
}
return true
}

func all(filters []Filter, t Tool) bool {
for _, f := range filters {
if !f(t) {
return false
}
}
return true
}

func hashFloats(f []float32) uint64 {
h := sha256.New()
var b [4]byte
for _, x := range f {
binary.LittleEndian.PutUint32(b[:], math.Float32bits(x))
h.Write(b[:])
}
return binary.LittleEndian.Uint64(h.Sum(nil))
}
“`

Four things to notice in the code, because they encode the whole thesis:

1. **`keyTuple` defines invalidation.** The slice is a pure function of intent embedding, session state, principal, tenant, and registry version. Cache correctness falls out of the key — no manual invalidation logic to get wrong.
2. **RBAC runs inside `Resolve`, before the slice is built.** `authorized()` is a discovery-time gate, not just a call-time check. This is the “prevention” control.
3. **`Invoke` re-authorizes.** Even a cache hit or a crafted tool name cannot bypass policy. This is the “enforcement” control.
4. **The registry is versioned.** `Version` in the key means a tool-description update invalidates every cached slice atomically — and each `Slice` records the version it was resolved against, giving you evidence for audits and replays.

In a real deployment you would add a bounded LRU instead of a bare `sync.Map` (to cap memory), swap the `Index` for an HNSW implementation behind the same interface, and emit the `keyTuple` and slice ID onto the trace span. The five latency techniques from §4 are all refactors of `Resolve`, not rewrites — embed caching, in-process ANN, a bounded cross-encoder rerank between steps 2 and 3, and cache-hit telemetry layer in cleanly.

—

## The Through-Line

Tool inflation is not solved by better prompts or bigger context windows. It is solved by treating your tool registry like a **search index with a security policy attached** — retrieve narrowly, filter by intent *and* state *and* principal, cache the resolved slice, and re-authorize at the edge.

Do that, and a server with 200 tools behaves like one with six: lower latency, higher selection accuracy, a smaller attack surface, and an audit trail precise enough to explain every decision the agent made. Discovery and filtering are not optimizations you bolt on later. In any serious MCP deployment, they *are* the architecture.

—

**Building an MCP server and hitting the wall at 30 tools?** OnlineCloud designs retrieval-backed tool registries, RBAC-enforced discovery, and observable agent runtimes — wired into the databases and CRMs you already run. We ship evaluated, auditable systems in 6–10 weeks, with senior engineers only.

**[Start your MCP project →](https://www.onlinecloud.dev/contact)** · **[See our agent and MCP work](https://www.onlinecloud.dev/work)**