Building a Custom Tool Discovery Service for Large-Scale Agent Fleets

# Building a Custom Tool Discovery Service for Large-Scale Agent Fleets


*By the infrastructure engineering team at OnlineCloud — a US-based Agentic AI & MCP engineering studio*

—

## 1. Introduction: The 1,000-Tool Problem — Scaling Beyond Model Context Limits

Every agent platform hits the same wall. A few tools — ticket lookup, refund action, warehouse query — and the model handles them effortlessly. Then the integrations multiply: CRM, ERP, billing, each customer’s bespoke connectors. Before long the fleet can reach a thousand tools, and the platform collapses under the weight of its own capability.

The collapse is quantitative. A typical MCP tool definition costs 200 to 800 tokens serialized, so a thousand tools means 300,000 to 700,000 tokens of schema if you mount them all into context — beyond most production models’ entire window. Even where it fits, it is ruinous: function-calling accuracy degrades as the visible tool count passes a few dozen, the model confuses similarly named tools, and every request pays latency and dollar cost for schemas it never uses.

The mistake is treating the context window as the catalog. It is a *working set* — small, relevant, loaded on demand. A thousand-tool fleet needs a **discovery layer**: infrastructure that answers, per task, *which of these capabilities do I need, and how do I reach them?* Search engines made this shift decades ago — when the corpus outgrows any client, you stop shipping the corpus and start shipping retrieval.

Below is the blueprint we use at OnlineCloud, built on Model Context Protocol (MCP): a semantic search layer, a centralized tool catalog, dynamic runtime registration, tool-level RBAC, and HNSW-based performance engineering. The recurring theme: **discovery is a control-plane service, never a prompt-engineering trick.**

## 2. Tool Discovery: Implementing a Semantic Search Layer for Tool Registration

Tool selection is retrieval wearing an LLM costume. The naive implementation asks the model to choose from everything; the correct one embeds every tool into a vector space and retrieves the top-k candidates for the task, letting the model choose among a handful of well-matched options.

Registration is where the semantic layer is born. When an MCP server starts, it declares its tools via the `tools/list` handshake. The discovery service intercepts the declaration and turns each tool into a searchable document — the **name**, the **description**, and the **input schema**, whose field names and enums carry latent semantic signal — embedded with a fixed text-embedding model and stored beside the tool’s metadata. Freeze the embedding model per index version — re-embedding is a scheduled event, not a surprise.

The query side mirrors registration. At task time, the orchestrator embeds the agent’s goal — user request, current sub-task, safe conversation context — into the same space and runs nearest-neighbor search. Production retrieval is never pure vectors: semantic search misses exact identifiers, so we run a **hybrid** pipeline — dense vectors for meaning, BM25-style lexical match for names and IDs, and metadata filters for hard constraints — fused by reciprocal rank fusion. Cold start is solved with logs and description-quality review: precise descriptions retrieve; “do refund stuff” does not.

One rule is non-negotiable: **discovery is advisory; invocation is authoritative.** Retrieval proposes candidates; every call is validated against the catalog at invoke time — schema, version, endpoint, authorization. Never conflate the two.

## 3. Architecture: Designing a Centralized MCP Tool Catalog with Metadata Tags

The semantic layer needs a home: a **centralized tool catalog** — a control-plane service that owns each tool’s lifecycle and records what exists, who owns it, who may call it, and where it runs.

The data model separates a tool’s *definition* from its *deployment*. A **definition** is a versioned record: name, description, schemas, embedding, tags, owner, risk tier, data classification. A **deployment** is a live instance: transport, resolved endpoint, health state, and the schema hash currently served. Three healthy deployments of one definition are a single logical capability the gateway load-balances across; when a server’s live schema drifts from the catalog — teams deploy faster than they document — the hash mismatches and the catalog raises a drift alert instead of serving stale contracts.

**Tags are the metadata spine**, drawn from controlled vocabularies because free-form tags decay into noise within a quarter:

“`json
{
“domain”: “billing”,
“risk_tier”: “high”,
“access”: “write”,
“data_class”: “pii”,
“cost_tier”: 2,
“owner”: “team-billing”,
“status”: “active”
}
“`

Tags feed the retrieval filters, define the RBAC policy surface (Section 5), and let platform teams answer fleet-level questions — “which tools touch PII?” — in one query. Risk tier and data classification are load-bearing: a discovery service that cannot flag tools touching expensive, regulated data cannot protect the budget or the compliance posture.

Every catalog mutation is auditable, so the change log doubles as the compliance record for why an agent could call that tool on a given day. Run it as a stateless API over Postgres with pgvector (comfortable at tens of thousands of definitions) and treat availability as critical: when discovery is down, agents fall back to a cached, authorized toolset — never an unconstrained mode.

## 4. Dynamic Registration: How Agents Discover and Connect to New Tools at Runtime

A static catalog is a glorified OpenAPI registry. The payoff is **dynamic registration** — agents picking up new tools, versions, and entire MCP servers mid-task, with no redeploys or prompt rewrites.

The runtime flow: the orchestrator builds a **discovery query** from the current step; the catalog runs **hybrid search scoped by the agent’s authorization** and returns candidate tool stubs with deployment metadata; the gateway **reconciles schemas** (via schema hash) so the model only sees contracts the executor will enforce; the agent opens a **lazy, pooled MCP session** to the resolved endpoint bound to a short-lived, audience-scoped credential (Section 5); and **feedback** reinforces success while repeated schema failures or timeouts demote a tool and page its owner.

New tools join the fleet the moment their server completes registration — a **push event, not a polling chore**. Servers register via signed webhook; the catalog validates the manifest, embeds the definitions, and publishes a change event that subscribed agents use to warm connections proactively.

The hard part is **version transitions**. We version tools semantically and gate rollout through the catalog: when `create_refund` v4 publishes, v3 stays active for in-flight sessions while v4 serves new discovery queries, with a deprecation deadline attached. A mid-task agent finishes its current call against v3 — the session binding is pinned at connection time — while the next step discovers v4. No mid-call contract swaps, no orphaned sessions. Revocation is the same mechanism with urgency: the tool vanishes from discovery and sessions drain.

Graceful degradation completes the design. Each agent role pins a **baseline toolset** — a small, curated, always-authorized set covering most routine tasks. If the catalog is unreachable, agents run against baseline and flag the session as degraded instead of failing open.

## 5. Security: Enforcing Tool-Level RBAC in Dynamic Discovery Environments

Dynamic discovery is a double-edged sword: done right, it shrinks the model’s attack surface to the tools a task needs; done naively, it becomes a **capability disclosure service** — every query leaks tool existence, and every surfaced tool is one a prompt injection could aim.

The founding rule: **never return a tool the caller is not authorized to invoke.** Authorization applies at query time, inside the catalog, as a hard filter — never as post-processing. If an agent cannot call a tool, it must not know it exists. That is the difference between filtering and *scoping*: filtering hides results after the fact; scoping constrains the search space so even the embeddings of unauthorized tools never cross the trust boundary.

Authorization is two-layered. **RBAC** maps principals — agent identities, or the humans whose authority an agent inherits — to roles, then roles to allowed tool IDs or tag patterns. **ABAC** adds attribute conditions: “only 09:00–18:00,” “only cost tier ≤ 2,” “only when the delegated user holds `refund_approve` in the source system.” Tool tags feed both layers — the Section 3 vocabularies are security infrastructure, not documentation.

“`json
{
“principal”: “agent:refund-workflow”,
“effect”: “allow”,
“match_tags”: { “domain”: “billing”, “access”: “read” },
“conditions”: {
“data_class”: [“internal”, “pii”],
“cost_tier”: { “lte”: 2 },
“delegated_user_entitlement”: “refund_approve”
}
}
“`

Connections minted through dynamic registration carry **capability-bound credentials**. Instead of a fleet-wide API key, the catalog issues a short-lived token per session whose audience is exactly the resolved endpoint, scopes exactly the authorized tool IDs, and TTL outlives the task by minutes, not months. A leaked token is useless against every other tool in the fleet — blast radius is one session, one endpoint.

Three threat models deserve attention. **Prompt injection:** scoped discovery plus a whitelist-only tool view means an injected instruction cannot aim the agent at tools it was never shown, and discovery results must never widen privileges mid-task. **Endpoint spoofing:** the catalog resolves endpoints only from its own registration records and agent traffic flows through the gateway, so outbound calls to unregistered endpoints are blocked at the transport layer. **Discovery as an oracle:** discovery queries are logged, rate-limited, and audited like invocations — *who searched for what they could not have* is often the earliest signal of a compromised agent.

## 6. Performance: Optimizing Discovery Latency with Vector Caching and HNSW

Discovery sits on the hot path — a single turn may issue several queries as sub-tasks decompose. Our budget is unforgiving: **p50 under 10 ms, p95 under 30 ms**, end-to-end — an order of magnitude faster than the LLM call around it, so discovery is never the bottleneck.

Latency engineering is a caching problem first, an index problem second. Fleets repeat themselves — same roles, task archetypes, and hot tools hour after hour. We cache in three tiers. **L1, in-process:** keyed by (principal scope, task class, embedding bucket) so the top-20 tools for “refund escalation” resolve in microseconds. **L2, distributed:** a shared cache so a cold orchestrator inherits what its siblings computed minutes ago. **Index:** the vector index, hit only on genuine misses — the long tail of novel tasks. Invalidation is event-driven, not TTL-approximate: registration, schema change, and revocation events invalidate affected keys synchronously, because a cache serving a revoked tool for thirty extra seconds is a security bug wearing a performance costume.

For index misses, **HNSW** earns its place. It builds a hierarchical navigable small-world graph over the embedding space and walks it coarse-to-fine, trading a controllable slice of recall for roughly two orders of magnitude less latency than brute force. At 50,000 definitions with 1,024-dimension embeddings, exhaustive scan costs tens of milliseconds; a tuned HNSW index answers in 2–8 ms at 95–99% recall.

“`json
{
“index”: “hnsw”,
“dims”: 1024,
“m”: 16,
“ef_construction”: 200,
“ef_search”: 128,
“quantization”: “scalar_int8”
}
“`

Configuration discipline matters: `m` and `ef_construction` trade build cost for graph quality, `ef_search` trades query latency for recall — tune it per traffic class, since a batch job tolerates higher latency than an interactive chat turn. Memory is the silent killer: float32 embeddings for 100,000 tools × 1,024 dimensions exceed 400 MB before a single tag; scalar int8 quantization cuts that fourfold with negligible recall loss.

The subtle trap is **filtered search**. HNSW shines on pure nearest-neighbor queries, but metadata filters (namespace, risk tier, authorization scope) interact badly with graph traversal — a walk can exhaust its budget where every neighbor fails the filter. Our approach is two-phase: small per-scope indexes for common hot scopes, enlarged `ef_search` with post-filtering for the long tail. Recall is a measurable SLO: periodic audits compare approximate results against exhaustive ground truth, and a filter change that silently drops recall from 97% to 91% is a paged incident — a missed tool is a failed task.

Finally, measure the whole path: discovery telemetry — cache hit rate, index latency, recall audits, invalidation lag — belongs on the same dashboard as invocation metrics, because slow discovery means slow agents.

—

## Conclusion: Discovery Is the Control Plane of the Agent Fleet

The thousand-tool problem is not a context-window problem; it is an architecture problem. Teams that win with agents at scale are not the ones that squeeze the most schemas into a prompt — they are the ones that built a control plane where **capability is registered once, discovered semantically, authorized explicitly, and invoked through validated contracts.** Semantic search makes a thousand tools navigable; the catalog makes them governable; dynamic registration makes them deployable; tool-level RBAC makes them safe; vector caching over HNSW makes them fast enough that agents never feel the machinery between them and the work.

That is the architecture we ship at OnlineCloud, wired into real estates — Oracle, SQL Server, Postgres, Snowflake, CRMs — with audited authorization and production observability from day one.

**Ready to scale your agent fleet past the context window? Talk to the OnlineCloud architects.** We are a US-based engineering studio — MCP-native, senior engineers only, no offshore handoff — and we deliver evaluated, observable discovery infrastructure in production within 6–10 weeks. Contact us at [contact@onlinecloud.dev](mailto:contact@onlinecloud.dev) or visit [onlinecloud.dev](https://www.onlinecloud.dev/) for a free initial consultation. We respond within 24 hours.