Building a Custom Tool Discovery Service for Billion-Scale Agent Fleets

# Building a Custom Tool Discovery Service for Billion-Scale Agent Fleets


*An infrastructure architect’s field guide to tool cataloging, semantic retrieval, and dynamic context injection across fleets that expose millions of MCP tools.*

—

## 1. The Context-Window Limit Problem

Every agent platform starts the same way. You have ten tools, you paste their JSON schemas into the system prompt, and the model picks the right one. It feels like magic. Then you have fifty tools. Then a hundred and fifty. Then the platform team tells you that internal teams want to register their own tools, and suddenly the catalog is a shared surface with thousands of entries growing weekly.

At that point the naive architecture collapses on three fronts at once, and the failure modes are not subtle.

**First, the token budget dies.** A well-described MCP tool schema — name, description, parameters, enums, nested objects, examples — averages 200 to 800 tokens once serialized. At 150 tools you are consuming 30k–120k tokens *per request* before the model has read a single word of the actual task. Even with a 200k-token window, you have converted the majority of your context into a static manifest that the model must re-read on every turn. Cache hits help, but prompt-caching only reduces cost; it does not reduce the *attention* cost. Every extra schema is one more distractor competing with the user’s actual intent.

**Second, selection accuracy degrades non-linearly.** This is the part teams underestimate. Models do not degrade gracefully as the tool list grows — they degrade sharply. Research on tool-selection benchmarks consistently shows accuracy falling off a cliff well before the list reaches triple digits. The mechanisms are predictable: near-duplicate tools (`get_customer` vs. `fetch_client` vs. `lookup_account`) become indistinguishable; long lists push the model toward shallow heuristics like recency and lexical overlap; and instruction-following budget gets spent on *disambiguation* rather than execution.

**Third, the fleet becomes ungovernable.** When the manifest is baked into the prompt, every new tool is a global change. One team shipping a tool means every agent in the fleet pays for it in tokens and accuracy. There is no blast radius control, no way to scope tools per tenant, and no signal telling you which tools are ever actually used. You have built a monolith disguised as a tool catalog.

The architectural answer is to stop treating the tool catalog as *prompt content* and start treating it as a *queryable service*. The model should receive a small, task-specific slice of the catalog — retrieved at runtime, ranked by relevance, and injected dynamically. Everything else lives behind an index. That reframing is the entire subject of this post.

—

## 2. Tool Discovery: A Semantic Search Layer for Tool Cataloging

The core insight is that *tool selection is a retrieval problem before it is a reasoning problem*. If you can retrieve the top twenty semantically relevant tools for a given task, the model’s job shrinks from “search a haystack of 150,000” to “pick from a shortlist of 20.” That is a problem today’s models solve reliably.

So the first component you build is not an agent feature — it is a search index. Specifically, a vector index over a normalized representation of every tool in your catalog.

**What gets embedded.** You do not embed the raw JSON schema. You embed a *synthetic document* assembled from the fields that carry retrieval signal:

“`python
def build_tool_document(tool: McpTool) -> str:
return “\n”.join([
f”name: {tool.name}”,
f”server: {tool.server_id}”,
f”description: {tool.description}”,
f”tags: {‘, ‘.join(tool.tags)}”,
f”parameters: {format_param_signature(tool.input_schema)}”,
f”examples: {tool.usage_examples}”,
])
“`

The `tags` and `description` fields do the heavy lifting. A tool named `oc.sql.oracle.run_query` tells a user nothing; tagged `[“oracle”, “sql”, “read-only”, “finance”]` with a description like “Execute a read-only SQL query against the Oracle financial warehouse, returning up to 10k rows” it becomes trivially retrievable.

**Metadata is a first-class citizen.** Every catalog entry carries structured metadata alongside its embedding: `server_id`, `owner`, `auth_scope`, `cost_class`, `latency_p99`, `side_effects` (read/write/destructive), `tenant_allowlist`, and `version`. This metadata serves two purposes: it enables *hard filters* that run before vector search (a tenant should never retrieve a tool they are not authorized to call), and it enables *post-retrieval re-ranking* (prefer low-latency, read-only tools when the task does not require writes).

**The retrieval pipeline.** A query arrives — the user’s task, or better, a condensed intent representation produced by a cheap model — and flows through:

1. **Hard filter** on `tenant_allowlist`, `auth_scope`, and `enabled` flags.
2. **Vector search** (HNSW, covered in §5) to get the top ~200 candidates.
3. **Hybrid re-rank** combining dense similarity with lexical BM25 over the tool documents. Dense handles paraphrase; lexical handles exact API names and error codes.
4. **Metadata boost**, adjusting scores by cost class, latency, and historical success rate for similar queries.

The output is a ranked list. The agent never sees it directly.

—

## 3. Dynamic Injection: Top-k Schemas Into the Agent Context

Retrieval gives you a ranked list of 200. The model gets *k* of them. Choosing *k* correctly is where most implementations get sloppy.

**k is a budget, not a constant.** A hard-coded `k=20` wastes context on simple tasks and starves complex ones. The right approach is to express the decision as a token budget: the injection layer fills a context allotment (say 4k tokens for tool schemas) with the highest-ranked tools until the budget is exhausted. A trivial lookup might fit 5 tools; a multi-step orchestration might fit 40. The invariant is the budget, not the count.

**Inject a tiered representation.** Full schemas are expensive. A better design injects a two-tier payload:

– **Tier 1 — Signatures.** Compact one-line signatures for the top 40 tools (~30 tokens each). This gives the model *awareness* of the broader shortlist without the full cost.
– **Tier 2 — Full schemas.** Detailed schemas for the top 5–10, expanded on the assumption these are the execution candidates.

This mirrors how a human engineer works: they skim a table of contents, then read the two chapters they need.

**Re-retrieve mid-trajectory.** A single injection at turn zero is wrong for multi-step agents. The task evolves; the tools needed at step one are not the tools needed at step five. Re-run discovery between planning phases, passing the accumulated scratchpad as part of the query so retrieval stays grounded in what the agent has actually learned. In practice this *reduces* total tokens because you never carry stale schemas forward.

“`python
def inject_tools(task, scratchpad, budget=4096):
query = condense_intent(task, scratchpad)
ranked = registry.search(query, top_k=200) # §2 pipeline
tier1 = ranked[:40]
tier2 = ranked[:8]
payload, used = [], 0
for tool in tier2: # full schemas first
cost = token_count(tool.full_schema)
if used + cost > budget:
break
payload.append(tool.full_schema)
used += cost
for tool in tier1[len(tier2):]: # signatures fill remainder
payload.append(tool.signature)
return payload
“`

**Guardrail: always inject a fallback.** If retrieval confidence is low (top score below a threshold, or a large score gap), inject a small set of universally-safe tools — a search tool, a clarification tool — rather than a confident-but-wrong shortlist. Silent retrieval failure is more dangerous than open uncertainty.

—

## 4. Architecture: A Centralized MCP Tool Registry

The registry is the control plane of the whole system. Treat it like a service mesh control plane: it owns identity, metadata, versioning, and the discovery API, while the tools themselves remain decentralized behind their own MCP servers.

**Core components.**

– **Ingestion service.** MCP servers register by pushing a manifest, or the registry polls each server’s `tools/list` on a schedule. Manifests are validated against a schema, normalized, and versioned. A tool that changes its input schema creates a new version; old versions remain resolvable for pinned agents.
– **Metadata store.** A relational database (Postgres is a sensible default) holding tool identity, ownership, tags, access policies, and version history. This is the source of truth; the vector index is a derived, rebuildable projection.
– **Vector index.** Recomputed incrementally as manifests change. Keys are stable tool IDs so an update is an upsert, not a rebuild.
– **Discovery API.** A single low-latency endpoint exposing `search(query, filters, budget)`. This is what agents and orchestration layers call. It returns ranked tool descriptors plus a confidence score.
– **Audit and telemetry sink.** Every discovery query and every tool invocation is logged with the agent ID, tenant, retrieved set, and selected tool. This closes the feedback loop — retrieval quality becomes measurable, and *unused tools become visible*.

**Why centralized, not per-agent.** A centralized registry gives you one place to enforce authorization, one place to measure usage, and one place to improve ranking for the entire fleet simultaneously. A per-agent catalog means N divergent indexes and no shared learning signal. The trade-off is a latency-critical dependency, which is why the discovery API must be aggressively cached and horizontally scaled — but that is a well-understood engineering problem, whereas fleet-wide consistency without a central plane is not.

**Tagging taxonomy matters.** Tags are the connective tissue between non-vector filters and human discoverability. Establish a controlled vocabulary early: domain (`finance`, `crm`, `observability`), operation (`read`, `write`, `delete`, `search`), and compliance (`pii`, `pci`, `read-only`). Free-form tags rot within a quarter.

—

## 5. Performance: HNSW-Based Tool Retrieval

Exact nearest-neighbor search over millions of tool vectors is too slow for a synchronous agent loop. Approximate nearest-neighbor (ANN) with **Hierarchical Navigable Small World (HNSW)** graphs is the standard answer, and its characteristics map well onto tool discovery.

**Why HNSW fits.** HNSW builds a multi-layer graph where each node connects to its approximate nearest neighbors. Search descends from a sparse top layer to a dense bottom layer, greedily walking toward the query vector. It delivers logarithmic-ish query time with high recall, supports incremental inserts (new tools can be added without a full rebuild — important when the catalog changes daily), and has no training step, unlike IVF-family indexes that require re-clustering.

**Tuning that actually matters.**

– **`M` (max connections per node).** 16–32 is the sweet spot for tool vectors in the 768–1536 dimension range. Higher `M` improves recall at the cost of memory and insert time.
– **`ef_construction`.** 200–400 for index build. This is a one-time cost; be generous.
– **`ef_search`.** The real latency dial. Higher values trade query time for recall. Expose this per-request: interactive agent turns can use `ef_search=64`, while offline batch re-ranking can afford `ef_search=512`.
– **Quantization.** At millions of vectors, memory dominates. Scalar quantization (int8) typically cuts index memory ~4× with minimal recall loss; product quantization squeezes harder for larger catalogs. Combine with HNSW or HNSW-on-quantized-vectors depending on your recall floor.

**Latency budget.** A realistic target for the discovery call is **p95 under 50ms** end to end, decomposed as: hard-filter metadata lookup (~5ms, cacheable), HNSW search (~10–20ms), hybrid re-rank of 200 candidates (~10ms), and serialization (~5ms). Note where the wins are: *re-ranking a bounded candidate set is cheap and can afford a heavier model*, whereas the ANN search must stay in the tens of milliseconds. Keep the expensive scoring after retrieval, never inside it.

**Two more optimizations.** First, **cache by condensed-intent hash** — many agents ask semantically identical queries, and a short-TTL cache absorbs the storm. Second, **pre-filter before ANN** using metadata partitions rather than relying on post-filtering, which degrades recall badly when filters are selective. Shard indexes by tenant where data residency demands it.

—

## 6. The OnlineCloud Protocol: Massive Toolsets in 6–10 Week Builds

The retrieval architecture above is not exotic — every component is a known quantity. The hard part is shipping it into production without a year of yak-shaving. That is precisely the gap OnlineCloud was built to close.

**OnlineCloud** is a US-based engineering studio focused on Model Context Protocol, autonomous agents, and applied machine learning. The team builds MCP-native architecture that is audited end to end, and — critically for the problem in this post — ships to production in **6–10 weeks, not quarters**. When a platform team is staring down a catalog that has already outgrown the naive manifest approach, that timeline is the difference between a working control plane and a stalled migration.

What that looks like applied to a billion-scale tool fleet:

– **A registry built on real constraints.** OnlineCloud designs MCP servers and registries around actual database schemas, real auth boundaries, and real cost ceilings — wrapping internal APIs as typed tools with clean error semantics, stateless servers with auth, rate limits, and audit logs out of the box.
– **Data connectors where the tools actually live.** Oracle, MS SQL, Postgres, and Snowflake are bridged with row-level access policies, so a tool’s authorization metadata is enforced at the data layer, not just the registry.
– **Transports tuned for the loop.** Streamable HTTP and SSE support keeps discovery and long-running tool calls low-latency — the operational reality behind the p95 budget in §5.
– **Orchestration with guardrails.** Plan/act/reflect loops with cost ceilings and human checkpoints, so dynamic injection is paired with the execution discipline that makes it safe.
– **Telemetry from day one.** Eval suites, observability, and rollback paths are built in, not bolted on. The discovery telemetry in §4 only becomes useful if someone instruments it from the first sprint.

The delivery model is deliberately narrow: senior engineers only, no offshore handoffs, and the architect who scopes the engagement is the engineer pushing commits. Discovery is a one-week fixed-scope sprint that maps your data, picks the model class, and writes the eval that defines “done.” From there, work proceeds in vertical slices — one usable slice every one to two weeks, against real data and real users, iterated against the eval rather than against opinions.

For an infrastructure architect, that is the relevant point. The retrieval design in this post is a solved problem *in principle*. Shipping it — index tuning, filter partitions, injection budgets, fleet-wide governance — across a production catalog is where schedules die. The OnlineCloud protocol compresses that from a multi-quarter slog into a single-digit-week build.

—

## Conclusion

Scaling agents past a hundred tools is not a prompt-engineering problem — it is a distributed-systems problem wearing a prompt-engineering costume. The fix is architectural and non-negotiable: move the catalog out of the prompt and behind a semantic index, retrieve task-specific tools at runtime, inject only a budgeted top-k, and govern the whole thing from a centralized, metadata-rich MCP registry. HNSW keeps retrieval fast enough to sit inside the agent loop. Nothing here is speculative; every piece is a known engineering pattern applied to a new problem.

The teams that build this well will scale their fleets without scaling their prompts. The teams that don’t will keep hitting the same wall, one tool at a time.

—

## Ready to Build Your Tool Discovery Service?

If your catalog has outgrown the naive manifest approach and you need a registry, index, and injection layer in production this quarter — not next year — OnlineCloud builds MCP-native systems in weeks.

**[Talk to an OnlineCloud architect →](https://www.onlinecloud.dev/)**

– **US-based support**, with a free initial consultation and response within 24 hours
– **Production in 6–10 weeks**, with weekly demos of working code — not status updates
– **Email:** [contact@onlinecloud.dev](mailto:contact@onlinecloud.dev)

*Stop stuffing your context window. Start querying it.*