Fine-Tuning Llama 3 for Tool Selection Accuracy: A Comparative Accuracy Study

# Fine-Tuning Llama 3 for Tool Selection Accuracy: A Comparative Accuracy Study


## 1. Introduction: Why Prompting Frontier Models Hits a Tool-Call Ceiling

There is a moment every agentic-AI team hits around week six. The orchestration loop is solid, the tool schemas are typed, the eval harness is green — and the model still calls the wrong tool. It reaches for the Oracle `ORDER_UPDATE` endpoint when the user asked for an invoice status. It formats a Salesforce `Lead` insert with `Account` fields. The system prompt is a work of art, few-shot examples have been curated to death, and accuracy still plateaus between 70% and 85% on the trajectory-level metric that predicts production behavior.

This plateau is structural, not a prompt-engineering failure. Frontier models like GPT-4o and Claude 3.5 Sonnet are trained on web-scale general function calling, not on *your* API dialect. They are excellent at guessing what a plausible tool call looks like; they are mediocre at remembering that one Oracle endpoint wants a SOAP envelope while another expects a REST payload. Every token of prompt spent disambiguating your API is budget the model cannot spend reasoning about user intent — and once your surface holds 40, 60, or 200 tools, the few-shot prompt becomes the bottleneck before the model does.

At OnlineCloud, we kept hitting this ceiling building MCP-connected agents for Oracle and Salesforce estates, so we ran a controlled study: instead of fighting the prompt, we fine-tuned an open-weights Llama 3 model on golden tool-call trajectories and benchmarked it head-to-head against GPT-4o and Claude 3.5 Sonnet on the same held-out eval set. This post is the full write-up: methodology, dataset curation, the accuracy numbers, deployment on Bedrock and EKS, and an honest ROI read on when fine-tuning pays for itself.

## 2. Fine-Tuning Methodology: SFT, DPO, and LoRA for API Dialects

Fine-tuning for tool selection is not chat fine-tuning. Chat tuning optimizes for fluent text; tool-selection tuning optimizes one narrow decision — given a conversation state, which tool from a fixed registry, with which arguments, in which serialization? The output grammar is small; the decision boundary is everything.

**Stage one: SFT.** Supervised fine-tuning teaches the skeleton. For a surface of ~60 Oracle and Salesforce operations, we generated roughly 12,000 SFT examples teaching exact tool names, argument casing (`recordTypeId`, not `record_type_id`), required-field sets, and serialization conventions. SFT alone takes a model from “plausible but often wrong” to “mostly right on seen shapes.” What it does not fix is *discrimination* — preferring the correct tool when two candidates are both syntactically valid. That is a preference problem, so we add a second stage.

**Stage two: DPO.** Rather than run full RLHF — reward models, PPO rollout farms, instability — we used Direct Preference Optimization, which reuses the SFT model as its reference and optimizes the policy against paired preferred/rejected trajectories. Our pairs came from production logs: the same request answered by a trajectory that ended in successful tool execution (preferred) versus one that selected the wrong operation or hallucinated arguments (rejected). DPO sharpened exactly the axis SFT leaves blunt, contributing roughly half the total accuracy gain over the base model at a fraction of RLHF’s cost.

**Stage three: LoRA.** Full fine-tuning of a 70B model is a capital project. Low-Rank Adaptation freezes the base weights and trains small adapter matrices (we used rank 16 on attention, rank 32 on MLP, alpha at 2× rank). Our adapter for the 8B model was ~160 MB — a few percent of the base weights — and trained on a single A100-class instance in under 24 hours. Because the base model stays intact, one deployment can host multiple dialect adapters behind a router and swap them per request. That adapter multiplexing drives most of the ROI math in Section 6.

## 3. Dataset Curation: Generating Golden Tool-Call Trajectories for Oracle and Salesforce

Fine-tuning quality is dataset quality, and tool-call datasets are the hardest kind to curate because ground truth is a *procedure*, not a string. We built ours in four layers.

**Schema-derived scaffolding.** For each Oracle EBS and Salesforce operation, we generated synthetic user intents from the operation’s own metadata — description, parameters, typical calling context — yielding thousands of cheap, unambiguous examples. Scaffolding establishes coverage; it does not establish realism.

**Production-log distillation.** The highest-value data came from logged production traffic on existing MCP servers: real utterances, attempted tools, argument values, and execution outcomes. We kept only trajectories that ended in verified success and deduplicated aggressively — 60% of requests map to roughly 15% of the operation space, and without dedup the model overfits the popular path.

**LLM-assisted labeling with verification.** We used frontier models to propose trajectories for the hard tail — ambiguous requests, cross-system intents like “sync this account’s new orders from Oracle into Salesforce” — then verified every proposal against the target APIs’ OpenAPI/WSDL schemas and, for anything with a side effect, an execution check in a sandbox tenant. Proposals that failed were rejected, not repaired. Labels were canonicalized into one strict grammar per dialect, so the model never had to learn two serializations for one decision.

**Adversarial negative mining.** We deliberately constructed the base model’s failure cases — requests where an entity like “order” maps to a different operation in context, requests naming a field from the wrong object, cross-dialect confusions — and used them as DPO rejected pairs. The final corpus: ~12,000 SFT examples and ~4,000 preference pairs, split 90/10 with the eval stratified so no tool appears only in training. Every number in Section 4 comes from that held-out set.

## 4. Accuracy Benchmarks: Llama 3 (Fine-Tuned) vs. GPT-4o vs. Claude 3.5 Sonnet

We benchmarked four configurations on 800 held-out real-world-derived requests (400 Oracle, 400 Salesforce), scored at the trajectory level: a call is correct only if the *operation*, *arguments*, and *serialization* all match the golden trajectory. Models: GPT-4o and Claude 3.5 Sonnet via hosted APIs (zero-shot tool-use prompting), base Llama 3 8B (zero-shot), and our SFT+DPO LoRA-tuned Llama 3 8B. Temperature 0, five runs per request, majority vote on disagreements.

| Configuration | Oracle tool-select | Salesforce tool-select | Argument accuracy | Composite trajectory |
|—|—|—|—|—|
| Llama 3 8B (base, zero-shot) | 61.4% | 64.2% | 58.7% | 56.9% |
| GPT-4o (zero-shot) | 82.1% | 84.6% | 81.3% | 78.4% |
| Claude 3.5 Sonnet (zero-shot) | 84.8% | 86.9% | 83.5% | 80.2% |
| **Llama 3 8B (SFT + DPO LoRA)** | **93.6%** | **94.8%** | **91.2%** | **89.7%** |

Three findings matter more than the headlines. **First, the tuned 8B beats both frontier models on its own dialect by roughly nine to eleven points on composite accuracy.** That is the expected result of training on the actual distribution: the frontier models are generalists; our adapter is a specialist in one narrow decision. This is not a claim that fine-tuned Llama 3 is smarter — it is a claim that it is *more calibrated to this tool surface*.

**Second, the gap widens exactly where prompting hurts most.** On the adversarial subset, GPT-4o fell to 71.2% and Claude 3.5 Sonnet to 73.8%, while the tuned model held 88.4%. Few-shotting closed maybe two points of the frontier gap; it never closed the structural one. Moving from 20 tools to the full 60-tool registry cost the frontier models three to four points and the tuned model less than one — prompting degrades with surface size; fine-tuning does not.

**Third, the base-model delta is the entire story.** Base Llama 3 8B at 56.9% is unusable here; the tuned model at 89.7% is production-grade. The whole gain — 33 points — came from ~160 MB of adapters and a training run measured in hours, at roughly 3–5× lower per-request cost than the hosted frontier models. We ran the same pipeline on Llama 3 70B; it buys one to two points for 8× the inference cost. The 8B adapter is the economic optimum.

## 5. Implementation: Deploying LoRA Adapters on AWS Bedrock or EKS

An adapter that lives in a notebook is a demo. Shipping it means putting it behind the same infrastructure as the rest of the agent — typed tool surfaces, auth boundaries, observability, rollback. We deploy on two AWS paths, and the choice is usually dictated by existing platform posture.

**Option A: AWS Bedrock (managed, minimal ops).** Bedrock supports importing custom models: the LoRA adapter is merged with the base weights at export, uploaded to S3, and registered through the API. You get managed inference with IAM access, CloudWatch logging, and auto scaling — no GPU fleet to babysit. The trade-offs: managed-inference margins, Bedrock’s import cadence (slower adapter iteration than a hot-swap), and no multi-adapter multiplexing behind one endpoint. One dialect, one region — Bedrock is the right call.

**Option B: EKS with vLLM (self-hosted, adapter-native).** For multi-dialect estates — Oracle in production, Salesforce in the same control plane, MS SQL next quarter — we deploy on EKS inside the client’s VPC. vLLM natively serves LoRA adapters, so one base-model deployment exposes multiple adapter endpoints and swaps them per request with near-zero overhead. The stack is standard: an inference Deployment behind a service, an MCP server as the tool-orchestration layer in front, OIDC-auth’d ingress, and a HorizontalPodAutoscaler on GPU utilization. Oracle traffic routes to its adapter; Salesforce traffic to its own; the base weights are shared, so memory is dominated by one model plus small adapter slabs.

Our rule of thumb: **Bedrock if you have one dialect and want someone else to own the GPU; EKS + vLLM if you have multiple adapters, regulated data that must stay in your VPC, or a retraining cadence measured in weeks.** Either way, three non-negotiables apply. The endpoint sits behind the same MCP tool layer as any hosted model, so swapping GPT-4o for the adapter is a routing change, not a rewrite. Every tool call is logged as a structured event for eval replay. And the previous adapter version stays live for one release cycle, so a regression is a routing rollback, not a redeploy. Instrument production accuracy the same way you ran the eval — sample traffic, replay against the golden set, and page on drift when the Oracle or Salesforce schemas change.

## 6. ROI: Performance Gains vs. Training and Inference Cost Trade-Offs

The honest question is never “does fine-tuning work?” — Section 4 answers that. It is whether fine-tuning pays for itself on *your* volume, *your* surface, and *your* accuracy requirement — a function of four numbers.

**Training cost is a rounding error.** Our SFT + DPO LoRA run on Llama 3 8B took roughly 12 GPU-hours on an A100-class instance, eval and ablations included — a few hundred dollars at current rates, and the low thousands even with a 70B experiment and a second dialect. That is less than one week of GPT-4o spend for a moderately busy agent, and it is capital cost amortized over every subsequent request. The real constraint is dataset curation labor, not compute.

**Inference cost is where the trade-off lives.** The tuned 8B runs at roughly 20–30% of GPT-4o’s per-token cost and lower still than Claude 3.5 Sonnet. But self-hosting has a floor: a GPU pod costs the same at one request a minute or a thousand. Below roughly 5,000–10,000 tool-call requests per day, hosted frontier models are often cheaper — you pay for consumed tokens, not reserved silicon. Above that volume the crossover favors the self-hosted adapter, and the gap widens linearly. The tuned model also removes the giant few-shot system prompt from every request and eliminates prompting’s worst failure mode: silent accuracy cliffs when the prompt drifts or a tool is added.

**The accuracy-point valuation is the number most teams skip.** A mis-selected tool costs one of three things: a failed API call retried (cheap), a wrong side-effect manually reversed in Oracle or Salesforce (expensive — the reason sandbox verification and human checkpoints before irreversible actions matter), or a silent wrong answer that reaches the user (most expensive). At 8,000 requests per day, moving from 80% to 90% trajectory accuracy is 800 fewer failures daily. If even 10% are wrong side-effects costing fifteen minutes of admin time to reverse, the tuned model pays for its entire training run before lunch. That is why we write the eval before the engagement: baseline accuracy and failure cost are both knowable in week one.

**When NOT to fine-tune.** Three conditions push us back toward prompting: low volume (a few thousand requests daily or less, where hosted models win at the margin); a tool surface still changing weekly (adapter retraining churns); and accuracy requirements a router already satisfies — if 85% of traffic is three well-handled tools, a small classifier routing those to a cheap prompt path beats fine-tuning on ROI. Fine-tuning is for the workload that is high-volume, stable-shaped, and where a wrong call has real cost. That matches most production Oracle and Salesforce agents we see — which is exactly why the prompting ceiling, not model capability, is what our clients keep hitting.

## Conclusion: When Prompting Ceils, Fine-Tune the Decision

Frontier models will keep improving at general tool use, and your API dialect will keep getting more specific. That asymmetry is permanent — which is why a small, fine-tuned specialist that knows your Oracle envelope and your Salesforce object model cold is not a workaround; it is the correct architecture for a stable, high-volume tool surface. In our study, a ~160 MB LoRA adapter on Llama 3 8B outperformed GPT-4o by 11 points and Claude 3.5 Sonnet by 9 points on composite trajectory accuracy, for a training cost in the hundreds of dollars and inference at 8B pricing. The bottleneck was never the model — it was the dataset, and dataset curation is an engineering problem with a known solution.

If your agent team is staring at a tool-selection plateau — Oracle, Salesforce, MS SQL, or a custom surface — bring us the eval and we will tell you in one discovery week whether fine-tuning earns its keep. OnlineCloud builds the MCP tool layer, curates the golden trajectories, runs the SFT + DPO + LoRA pipeline, and deploys the adapter on Bedrock or in your VPC on EKS — with the accuracy dashboards to prove it. [Contact the OnlineCloud team](https://www.onlinecloud.dev/) and let’s measure your baseline: prompting where it works, fine-tuning where it doesn’t, and an honest cost model for the gap in between.