The cost gap between small and frontier models is large enough to change architecture decisions. Published comparisons put a fine-tuned small model deployed in your own infrastructure at $150-$800 per month for volumes where hosted frontier inference runs $15,000-$75,000. Serving a 7B model is roughly 10-30x cheaper than a 70-175B model on tasks where accuracy is equivalent.
The phrase carrying the weight is *where accuracy is equivalent*. Everything below is about establishing that.
What small models are genuinely good at
- Classification and routing — intent detection, triage, tagging. Narrow output space, and fine-tuning closes most of the gap.
- Extraction — pulling structured fields out of documents against a known schema.
- Transformation — reformatting, summarising to a fixed template, normalising text.
- High-frequency, low-ambiguity steps inside an agent loop — the calls that happen thousands of times and rarely require judgement.
Where frontier models still win
Small models remain behind on open-ended reasoning, multi-step problem solving, long-context retention and niche factual accuracy. If a task requires holding several constraints in mind and deciding between them, that is not where to save money.
The routing pattern
The architecture that has settled out in 2026 is heterogeneous rather than either-or: a frontier model for planning and genuinely ambiguous work, mid-tier for standard tasks, small models for high-frequency execution. Roughly 80% of predictable queries route to the small model and the remainder escalates.
def route(task):
# Cheap deterministic checks first - no model call at all
if task.matches_known_template():
return SMALL
# Confidence-gated: run small, escalate when it is unsure
result = small_model(task)
if result.confidence < THRESHOLD or result.requested_escalation:
return FRONTIER
return resultYou cannot route without evals
Every routing decision is a claim that the smaller model is good enough for that task. Without an eval suite you are not optimising cost, you are degrading quality invisibly and finding out from customers. Build the evals first, then downgrade one task at a time and watch the scores.
The costs that are not per-token
Self-hosting a small model moves spend from a per-token line to places that are easy to leave out of the comparison:
- GPU capacity that idles. You pay for provisioned hardware whether or not traffic arrives; per-token pricing does not charge you for quiet Sundays.
- Fine-tuning and re-tuning. The accuracy that makes a small model viable usually comes from tuning, and tuning is repeated whenever the task drifts.
- Operational load. Serving, scaling, upgrading and monitoring inference infrastructure is a real team cost.
- Evaluation. Running a suite continuously across two model tiers costs money and engineering attention.
Below a certain volume the frontier API is genuinely cheaper once these are counted. The break-even is a function of your traffic, not a universal number — which is why this belongs in an estimate rather than a rule of thumb.
How to decide
- Instrument current usage by task type: volume, token counts, latency needs.
- Rank tasks by volume. The savings live in the top few.
- Build evals for the highest-volume task before touching it.
- Try the smallest credible model, tune if needed, compare against the eval baseline.
- Move it only if quality holds — then repeat with the next task.
Frequently Asked Questions
Is a fine-tuned small model always cheaper than a frontier API?
No. Below a certain request volume, provisioned GPU capacity that sits idle costs more than per-token pricing. The comparison has to include idle capacity, tuning and operations, not just inference.
Can a small model plan an agent workflow?
Usually not well. Planning is the open-ended reasoning small models are weakest at, which is why the plan-and-execute split puts a capable model on the plan and cheap models on the steps.
How much accuracy loss is acceptable?
That is a product question, not a model question. Define the acceptable rate per task before you measure, or you will rationalise whatever number you get.
References
- Small Language Models Outperform Frontier AI On Cost, Speed And Accuracy — Forbes
- How to Choose Between Small and Frontier Models — Towards Data Science
About Jishu Labs
Jishu Labs is a software development company founded in 2016. We build custom software, AI/ML systems, and full-stack web and mobile applications for clients, and we make eight AI tools for software teams.