The question is usually posed as a choice. It is better read as a sequence, because the three change different things: prompting changes instructions, retrieval changes knowledge, fine-tuning changes behaviour. Most production systems end up using all three for different reasons.
What each one actually changes
- Prompting — how the model interprets the task. Cheapest, instant to iterate, no infrastructure.
- Retrieval — what facts the model has available. The right answer whenever the problem is *the model does not know this*.
- Fine-tuning — the model's default behaviour, format and style. The right answer when the problem is *the model knows, but will not do it consistently*.
The diagnostic question
Ask what kind of wrong the output is:
- Wrong facts → retrieval. No amount of tuning teaches facts reliably, and tuning on facts produces confident hallucination of near-misses.
- Right facts, wrong shape → fine-tuning, or structured outputs if the shape is the only issue.
- Right facts, right shape, wrong emphasis → prompting.
- Inconsistent across similar inputs → fine-tuning, or a smaller and more constrained task.
Try them in this order
Each step costs more than the last, so exhaust the cheap ones first:
- 1. Prompt. Including few-shot examples, which cover a surprising amount of what people reach for tuning to fix.
- 2. Retrieval. If the gap is knowledge, this is the only correct answer regardless of budget.
- 3. Structured outputs / constrained decoding. If the problem is format, enforce the format rather than asking for it.
- 4. Fine-tune. When you have evidence the first three cannot get there, and you have the labelled data to do it.
Fine-tuning is not how you add knowledge
The most expensive mistake in this area is tuning a model on a document corpus to 'teach it' the content. Tuning adjusts behaviour, not a retrievable fact store — the result is a model that produces text in the right register while inventing specifics. If the requirement is that it knows something, retrieve it.
What fine-tuning genuinely buys
- Consistent output format without spending context on instructions and examples.
- A domain register or house style that examples alone do not hold.
- Meaningfully smaller prompts, which at high volume is a real cost reduction.
- A small model performing a narrow task at a level that otherwise needs a much larger one — the main lever behind the cost gap between small and frontier models.
The costs people underestimate
- Labelled data. Hundreds to thousands of examples of exactly the behaviour you want. Producing them is the actual project.
- Re-tuning. When the task drifts or the base model is upgraded, you do it again.
- Evaluation. You now need a suite to prove the tuned model is better, and to detect when it stops being.
- Lock-in. A tuned model ties you to a provider and a base version in a way prompting does not.
Frequently Asked Questions
Can I use RAG and fine-tuning together?
Yes, and it is a common production shape: tune for format and domain register, retrieve for facts. They address different failures, so combining them is complementary rather than redundant.
How much data do I need to fine-tune?
Fewer examples than people expect for narrow tasks - often hundreds rather than thousands - but they must be consistent. A thousand inconsistent examples teach inconsistency.
Is prompt engineering obsolete?
No, it got smaller. It is still the first thing to try and the cheapest to change, and the framing of the task still matters after the other layers are in place.
References
- Context Engineering: A Practical Guide for AI Agents (2026) — Sourcegraph
- How to Choose Between Small and Frontier Models — Towards Data Science
About Jishu Labs
Jishu Labs is a software development company founded in 2016. We build custom software, AI/ML systems, and full-stack web and mobile applications for clients, and we make eight AI tools for software teams.