RAG was the default answer to "how does the model know about my data" for two years. Two things changed: context windows got large enough to hold whole documents, and tool calling got reliable enough to fetch facts on demand. Both take work away from retrieval, and neither replaces it.
The three ways to get data into a model
- Load it directly. Put the document in the context window. Simple, exact, and bounded by window size and cost per call.
- Retrieve it. Embed a corpus, search it, insert the top results. Scales past any window, at the cost of a ranking problem you now own.
- Call a tool. Ask a live system. Always current, structured, and authorisable — but only for data behind an API.
When to just load it
If the relevant material fits comfortably in the window and does not change per request, loading it is better than retrieving it. No embedding pipeline, no ranking failures, no stale index. Teams that built RAG for a corpus of forty documents built a search engine they did not need.
The limits are real, though: cost scales with every call, and stuffing a window degrades attention to the parts that matter. "It fits" is not the same as "it should all be there".
When to retrieve
- The corpus is far larger than any window — thousands of documents, a knowledge base, a ticket history.
- Relevance varies sharply per request, so most of the corpus is noise for any given question.
- The material is unstructured prose where meaning matters more than exact fields.
When to call a tool instead
If the answer lives in a system of record — order status, account balance, current on-call — retrieval is the wrong mechanism. An embedded snapshot is stale the moment it is indexed, and it cannot be authorised per user. A tool call is current, structured and enforceable.
The rule of thumb
Prose that changes slowly → retrieve. Facts that change constantly → call a tool. A small fixed corpus → just load it. Most disappointing RAG systems are one of the latter two problems solved with the first mechanism.
What still breaks in retrieval
- Exact identifiers. Semantic search is poor at order numbers and error codes. Hybrid keyword-plus-vector with reranking is the standard fix.
- Freshness. An index is a snapshot; nothing tells the model a chunk is out of date unless you put the timestamp in the chunk.
- Permissions. Retrieval must filter by what the asking user may see, before the search rather than after it.
- Evaluation. Most teams cannot say whether retrieval or generation caused a bad answer, because they never scored retrieval separately.
A pragmatic path
Start by loading. Move to retrieval when the corpus outgrows the window or when you can measure that dilution is hurting answers. Add tools for anything with a system of record. Most production systems end up using all three, chosen per data source rather than adopted wholesale.
Frequently Asked Questions
Do long context windows make RAG obsolete?
No. They raise the threshold at which retrieval is worth building. A corpus of thousands of documents still exceeds any window, and cost per call still scales with what you put in it.
Should I retrieve and also load a summary?
Often yes. A stable summary of the domain plus retrieved specifics tends to outperform retrieval alone, because it gives the model framing it can interpret the chunks against.
How do I know if retrieval is the problem?
Score it separately. Check whether the correct chunk was in the retrieved set at all. If it was and the answer is still wrong, the problem is generation or context ordering, not retrieval.
References
- Context Engineering: Going Beyond Prompt Engineering and RAG — The New Stack
- Context Engineering: A Practical Guide for AI Agents (2026) — Sourcegraph
About Jishu Labs
Jishu Labs is a software development company founded in 2016. We build custom software, AI/ML systems, and full-stack web and mobile applications for clients, and we make eight AI tools for software teams.