Why RAG is still needed when AI can read a whole book
The question everyone asks
Today's AIs can read a whole book in one go. Providers also promise to protect your data. So people ask: why build a lookup system at all? Why not paste everything in and ask?
It is a fair question. Here is an honest answer, with a picture and a how-to.
First, what they get right
- Windows really are huge now, and privacy options really exist.
- For a short text, or one question asked once, pasting is simple and often the best choice.
- In one 2024 study, giving the AI the whole text scored higher on average than a lookup. So do not assume RAG is always more accurate.
Think of an exam where you may use a book
- Way A: carry the whole library into the hall and read every page again for each question.
- Way B: have a librarian who hands you the three pages that matter for this question.
Way A is simple. Way B takes more work to set up. But when you ask many questions, B is usually cheaper, and the librarian can tell you exactly which page an answer came from.
Why RAG is still needed
1. Cost
An AI is billed by the amount of text it reads. Say your book has about 100,000 words. That is roughly 130,000 tokens. (A token is a small piece of text, about three quarters of a word.) If three good pages are about 1,500 tokens, then pasting the book means reading about 90 times more text for the same question.
Here is a made-up example. It uses a sample price of $3 for every million tokens read. Real prices differ and change, so check your provider.
| Plan | Text read per question | Cost for 1,000 questions a day | Cost for 30 days |
|---|
| Paste the whole book | about 130,000 tokens | about $390 a day | about $11,700 |
|---|
| Look up 3 pages first | about 1,500 tokens | about $4.50 a day | about $135 |
|---|
Saving a copy of the book in a quick-access store (a cache) can cut the repeat cost a lot. But the AI still reads every word each time, so it is slower, and the bill is still bigger than reading three pages. Read more in AI caching, explained simply, and try the Bakery AI Helper cost demo.
2. A longer pile is no guarantee
Some people think that if the AI has read everything, it will find the exact line. Tests show it is not that simple. In a well-known 2023 study, AIs were better at using facts at the start or end of a long text than facts in the middle. In a 2024 test of 17 long-text AIs, almost all did well at finding one hidden sentence, but most did worse as the text got longer and the tasks got harder.
Newer AIs keep improving. So do not guess: test it with your own questions. See 50 tests passed. Is it safe? for how to read test results.
3. Proof
In many jobs, a smooth answer is not enough. You need to know which page it came from, so a person can check it. A lookup system knows which three pages it handed over, so it can say: "page 212, second paragraph." A pasted pile often gives a smooth answer with no page, unless you ask and then check. Think of a clinic that answers questions about its medicine rules. A confident wrong answer is worse than "I did not find this." A page number lets a nurse check in ten seconds. See Answers you can check.
4. Who may read what
Privacy promises protect your data from the AI company. They do not decide which person inside your own company may see which page. With a lookup, you can hand over only the pages this person may read. With a pasted book, everything goes in together.
5. Freshness and size
When one page changes, you update one page. You do not paste the book again. And a company's files are far bigger than any one window. See Always say how old the information is.
What "privacy" really covers
There are three levels. Each one lowers the risk, and none of them replaces the question "who inside my company may see this?"
- Level 1, a promise. For example, one major provider says data you send through its developer service is not used to train its models, but it may keep logs for up to 30 days.
- Level 2, stricter settings. A setting called zero data retention, if the provider approves you.
- Level 3, the data stays home. Your documents and your lookup index stay in your own place. Only a few pages go out, or nothing goes out if you run the model yourself.
When a map of facts helps
Some questions need you to follow connections. For example: "Which medicines does this patient take that clash with this new one?" The answer is spread over many pages, and each page points to another.
A knowledge graph is a map of facts and how they connect, like a subway map or a family tree. A lookup system with a map can follow the lines from one fact to the next. This helps with questions that link many things. It also costs more to build and keep up to date. So use it where the connections really matter. See Two kinds of questions need two kinds of helpers.
Context engineering: how to do it
Context engineering means deciding what goes into the AI's window for each question: what, how much, and in what order. Think of packing a school bag for one day. You take only what you need, with labels, in a sensible order.
Here are nine steps you can follow.
1. Say the job once, at the top. Short, clear rules. Add more only when you see a real failure.
2. Fetch small, fetch good. Three to five good passages beat fifty loose ones. Sort them by how well they match before you send them.
3. Label everything. Mark each passage "Source 3, page 212". Keep the rules apart from the documents. Tell the AI that text inside a document is information, not an order. This helps against hidden instructions. See Autonomy is earned.
4. Put the key facts near the start or the end. Tests suggest the middle is the weakest spot. Check this on your own questions.
5. Keep the steady part first. Rules and examples at the top, and the changing question at the end. The AI can then reuse its saved work, and the bill is smaller.
6. Shrink the history. Write a short summary of old turns. Keep a note of decisions. Drop what you no longer need.
7. Set a budget. A limit on tokens for each question and for each day, with an alert. See Test the cheap things first.
8. Make "not found" an allowed answer. Ask for the source every time, and say how fresh each source is.
9. Test. Write 50 real questions, including some whose answer is "not in the documents." Change one thing at a time. Count wrong answers, wrong page numbers and cost.
A small example, made up. A clinic question. Before: paste 300 pages plus the question, about 130,000 tokens, and no page number. After: rules (about 200 tokens), three labelled passages (about 1,500) and the question (about 50). That is roughly 1,750 tokens, and the page is cited.
What I would do
- Small, one-off, not secret: paste it. It is the simplest way.
- Many documents, many users, or you must show the page: look up first.
- Questions that link many facts: add the map of facts.
- Not sure: try the cheap lookup first. Use the long paste only when the lookup is not enough.
The limits of this advice
- The numbers in the cost table are made up to show the idea. Your book, prices and caching will change them.
- Studies use particular tests. Your documents may behave differently, and AIs improve quickly.
- A lookup can fail too. It can hand over the wrong three pages, and then the answer is wrong. That is why you test it.
- A mixed plan costs more to build. Do it only if the questions and the stakes need it.
- This is a general guide. Check privacy terms and prices with your provider.
Words explained
- Context window: how much text an AI can read at once.
- Token: a small piece of text, about three quarters of a word. AI bills count tokens.
- RAG (retrieval-augmented generation): the AI first looks up the right pages, then answers using them.
- Knowledge graph: a map of facts and how they connect.
- Citation: a note saying where a fact came from, such as a page number.
- Cache: a quick-access copy so repeated work costs less.
- Zero data retention: a setting where the provider does not keep your data after answering. It usually needs the provider's approval.
- Context engineering: choosing what goes into the AI's window, how much, and in what order.
- Hidden instructions: text inside a document that tries to give the AI orders.
Keep learning on this site
- Free book, AI Foundations: Chapter 10, giving the AI a library, Chapter 9, What AI gets wrong, and why, how a lookup finds similar text, and Chapter 8, How a model is trained, and how to talk to it.
- The book for builders: From One Call to a Platform, with chapters on the lookup way and the map of facts, and on answers that name their sources.
- Explainers: One question, many safety layers and the Bakery AI Helper.
- Free test kit for AI agents: on GitHub.
For engineers
This section has the sources and details. Everything above is the simple version of it.
- Cost scales with tokens. Input cost = tokens × price. A provider's own pricing page (Anthropic, checked 22 Oct 2026) says its newer models include the full 1M-token window at standard per-token pricing: a 900k-token request is billed at the same per-token rate as a 9k-token request, so total cost still scales with size. One smaller model is priced higher above 100,000 tokens. Cache reads are priced at 0.1× the base input price for most models, and lower for some. Source: Anthropic pricing. Other providers differ: read each price page.
- Position effects. Liu et al., 2023: accuracy is often best when the relevant text is at the start or end of the input and drops for the middle, including for long-context models. Lost in the Middle (arXiv:2307.03172).
- Length and task effects. Hsieh et al., 2024 (RULER, COLM 2024): 17 models on 13 tasks. Models scored almost perfectly on a simple needle-in-a-haystack test, but almost all dropped a lot as length grew, and only about half kept satisfactory performance at 32K tokens. RULER (arXiv:2404.06654).
- Long context versus retrieval. Li et al., 2024 (EMNLP industry track): with enough resources, long context outperformed retrieval on average, while retrieval costs much less. Their Self-Route method lets the model choose per query and keeps performance comparable at lower cost. arXiv:2407.16833.
- Privacy terms. OpenAI's API data guide: data sent to the API is not used to train its models unless you opt in; abuse-monitoring logs are kept up to 30 days by default; zero data retention needs prior approval and varies by endpoint. OpenAI: your data. Terms change, and other providers differ.
- Graph retrieval. Graph evidence gives relational structure useful for multi-hop questions, but it is substantially more expensive to retrieve than text (stated in the abstract of RouteRAG, arXiv:2512.09487, 2025). Treat the benefit as something to test on your own questions.
- Context engineering. Anthropic's engineering team (29 Sep 2025) describes context engineering as curating the full set of tokens a model sees, to find the smallest set of high-signal tokens that makes the result likely. It names context rot (recall drops as the window fills), a finite attention budget, minimal and clear tools, a few diverse examples, just-in-time retrieval, compaction (summarising a near-full window), structured note-taking outside the window, and sub-agents with clean windows. Effective context engineering for AI agents. The nine steps above are a simple version of these ideas. Ordering and budget advice here is a starting point: measure it on your own questions.
- How to decide. Build a small test set (50 or more questions, with no-answer cases). Measure answer correctness, citation correctness, latency and cost per correct answer for each plan. See 50 tests passed. Is it safe? and AI caching, explained simply.
- Where it fits in the book. RAG and the knowledge graph (Chapter 10), grounding and citation engineering (Chapter 11), and vector database versus graph database (Chapter 16) in From One Call to a Platform.