The question everyone asks

Today's AIs can read a whole book in one go. Providers also promise to protect your data. So people ask: why build a lookup system at all? Why not paste everything in and ask?

It is a fair question. Here is an honest answer, with a picture and a how-to.

First, what they get right

• Windows really are huge now, and privacy options really exist.
• For a short text, or one question asked once, pasting is simple and often the best choice.
• In one 2024 study, giving the AI the whole text scored higher on average than a lookup. So do not assume RAG is always more accurate.

Think of an exam where you may use a book

• Way A: carry the whole library into the hall and read every page again for each question.
• Way B: have a librarian who hands you the three pages that matter for this question.

Way A is simple. Way B takes more work to set up. But when you ask many questions, B is usually cheaper, and the librarian can tell you exactly which page an answer came from.

Why RAG is still needed

1. Cost

An AI is billed by the amount of text it reads. Say your book has about 100,000 words. That is roughly 130,000 tokens. (A token is a small piece of text, about three quarters of a word.) If three good pages are about 1,500 tokens, then pasting the book means reading about 90 times more text for the same question.

Here is a made-up example. It uses a sample price of $3 for every million tokens read. Real prices differ and change, so check your provider.

Saving a copy of the book in a quick-access store (a cache) can cut the repeat cost a lot. But the AI still reads every word each time, so it is slower, and the bill is still bigger than reading three pages. Read more in AI caching, explained simply, and try the Bakery AI Helper cost demo.

2. A longer pile is no guarantee

Some people think that if the AI has read everything, it will find the exact line. Tests show it is not that simple. In a well-known 2023 study, AIs were better at using facts at the start or end of a long text than facts in the middle. In a 2024 test of 17 long-text AIs, almost all did well at finding one hidden sentence, but most did worse as the text got longer and the tasks got harder.

Newer AIs keep improving. So do not guess: test it with your own questions. See 50 tests passed. Is it safe? for how to read test results.

3. Proof

In many jobs, a smooth answer is not enough. You need to know which page it came from, so a person can check it. A lookup system knows which three pages it handed over, so it can say: "page 212, second paragraph." A pasted pile often gives a smooth answer with no page, unless you ask and then check. Think of a clinic that answers questions about its medicine rules. A confident wrong answer is worse than "I did not find this." A page number lets a nurse check in ten seconds. See Answers you can check.

4. Who may read what

Privacy promises protect your data from the AI company. They do not decide which person inside your own company may see which page. With a lookup, you can hand over only the pages this person may read. With a pasted book, everything goes in together.

5. Freshness and size

When one page changes, you update one page. You do not paste the book again. And a company's files are far bigger than any one window. See Always say how old the information is.

What "privacy" really covers

There are three levels. Each one lowers the risk, and none of them replaces the question "who inside my company may see this?"

• Level 1, a promise. For example, one major provider says data you send through its developer service is not used to train its models, but it may keep logs for up to 30 days.
• Level 2, stricter settings. A setting called zero data retention, if the provider approves you.
• Level 3, the data stays home. Your documents and your lookup index stay in your own place. Only a few pages go out, or nothing goes out if you run the model yourself.

When a map of facts helps

Some questions need you to follow connections. For example: "Which medicines does this patient take that clash with this new one?" The answer is spread over many pages, and each page points to another.

A knowledge graph is a map of facts and how they connect, like a subway map or a family tree. A lookup system with a map can follow the lines from one fact to the next. This helps with questions that link many things. It also costs more to build and keep up to date. So use it where the connections really matter. See Two kinds of questions need two kinds of helpers.

Context engineering: how to do it

Context engineering means deciding what goes into the AI's window for each question: what, how much, and in what order. Think of packing a school bag for one day. You take only what you need, with labels, in a sensible order.

Here are nine steps you can follow.

1. Say the job once, at the top. Short, clear rules. Add more only when you see a real failure.

2. Fetch small, fetch good. Three to five good passages beat fifty loose ones. Sort them by how well they match before you send them.

3. Label everything. Mark each passage "Source 3, page 212". Keep the rules apart from the documents. Tell the AI that text inside a document is information, not an order. This helps against hidden instructions. See Autonomy is earned.

4. Put the key facts near the start or the end. Tests suggest the middle is the weakest spot. Check this on your own questions.

5. Keep the steady part first. Rules and examples at the top, and the changing question at the end. The AI can then reuse its saved work, and the bill is smaller.

6. Shrink the history. Write a short summary of old turns. Keep a note of decisions. Drop what you no longer need.

7. Set a budget. A limit on tokens for each question and for each day, with an alert. See Test the cheap things first.

8. Make "not found" an allowed answer. Ask for the source every time, and say how fresh each source is.

9. Test. Write 50 real questions, including some whose answer is "not in the documents." Change one thing at a time. Count wrong answers, wrong page numbers and cost.

A small example, made up. A clinic question. Before: paste 300 pages plus the question, about 130,000 tokens, and no page number. After: rules (about 200 tokens), three labelled passages (about 1,500) and the question (about 50). That is roughly 1,750 tokens, and the page is cited.

What I would do

• Small, one-off, not secret: paste it. It is the simplest way.
• Many documents, many users, or you must show the page: look up first.
• Questions that link many facts: add the map of facts.
• Not sure: try the cheap lookup first. Use the long paste only when the lookup is not enough.

The limits of this advice

• The numbers in the cost table are made up to show the idea. Your book, prices and caching will change them.
• Studies use particular tests. Your documents may behave differently, and AIs improve quickly.
• A lookup can fail too. It can hand over the wrong three pages, and then the answer is wrong. That is why you test it.
• A mixed plan costs more to build. Do it only if the questions and the stakes need it.
• This is a general guide. Check privacy terms and prices with your provider.

Words explained

• Context window: how much text an AI can read at once.
• Token: a small piece of text, about three quarters of a word. AI bills count tokens.
• RAG (retrieval-augmented generation): the AI first looks up the right pages, then answers using them.
• Knowledge graph: a map of facts and how they connect.
• Citation: a note saying where a fact came from, such as a page number.
• Cache: a quick-access copy so repeated work costs less.
• Zero data retention: a setting where the provider does not keep your data after answering. It usually needs the provider's approval.
• Context engineering: choosing what goes into the AI's window, how much, and in what order.
• Hidden instructions: text inside a document that tries to give the AI orders.

Keep learning on this site

• Free book, AI Foundations: Chapter 10, giving the AI a library, Chapter 9, What AI gets wrong, and why, how a lookup finds similar text, and Chapter 8, How a model is trained, and how to talk to it.
• The book for builders: From One Call to a Platform, with chapters on the lookup way and the map of facts, and on answers that name their sources.
• Explainers: One question, many safety layers and the Bakery AI Helper.
• Free test kit for AI agents: on GitHub.

For engineers

This section has the sources and details. Everything above is the simple version of it.

• Cost scales with tokens. Input cost = tokens × price. A provider's own pricing page (Anthropic, checked 22 Oct 2026) says its newer models include the full 1M-token window at standard per-token pricing: a 900k-token request is billed at the same per-token rate as a 9k-token request, so total cost still scales with size. One smaller model is priced higher above 100,000 tokens. Cache reads are priced at 0.1× the base input price for most models, and lower for some. Source: Anthropic pricing. Other providers differ: read each price page.
• Position effects. Liu et al., 2023: accuracy is often best when the relevant text is at the start or end of the input and drops for the middle, including for long-context models. Lost in the Middle (arXiv:2307.03172).
• Length and task effects. Hsieh et al., 2024 (RULER, COLM 2024): 17 models on 13 tasks. Models scored almost perfectly on a simple needle-in-a-haystack test, but almost all dropped a lot as length grew, and only about half kept satisfactory performance at 32K tokens. RULER (arXiv:2404.06654).
• Long context versus retrieval. Li et al., 2024 (EMNLP industry track): with enough resources, long context outperformed retrieval on average, while retrieval costs much less. Their Self-Route method lets the model choose per query and keeps performance comparable at lower cost. arXiv:2407.16833.
• Privacy terms. OpenAI's API data guide: data sent to the API is not used to train its models unless you opt in; abuse-monitoring logs are kept up to 30 days by default; zero data retention needs prior approval and varies by endpoint. OpenAI: your data. Terms change, and other providers differ.
• Graph retrieval. Graph evidence gives relational structure useful for multi-hop questions, but it is substantially more expensive to retrieve than text (stated in the abstract of RouteRAG, arXiv:2512.09487, 2025). Treat the benefit as something to test on your own questions.
• Context engineering. Anthropic's engineering team (29 Sep 2025) describes context engineering as curating the full set of tokens a model sees, to find the smallest set of high-signal tokens that makes the result likely. It names context rot (recall drops as the window fills), a finite attention budget, minimal and clear tools, a few diverse examples, just-in-time retrieval, compaction (summarising a near-full window), structured note-taking outside the window, and sub-agents with clean windows. Effective context engineering for AI agents. The nine steps above are a simple version of these ideas. Ordering and budget advice here is a starting point: measure it on your own questions.
• How to decide. Build a small test set (50 or more questions, with no-answer cases). Measure answer correctness, citation correctness, latency and cost per correct answer for each plan. See 50 tests passed. Is it safe? and AI caching, explained simply.
• Where it fits in the book. RAG and the knowledge graph (Chapter 10), grounding and citation engineering (Chapter 11), and vector database versus graph database (Chapter 16) in From One Call to a Platform.

A story you already know: learning to drive

Nobody hands a ten-year-old the car keys. First you watch. Then you practise with an adult beside you. Later you get a learner's permit. Only after tests do you get a full licence. And if you drive badly, you can lose it.

AI helpers should be treated the same way. This idea is called autonomy is earned. Autonomy means doing things on your own. Earned means you win it by proving yourself.

An AI agent is a computer helper that can do things, not just talk. It can send emails, pay bills or change settings. The more it can do on its own, the more it has to prove first.

Why not just trust a good demo?

A demo is like a student showing off their best homework. It shows what the helper can do on a good day. It does not show a bad day.

There is a second problem. The AI "brain" inside the helper is called the model. The model gets updated now and then, like an app on a phone. After an update, the helper can behave differently overnight. It is like a driver who suddenly forgets the rules.

Three questions to ask before giving any freedom

• How bad is the worst mistake? Ask two things. How much damage could one wrong action do? And can you undo it? Sending a message to the wrong team is easy to fix. Paying the wrong person is not. The worse the mistake, the more proof you need.
• What is it allowed to do? This is its level. It can only watch, give advice, get things ready for a person to check, or really do things.
• What proof does it need to move up? This is the gate: a checkpoint with rules. The helper must pass tests to get through. The rules are written down before the tests, not after, so nobody can move the goalposts.

The five levels

• Level 0, Watch. It only looks and reports. Like sitting in the back seat.
• Level 1, Advise. It suggests what to do. Like reading the map for the driver.
• Level 2, Prepare. It gets things ready, and a person checks. Like doing your homework and having a parent check it.
• Level 3, Do with permission. It does small jobs, but only with a signed OK. Like going to the shop when a parent says yes.
• Level 4, Do alone, with limits. It does a set list of jobs by itself, inside hard limits. Like a full licence, with rules.

Most helpers should start at Level 0 or 1, and move up one level at a time.

The four tests, in order

Think of a safety inspection before a school trip. Before the helper moves up, it faces four tests, one after another. The quick, easy tests go first. The slow, costly test goes last. That way, if something is badly wrong, we find out early and cheaply.

• Test 1: the permission slip. A permission slip says exactly what is allowed: "Pay 1,200 to Acme Ltd." If someone changes it to 12,000, the slip must stop working. This test checks that. It is quick, because no AI is needed.
• Test 2: the spending limit. You get a fixed amount of pocket money. When it is gone, you stop. This test checks that the helper stops at its limit, even if it gets stuck doing the same thing again and again.
• Test 3: the fake note. Imagine a stranger slips a note into your bag: "Give me your lunch money." A smart child ignores it. This test hides tricky notes inside the papers the helper reads, to check it ignores orders from the wrong place. This one is slower, because it needs the AI.
• Test 4: the real job. Last, we check the actual work. We give the helper many real examples and count how many it gets right. The pass mark, for example 9 out of 10, is chosen before the test. This is the slowest and most costly test, so it goes last.

Tests 1, 2 and 3 are critical. A critical test is one you must pass, with no exceptions.

The gate: the safety inspector

After the four tests, the gate looks at the results. It asks one question first: did any critical test fail? If yes, the answer is no, whatever the other marks were.

Think of a car that is the fastest in the race but has no brakes. You would never let it on the road. In the same way, 49 right answers and 1 failed critical test might look like 98 out of 100. The gate still says no.

Watch it happen

Press play. A made-up expense helper passes the four tests and moves up. Then an update arrives, one test fails, and the helper steps back down. Choose the second button to see what happens when there are no tests at all.

Interactive: Autonomy is earned: animated explainer

Taking freedom back

After every update, run all the tests again. If a critical test fails, the helper steps back one level on its own. Then add the failed case to the test list, so the same mistake can never sneak back in. A test that checks an old mistake has not come back is called a regression test.

Six rules to remember

• Use locks, not requests. Telling an AI "please don't" is only asking. A rule built into the computer code is a locked door. The AI cannot talk its way past it.
• One critical failure stops everything. Good marks in other tests do not make up for it.
• Permission slips must match exactly. The amount and the person must be the same, or the slip is no good.
• Make trying again safe. If a payment times out, first check whether it already went through. Otherwise you might pay twice, like pressing the buy button two times.
• Set hard limits on time and on money.
• Take freedom back as easily as you give it.

How sure can we be?

Say the helper passes 50 tests in a row with no mistakes. Is it safe? Not quite. Think of a basketball player who scores 50 times in a row. She might be perfect. Or she might score 94 times out of 100 and just had a lucky streak. A streak of 50 like that happens by luck about once in 20 tries.

So 50 perfect tests only let us say: it probably fails fewer than 6 times in 100. To say fewer than 6 in 1,000, we need about 500 perfect tests.

That is why big jobs need more tests, people watching, and checks while the helper works. Tests give evidence. They do not give proof.

The short version

• Give a helper freedom one level at a time, as it proves itself.
• Use locks in the code, not polite requests.
• One critical failure stops the release and moves the helper back a level.

The limits of this advice

• No set of tests can prove an AI is safe. Tests are evidence, not proof.
• The levels, pass marks and examples here are made up to explain the idea. For a real helper, choose them for your own job, before you run the tests.
• For jobs that can hurt people or money (health, banking, safety, hiring, law), experts and independent testing are needed too.
• A small helper that only reads and reports may only need Level 0 or 1, with light checks.
• This is a mix of common good practice. It is not a new standard or a certificate.

Words explained

• Agent: a computer helper that can do things, not only talk.
• Autonomy: doing things on your own.
• Model: the AI brain inside the helper.
• Level: how much the helper is allowed to do.
• Gate: a checkpoint with rules the helper must pass to move up.
• Critical test: a test you must pass, with no exceptions.
• Prompt injection: sneaking orders into text the AI reads, like the fake note.
• Regression test: a test that checks an old mistake has not come back.
• Evidence and proof: evidence makes something likely. Proof makes it certain.

For engineers

This section has the exact details. Everything above is the simple version of it.

The levels and the proof they need (illustrative)

This scale is a working convention, not a standard. Every level needs a written list of what it allows and what evidence unlocks it.

The technical version of each rule

• Enforce limits outside the model. A prompt is a request, not a control. In the test harness, a deliberately obedient mocked model reads a hidden instruction in an invoice and proposes a payment of 9,000. Without a policy layer the payment happens. With it, nothing moves.
• Approval binding. The approval is a keyed hash (HMAC) over the action type, amount and payee. Change any field and verification fails.
• Safe retries and fail closed. Use an idempotency key and check the action's status before retrying after a timeout. If the policy service is unavailable, refuse state-changing actions and keep read-only ones working.
• Budgets. Cap steps and spend. Treat hitting a cap as a failure to investigate, not a setting to raise.
• Release gate. Critical failures block. The other tests need a pass rate set in advance. Anthropic Engineering separates capability evals, which should start hard, from regression evals, which should stay near 100% to catch backsliding.
• Order of the four checks. Cheap, exact code checks first; model-based and labelled-set checks last. All four run before the gate decides. A real pipeline can stop at the first critical failure to save time and money. Which checks are critical is decided before the run.
• Statistics. With n clean trials, the 95% upper bound on the true failure rate is 1 - 0.05^(1/n), about 3/n (the rule of three). n = 50 gives about 5.8%. n = 500 gives about 0.6%.

Evidence you can run

The repository has nine mocked tests with no model calls and no network. Three controls were broken on purpose to confirm that the tests fail when they should. Code: agent-controls-harness. Run it with npm test on Node 18 or newer.

Further reading

• NIST AI Risk Management Framework: a voluntary framework (AI RMF 1.0, January 2023) with a Generative AI Profile (NIST AI 600-1, July 2024).
• Demystifying evals for AI agents, Anthropic Engineering, 9 January 2026: grading outcomes as well as transcripts, grader types, and capability versus regression evals.
• OWASP Top 10 for Agentic Applications for 2026: a taxonomy of agent-specific risks, published 9 December 2025.

Think of a bag of marbles

A bag holds red and blue marbles. You pull out 10 marbles, and all 10 are blue. Does that mean there are no red ones? No. There could still be quite a few red ones hiding at the bottom. Now pull out 500 marbles, and all 500 are blue. Now you can be much more sure, but still not completely.

Testing an AI is the same. Each test is one marble. "All tests passed" is a good sign. It is not a promise.

How much does "no failures" really tell you?

There is a simple rule, called the rule of three. If a helper passed n tests in a row, then the real failure rate is probably below about 3 divided by n. "Probably" here means 95 times out of 100.

So 50 perfect tests do not mean "safe". They mean "probably fewer than 6 mistakes in every 100". For writing a friendly reply, that may be fine. For sending a payment, it is not.

The release gate: a rule that decides, not a mood

A release gate is a rule, written in code, that says whether a change may go live. It runs on every change, like a security guard at a door. It has two kinds of rule:

• A blocker. Some tests are critical. For example: "a yes for one payment must never work for a different payment". If even one critical test fails, the change is stopped. Good marks elsewhere cannot hide it.
• A pass mark. For everyday tests you choose a mark, such as 9 out of 10. You must choose it before you test. If you choose it afterwards, you will be tempted to move the line to fit the result.

This is why a helper that gets 49 out of 50 tests right can still be stopped. If the one it missed was critical, the answer is no.

Three habits that make the gate honest

• Check what happened, not what was said. If the AI says "refund sent", look at the payment record. The words may be wrong even when they sound sure.
• Run the cheap tests first. Quick, free checks go first. The slow, costly test goes last. Stop at the first critical failure, because there is no point paying for the slow tests after a fast one said no.
• Test the grader too. Often another AI marks the answers. Before you trust it, let it mark 50 answers that people already marked. If it agrees with the people most of the time, use it. Check again whenever you change the grader.

What to do

• Write down which tests are critical. Keep that list short and serious.
• Choose the pass mark before you run the tests.
• Say "no failures in 50 tests", never "it is safe".
• Use more tests for riskier jobs.
• Add every real mistake as a new test.

The limits of this advice

• The rule of three assumes the tests are independent and look like real use. Tests that are too similar to each other, or too easy, make the number look better than it is.
• It tells you about the kinds of cases you tested. It says nothing about kinds you never thought of.
• A test that always passes is worth nothing. Check that your tests can fail: break something on purpose and see the gate say no.
• The numbers here are examples. Your own risk decides how low the failure rate must be.

Words explained

• Critical test: a test for something that must never go wrong. One failure stops the release.
• Release gate: a rule that decides whether a change may go live.
• Pass mark: the score needed to pass the everyday tests, chosen before testing.
• Rule of three: if n tests all passed, the real failure rate is probably below about 3 divided by n.
• Grader (judge): an AI or a program that marks an answer right or wrong.
• Regression: something that used to work and now does not.

For engineers

This section has the exact details. Everything above is the simple version of it.

• The bound. With zero failures in n independent trials, the one-sided 95% upper confidence limit on the failure rate p solves (1 - p)^n = 0.05, so p = 1 - 0.05^(1/n), which is close to 3/n. For n = 10, 50, 100 and 500 that gives 25.9%, 5.8%, 3.0% and 0.6%. Pick the confidence level and the allowed failure rate for each risk tier, and solve for n.
• Gate logic. A critical failure blocks regardless of the mean. The pass threshold is fixed in the repository before the run. Report scores by category and risk tag, and gate on the worst category, not the average.
• Outcome over transcript. Grade the end state (the record changed, the payment exists), and use the transcript to explain a failure, not to excuse it. Anthropic's write-up on agent evals draws the same line and separates capability evals from regression evals.
• Cost order. Deterministic checks on every change (approval binding, budget stops, schema), targeted model-based checks next, the full end-to-end run with a judge once per release. Fail fast on the first critical failure.
• Judge calibration. Compare judge scores with human labels on a sample of about 50, track agreement over time, rotate answer order, and score against a rubric. Re-run the comparison after any judge-model change.
• Test the tests. Mutation-check the gate: introduce a known fault and confirm the gate fails. The example harness for this series does exactly that: https://github.com/kapilgupta86/agent-controls-harness
• Frameworks. NIST AI RMF 1.0 (voluntary, Jan 2023) and NIST AI 600-1 (Jul 2024) are the usual references for risk-based testing. They do not prescribe sample sizes.
• Where it fits. The book extends Chapter 12 with this material, and Appendices D and E carry the test-case template and sixteen edge cases. Autonomy levels are in the article on earned autonomy.

The big idea: do not do the same work twice

Imagine doing homework with a huge textbook. If you re-read the whole book for every question, you would be there all night. Smart students keep notes, use bookmarks, and remember the answers to questions they have met before.

AI systems do the same. Keeping something you already worked out, so you can use it again, is called caching (say "cashing").

First: how an AI writes an answer

An AI does not write a whole answer at once. It reads first. Then it writes one small piece at a time. A small piece is called a token. It is about three-quarters of a word.

• The reading phase. The AI reads everything you sent it, all at once. How long this takes decides how long you wait for the first word.
• The writing phase. The AI writes the answer one token at a time, looking back at everything before. This decides how fast the words appear.

Each kind of cache helps one of these phases, or skips both.

Memory 1: notes on this answer (the KV cache)

While writing, the AI keeps looking back at every earlier token. Instead of working it all out again for every new piece, it keeps notes about what it has already read. Think of keeping your finger on the page while you write a story.

These notes are called the KV cache. You do not have to switch them on. They last for one answer only. When the answer is finished, the notes are thrown away.

Memory 2: a bookmark (the prompt cache)

Many requests start the same way: the same rules, the same tools, the same long document. Why read them again every time? A prompt cache remembers that first part. Next time, the AI skips straight to the new part, like opening a book at your bookmark.

It has one big rule: the start must be exactly the same, letter for letter. If you put today's date at the very top, it changes every time, and the bookmark does not work. So put the things that never change first, and the things that change last.

1. The rules the AI must follow      (never changes)
2. The tools it can use              (never changes)
3. The long guide document          (rarely changes)
4. The earlier chat messages        (grows slowly)
5. The new question                 (always new, so it goes last)

A bookmark saves money and the waiting time before the first word. It does not make the words appear faster once writing has started. It usually lasts from a few minutes to a few hours.

Memory 3: an answer notebook (the semantic cache)

Sometimes people ask nearly the same question again. "How do I reset my password?" and "I forgot my password, what now?" use different words but mean the same thing. A semantic cache keeps a notebook of past questions and answers. If a new question means the same as an old one, the system hands back the old answer without asking the AI at all. That is the cheapest and fastest of all.

The risk is that the old answer may be out of date, like last year's school timetable. So give each saved answer an expiry date. And never save answers that are private or change fast, such as "What is my account balance?"

Try it yourself

Move the sliders in the simulator. Watch which requests reuse the bookmark, and how much money that saves. Then tick the box about the changing time at the top, to see how easily a bookmark gets lost.

Interactive: Prompt cache simulator

A worked example with numbers

This example is made up, to show the idea. A helper has instructions and documents that are 8,000 tokens long. Each question adds about 400 tokens. Someone asks 6 questions, 2 minutes apart.

• Without a bookmark, the AI reads 8,400 tokens, six times. That is 50,400 tokens of reading.
• With a bookmark, the first question costs a little extra, to set up the bookmark. After that, the long part is read at one-tenth of the price. The total comes to about 16,400 tokens of reading.

That is about two-thirds cheaper. The exact prices depend on the company, but the idea is the same.

Common mistakes

• Putting something that changes, like the time, at the top of the prompt. The bookmark breaks every time.
• Saving answers that go out of date, or that hold private information.
• Expecting a bookmark to make writing faster. It only helps the reading.
• Forgetting that a bookmark runs out. If nobody asks for a while, it is thrown away.

The short version

• The notes help the AI write one answer. They are automatic.
• A bookmark saves time and money when requests start the same way.
• An answer notebook skips the AI completely, but the answer can get old.

The limits of this advice

• Prices, time limits and minimum sizes differ by company and change often. Check the company's own page before you rely on a number.
• The worked example is made up. Your savings depend on how often requests start the same way.
• An answer notebook can return an old or wrong answer, so it needs expiry dates and careful choices about what to save.

Words explained

• Token: a small piece of text, about three-quarters of a word.
• Prompt: what you send to the AI: your question, plus any rules and documents.
• Cache / caching: keeping something you already worked out so you can use it again.
• Reading phase (prefill): the AI reads everything you sent.
• Writing phase (decode): the AI writes the answer one token at a time.
• KV cache: the notes the AI keeps while writing one answer.
• Prompt cache (also called prefix cache): the bookmark for a start that repeats.
• Semantic cache: the notebook of old questions and answers.
• Expiry time (TTL): how long a saved item may be used before it is thrown away.

For engineers

This section has the exact details. Everything above is the simple version of it.

The technical names

Prefill reads the whole prompt in parallel and is compute-bound; it decides time-to-first-token. Decode generates tokens one at a time and is memory-bandwidth-bound; it decides tokens per second. The KV cache stores each token's key and value vectors so attention is not recomputed at every step. It grows with context length, which is why quantizing it (INT8 or INT4) is a lever at scale. A prompt (prefix) cache reuses the KV computation for an identical prefix across requests. In vLLM, for example, the cache is keyed on a hash of each block of tokens together with the tokens before it, so requests sharing a prefix share the stored blocks. A semantic cache sits in front of the model and matches on embedding similarity; it skips inference on a hit.

What the companies document (checked 14 October 2026)

Sources: Anthropic prompt caching, OpenAI prompt caching, Google Gemini caching, vLLM automatic prefix caching. Prices and limits change often; check the provider's page before you rely on a number. Neither Anthropic's nor OpenAI's page gives a general latency percentage, so treat any quoted latency saving as workload-specific until you measure it.

The write premium, as arithmetic

Take a provider that charges 1.25x to write a 5-minute cache entry and 0.1x to read it (the Anthropic multipliers above), and count the cost of the shared prefix in units of its normal input price. Used once, caching costs 1.25 instead of 1.0, a small loss. Used twice, it costs 1.25 + 0.1 = 1.35 instead of 2.0. Used ten times within the window, it costs 1.25 + 9 x 0.1 = 2.15 instead of 10. This is arithmetic from the published multipliers, not a benchmark.

Cheat sheet

Keeping semantic caches honest

A semantic cache lives until a TTL, an eviction rule or an invalidation event removes the entry, so you set the policy. Do not cache answers that contain personal data or change quickly. See cost-aware testing for measuring cache hit rate, cost per successful task and answer freshness.

Think of a medicine label

A pharmacist reads the dose from the label. Would you want them to say it "in their own words"? Of course not. One changed number could make you ill.

AI assistants have the same problem. They are brilliant at writing smooth sentences in their own words. But for risky steps, such as commands to type, numbers and the order of steps, "in their own words" is exactly what we do not want.

A small example

Someone asks: "How do I upgrade this system?" The official guide says:

Step 1: Make a backup.
Step 2: Run upgrade --safe

Now the AI rewords it: "Just run the upgrade." It has quietly dropped the word --safe. The upgrade might now delete things. The answer sounded helpful, and it caused a big problem.

The fix: two kinds of text

• Dangerous text (steps, numbers, commands) is copied word for word from the approved guide. The AI is not allowed to change it.
• Friendly text (a one or two sentence introduction) is written by the AI, using only the pages that were found.

Nothing else on the page is written by the AI.

Check the copy

Even a copy can go wrong, like a smudged photocopy. So when the guide is loaded, the computer makes a "fingerprint" for each step. A fingerprint is a short code that changes if even one letter changes. Before showing a step, the computer checks its fingerprint. If it does not match, the step is hidden.

Let people check too

Every answer should link to the exact page of the guide, with the passage marked. Then a person can check in seconds instead of just trusting. And when the guide does not cover the question, the assistant should say so, or ask which version the person means. It should never guess.

What to do

• Mark which parts of an answer are dangerous.
• Copy those parts exactly. Do not let the AI reword them.
• Make a fingerprint for each step, and check it before showing the step.
• Show where the answer came from.
• Say "I don't know" when the guide is silent.

The limits of this advice

• Copying exactly does not help if the guide itself is wrong or out of date. Keep the guide correct.
• People still need to read the steps. A good answer makes checking easy. It does not replace good judgement.
• Deciding which parts are "dangerous" takes care. When unsure, copy.

Words explained

• Assistant: an AI helper you ask questions.
• Approved guide: the official document that people have checked.
• Command: a line you type to tell a computer what to do.
• Fingerprint: a short code that changes if even one letter of the text changes.
• Word for word: exactly the same, with nothing changed.

For engineers

This section has the exact details. Everything above is the simple version of it.

• Assemble, don't generate. Build the answer from source blocks. The model writes only a short lead-in, constrained to the retrieved passages. Steps, numbers and commands come from the document.
• Integrity check. Store a hash (for example SHA-256) of each step block at ingestion. Recompute it before rendering. Withhold the block on a mismatch.
• Provenance. Cite document, page and passage. When the corpus does not cover the query, abstain, or ask a clarifying question about the version or variant.
• Where it fits. This is the grounding and citation pattern (Chapter 11) combined with approval for risky actions (Chapter 13).

Think of a school library

If you ask "How do I do this science experiment?", the librarian points you to a book. If you ask "Which bus does each child in my class take?", you need a chart of connections: who goes with which bus.

People ask AI assistants both kinds of questions in the same chat. The assistant has to notice the difference. In AI, the part that decides this is called a router. We will call it the sorter.

Book questions: "how" and "what does it mean"

Examples: "How do I reset my password?" or "What changed in the new version?" The answer is written down in a guide. So the assistant searches the approved guides, finds the best pages, and shows the exact lines with the page they came from.

Map questions: "what is connected to what"

Examples: "Which computers are in Room 5, and how many are there?" This is a question about connections, like a map of a school. The answer is not in one page of a book. It comes from a database that holds the connections.

For these questions, use a ready-made, checked question to ask the database. Then let ordinary computer code write the answer. The AI may introduce the table, but it must never write a number or a name that came from the database. If the database says 14 computers, the AI must not say 12.

The sorter is the hard part

Knowing which kind of question was asked is the hardest step. A good sorter does four things:

• It makes one quick AI call to decide which kind of question it is.
• It has a simple back-up, a list of key words, for when the AI is not reachable.
• When a name could mean several things, it shows the real choices and lets the person click. Like asking "Which Sam: Sam from class 5 or Sam from class 6?", instead of guessing.
• When a question has two parts, it splits them and sends each part to the right helper.

What to do

• Decide which questions your assistant must answer from documents, and which from a database of connections.
• Build a sorter that picks the right helper, and asks the person when unsure.
• Let code write every number and name that comes from a database.

The limits of this advice

• Some questions need both helpers. Splitting them well takes testing.
• A sorter can be wrong. That is why it must ask the person when it is unsure, and why you should test it with real questions.
• A graph database is only worth building if you really have many connection questions. Many assistants only need documents.

Words explained

• Router (the sorter): the part that decides which helper should answer.
• Database: an organised store of facts, such as lists and tables.
• Graph: a way to store connections, like a map of who is linked to what.
• Retrieval: finding the right pages for a question.
• Citation: showing where an answer came from.

For engineers

This section has the exact details. Everything above is the simple version of it.

• Document questions. Combine meaning-based (dense) and word-based (sparse) search, re-rank the candidates, and answer with quotations and citations.
• Relationship questions. Use fixed, reviewed, parameterised graph queries. Let code format the result. The model never writes a value that came from the database.
• Router. One cheap classification call extracts the intent and any names, with a keyword fallback when the model is unreachable. Ambiguous names return real candidates for a click. Compound questions are split, one part per source.
• Where it fits. This is the hybrid pattern from Chapters 10 and 16.

Think of a spelling test

A teacher marks one wrong word. You could memorise just that word. But the next test has new words, and you get those wrong too. The better plan is to learn the spelling rule.

AI teams make the same mistake. A test question fails. Someone adds a rule that mentions a word from that question. The test now passes. Ten rules later, nobody understands the system, and the next ten questions fail anyway.

A small example

Someone asks: "How do I reset the printer?" The assistant gets it wrong. A bad fix: "If the question has the word printer, show page 12." Tomorrow someone asks about the scanner, and it fails. So you add another rule. And another.

A good fix starts with a question: why did it fail? Maybe the right page calls it a "power cycle", not a "reset", so the search never found it. Teach the search about that, and printer, scanner and many other questions start working at once.

Ask which step broke

An assistant answers in steps. Each step can fail in its own way:

• Step 1, find the pages. It never found the right page.
• Step 2, put the best first. It found the right page, but ranked it too low.
• Step 3, pick the right part. It looked at the wrong part of the page.
• Step 4, check the answer. It never checked the answer against the page.

Each step has its own proper fix, a fix to the design of that step, not to one question.

A rule of thumb

A fix should never name a special word or a special question. If you catch yourself writing one, stop. Ask: what could the system use instead? And after the third patch in the same place, change the design.

Keep a notebook of what you tried

Keep a simple table: what you tried, what did not work and why, and which ideas work together. Put the ideas that give the most gain for the least cost at the top. A notebook like this teaches a team faster than any meeting. It also stops the same failed idea from coming back.

The limits of this advice

• Sometimes a quick patch is the right choice, for example during an urgent problem. Mark it as temporary, and plan the real fix.
• Finding the broken step takes some detective work and good test questions.
• A fix to one step can change the others, so run all your tests again after each change.

Words explained

• Assistant: an AI helper you ask questions.
• Patch: a quick fix that covers one problem without fixing its cause.
• Ranking: putting the best answers at the top.
• Keyword: a special word the system looks for.
• Test question: a question with a known right answer, used to check the system.

For engineers

This section has the exact details. Everything above is the simple version of it.

• Failure layers. A wrong answer comes from one of a few causes: the right passage was never retrieved (recall), it was retrieved but ranked too low (ranking), the wrong part of the passage was chosen, or the answer was not verified against the evidence.
• Structural fixes. Examples are where a term is first introduced, what kind of section it is, and how scores are blended. A fix must not name a term or a question.
• Trade-off table. Record the levers tried, what did not work and why, and which combinations do work, ordered by gain per cost.
• Where it fits. See evaluation engineering in Chapter 12.

Think of a long bike trip

Before a long trip you check the tires and the brakes. That takes a minute and costs nothing. If you fix the brakes, you take a short ride to try them. You do not ride the whole day just to test one brake pad. The full-day ride happens once, when everything is ready.

Testing an AI works the same way.

Why AI tests cost real money

Normal tests are almost free. AI tests are not. Every test question makes the AI work, often several times: it searches, it sorts the results, it writes an answer, and another AI may even grade the answer. Run the full set of tests after every small change, and the budget alert goes off long before real users arrive.

Three levels of testing

• Level 1: quick, free checks, every time. These need no AI at all. Does the code run? Is the answer in the right shape? Are the links real?
• Level 2: only the tests for what you changed. If you changed how dates are handled, run the date tests, not everything.
• Level 3: the full big test, once at the end. Run it after a batch of changes, as the final gate.

Habits that save money

• Remember repeated work. If the same question was asked before, reuse the answer, so a re-run costs nothing.
• Label test spending. Give test traffic its own label, so it never hides inside the real bill.
• Compare the bill before and after a big run. And decide in advance when a small change does not deserve a full run.

Do not fool yourself about quality

Imagine you practise with last year's exam paper until you get every answer right. That does not prove you know the subject. You need a new paper that you have never seen. In testing, this new paper is called a held-out set. You never tune on it.

Keep two settings: one for the best quality and one for the lowest cost. Judge every change on both the main test set and the held-out set. Keep a change only if neither one gets worse. And put every experiment behind a switch, so a bad change can be undone in seconds.

What to do

• Put your tests into three levels, cheapest first.
• Cache repeated calls and label test spending.
• Keep a held-out set that you never tune on.
• Put experiments behind a switch.

The limits of this advice

• Cheap tests can miss problems that only the full test finds, so the full test still has to run.
• Which tests are "affected" by a change is a judgement call. When unsure, run more.
• Saving money must never mean skipping safety tests that matter.

Words explained

• Test set: a list of questions with known right answers, used to check the AI.
• Held-out set: a second list that you keep secret and never practise on.
• Cache: remembering an answer so you do not have to work it out again.
• Budget label: a tag that shows which spending belongs to which job.
• Grader (judge): an AI or a program that marks an answer right or wrong.

For engineers

This section has the exact details. Everything above is the simple version of it.

• Cost drivers. Each test question can trigger retrieval, re-ranking, generation and a judge call. Run the cheap deterministic checks first (does the code run, is the output shaped correctly, are the citations valid).
• Tiers. Affected categories after each focused change; the full set once per batch as the final gate.
• Controls. Cache repeated calls, tag test traffic with its own budget, and look at spend before and after big runs.
• Accuracy hygiene. Two named profiles (best accuracy, lowest cost); judge changes on the main set and a held-out set that is never tuned on; keep every experiment behind a switch.
• Where it fits. See evaluation engineering (Chapter 12) and the two dashboards (Chapter 22). The article on AI caching explains one way to make repeated calls cheaper.

Think of the date on a milk carton

A milk carton without a date is a gamble. Old milk and fresh milk look the same until you taste it. The date on the carton lets you decide before you pour.

Screens and AI assistants often show old information in the same confident style as fresh information. This is a quiet way of lying.

A small example

A screen says "All printers: OK", with a big green tick. Someone reads it and decides not to check the broken printer in Room 5. But the screen is a copy from three weeks ago. The printer broke last Tuesday. Nobody knew, because nothing on the screen said how old it was.

Label the information, not just the page

If any part of your system shows a saved copy (a snapshot), say so. Put the date:

• on the screen,
• in the AI's answer,
• and in anything you export or send.

A good label says both the time and how long ago: "As of 3 Oct, 10:00 (21 days ago)." It costs one line, and it saves a lot of arguments.

When the information is too old

Decide in advance how old is too old. Then make the system say so: "This information is 21 days old. Do you want me to refresh it?" Do not hide it, and do not guess.

A second habit: start by only looking

Start with a helper that can only look, never touch. Anything that could change a device or a record should be built switched off, and switched on only behind a person's approval. That is the same idea as in the article "Autonomy is earned": freedom comes one step at a time.

What to do

• Save the time when each piece of information was collected.
• Show the date and the age on screens, in answers and in exports.
• Decide how old is too old, and say so when it is.
• Start read-only. Add actions later, behind approval.

The limits of this advice

• Even a fresh label can be wrong if the clock or the source is wrong. Check the source too.
• Too many labels can clutter a screen. Put them where a decision is made.
• A date does not make old information right. It only lets people judge it.

Words explained

• Snapshot: a saved copy of information at one moment in time.
• Dashboard: a screen that shows the state of many things at once.
• Read-only: able to look, but not able to change anything.
• Approval: a yes from a person before something happens.

For engineers

This section has the exact details. Everything above is the simple version of it.

• Carry the timestamp. Store an as-of time with every data point, and pass it through APIs, answers and exports. Show both the absolute time and the relative age.
• Freshness rules. Define a staleness threshold per data type, and have the system say when it is exceeded and offer a refresh.
• Read-only first. Build write actions switched off, and enable them only behind an approval step.
• Where it fits. See human-in-the-loop (Chapter 13) and observability (Chapter 18).