Test the cheap things first: why your AI budget alert came from your tests
Think of a long bike trip
Before a long trip you check the tires and the brakes. That takes a minute and costs nothing. If you fix the brakes, you take a short ride to try them. You do not ride the whole day just to test one brake pad. The full-day ride happens once, when everything is ready.
Testing an AI works the same way.
Why AI tests cost real money
Normal tests are almost free. AI tests are not. Every test question makes the AI work, often several times: it searches, it sorts the results, it writes an answer, and another AI may even grade the answer. Run the full set of tests after every small change, and the budget alert goes off long before real users arrive.
Three levels of testing
- Level 1: quick, free checks, every time. These need no AI at all. Does the code run? Is the answer in the right shape? Are the links real?
- Level 2: only the tests for what you changed. If you changed how dates are handled, run the date tests, not everything.
- Level 3: the full big test, once at the end. Run it after a batch of changes, as the final gate.
Habits that save money
- Remember repeated work. If the same question was asked before, reuse the answer, so a re-run costs nothing.
- Label test spending. Give test traffic its own label, so it never hides inside the real bill.
- Compare the bill before and after a big run. And decide in advance when a small change does not deserve a full run.
Do not fool yourself about quality
Imagine you practise with last year's exam paper until you get every answer right. That does not prove you know the subject. You need a new paper that you have never seen. In testing, this new paper is called a held-out set. You never tune on it.
Keep two settings: one for the best quality and one for the lowest cost. Judge every change on both the main test set and the held-out set. Keep a change only if neither one gets worse. And put every experiment behind a switch, so a bad change can be undone in seconds.
What to do
- Put your tests into three levels, cheapest first.
- Cache repeated calls and label test spending.
- Keep a held-out set that you never tune on.
- Put experiments behind a switch.
The limits of this advice
- Cheap tests can miss problems that only the full test finds, so the full test still has to run.
- Which tests are "affected" by a change is a judgement call. When unsure, run more.
- Saving money must never mean skipping safety tests that matter.
Words explained
- Test set: a list of questions with known right answers, used to check the AI.
- Held-out set: a second list that you keep secret and never practise on.
- Cache: remembering an answer so you do not have to work it out again.
- Budget label: a tag that shows which spending belongs to which job.
- Grader (judge): an AI or a program that marks an answer right or wrong.
For engineers
This section has the exact details. Everything above is the simple version of it.
- Cost drivers. Each test question can trigger retrieval, re-ranking, generation and a judge call. Run the cheap deterministic checks first (does the code run, is the output shaped correctly, are the citations valid).
- Tiers. Affected categories after each focused change; the full set once per batch as the final gate.
- Controls. Cache repeated calls, tag test traffic with its own budget, and look at spend before and after big runs.
- Accuracy hygiene. Two named profiles (best accuracy, lowest cost); judge changes on the main set and a held-out set that is never tuned on; keep every experiment behind a switch.
- Where it fits. See evaluation engineering (Chapter 12) and the two dashboards (Chapter 22). The article on AI caching explains one way to make repeated calls cheaper.