Retrieval-augmented generation (RAG) gives a language model documents to consult. It is the default way to connect AI to an organisation’s information, and for good reason: it is quick to build and useful. Compile-Time Inference is Verificate’s approach to the cases where RAG’s trade-offs are the wrong ones. This comparison covers both honestly, including what Compile-Time Inference costs you.
The short version
RAG: find passages, then ask the model to answer. Compile-Time Inference: prepare the knowledge, select a supported result, then ask the model to explain it. The difference is which step decides the answer.
Side by side
| Typical generative RAG | Compile-Time Inference | |
|---|---|---|
| Where most work happens | When the question arrives. | Before questions arrive, then maintained. |
| Who decides the answer | The language model, using retrieved passages. | For supported tasks, software selecting from prepared records and rules. |
| The language model’s job | Read the passages and write the answer. | Interpret the question and explain the selected result. |
| Same question twice | May differ, because generation involves sampling. | Supported decisions repeat for the same inputs, records and rules. Wording may vary. |
| Missing evidence | Needs an explicit refusal design, or the model may fill the gap. | Designed to decline when support is absent. Test this on your questions. |
| When data changes | Re-index the changed documents. | Update the prepared knowledge. This has a cost and takes time. |
| Set-up effort | Lower. Quick to prototype. | Higher. Scope, sources and tasks are agreed first. |
| Best suited to | Search, exploration, broad or unpredictable questions. | Bounded questions and repeated decisions where evidence matters. |
Where RAG is the right choice
- Finding and summarising documents across a large, varied collection.
- Exploratory questions nobody could have predicted.
- Drafting, where a person will review and rewrite the result.
- Early projects, where you are still learning what people ask.
If that describes your need, keep RAG and improve it. Why RAG still hallucinates lists the fixes worth trying first.
Where Compile-Time Inference is worth evaluating
- Answers must point to a record a person can open and check.
- The same situation must lead to the same decision, under an agreed policy.
- The system must say when it does not have enough to answer.
- Versions matter: which record applied, and when.
- A wrong answer is expensive.
What “repeatable” does and does not mean
Repeatability applies to supported decisions with the same interpreted inputs, authorised records and rules. New evidence can, and should, change a result. The wording of an explanation may vary. And repeatability is not correctness: a system can be consistently wrong if its records are wrong. Measure correctness separately.
What it costs
Compile-Time Inference does not remove cost; it moves it. Preparation takes effort up front. Updates must be processed as records change. Storage grows. In exchange, less work is repeated on each question. Whether that is cheaper depends on how often your information changes and how many questions you answer. Include preparation, updates, storage, serving and human review in any comparison — not only the price of a model response.
There is also a scope cost. Compile-Time Inference works on tasks that have been agreed and prepared for. A question outside that scope should be declined, where a RAG system would attempt an answer. For some uses that is the point; for others it is a limitation.
You may want both
These approaches are not rivals in every organisation. A common pattern is RAG for search and exploration, and Compile-Time Inference for the narrower set of answers and decisions that must be checkable and repeatable. Start with the few questions where a wrong answer hurts most.
How to compare them fairly
- Write the test questions first, including unanswerable ones.
- Configure your current system as well as you can before comparing.
- Report citation validity, claim support, correctness, repeatability, coverage and refusal separately.
- Count total cost over a realistic period, including updates.
The evaluation guide sets this out in detail, and Evidence & evaluation shows how our own published results are scoped. To see the approach applied to video, try the broadcast demonstration.