Verificate
Articles·News & press·14 Sept 2026·5 min read

The most expensive place to do your thinking is at query time

This is the short form of an essay Craig Atkinson published with AI Infra Summit on 14 September 2026, the day before Verificate launched Compile-Time Inference in Santa Clara. The full piece is worth reading; the argument is summarised here.

The arithmetic

Three years ago, running a frontier-class language model cost roughly a thousand times what it costs today. By any normal logic the bills should have followed. They did not:

  • OpenAI's inference spending rose from $8.4 billion in 2025 toward a projected $14.1 billion this year.
  • The average enterprise AI budget grew from $1.2 million in 2024 to about $7 million in 2026.
  • Between 55% and 80% of corporate GPU spend now goes to running models, not training them.
  • Typical enterprise deployments report utilisation of 15–30%. The machines are not thinking; they are waiting — for memory, for the next request, for a user to finish typing.

Put together, the industry has become dramatically more efficient at a unit of work it performs far too often, in the least efficient way available, at the worst possible moment.

Modern AI systems are interpreted

Nearly every corporate AI system built in the past three years has one shape. A question arrives; the system searches a document store, retrieves passages, packs them into a prompt and asks a model for an answer. That is retrieval-augmented generation, and it makes a decision most people never notice: it defers the expensive work — the searching and the reasoning — until the instant a person is waiting for a reply. Ask the same thing twice and you pay twice. It also degrades as it succeeds: more documents mean more candidates of lower relevance, context windows fill, systems truncate, and truncation is among the most reliable ways to make a model invent an answer.

Computing settled this argument in the 1970s. Interpreted languages analyse a program every time it runs; compiled languages do the analysis once and leave an artifact behind. Modern AI systems are interpreted. They re-derive their understanding of the same corpus on every question.

Compile once, at ingest

The alternative — the reason it is called compile-time inference — is to move the thinking to the moment the data arrives. When a document, a broadcast frame or a transaction enters the system, that is when the facts are extracted, scored for quality, stripped of duplicates and checked against one another, then compiled into a live, cited structure. When a question finally arrives, the system selects from that structure instead of reasoning across the archive from scratch. For a data-centre operator the consequences run opposite to the current design in every dimension that matters:

  • Batchable. You control the queue, so ingest can run at high utilisation rather than 15%.
  • Schedulable. It can wait for cheap capacity, off-peak power, spot instances or a quiet night.
  • Paid once. A fact extracted today serves every question asked about it for the next decade.
  • Not latency-critical. Nobody drums their fingers while a frame is indexed, so the work can run on hardware that would be unacceptable in a request path — ordinary processors included.

Query time becomes a lookup against a compact compiled structure, and lookups do not slow down because the archive grew. The bill stops scaling with questions asked and starts scaling with data ingested — a number that can be forecast, capped and scheduled.

Ontology at the centre, model at the edge

Verificate is not the first to argue that the model sits in the wrong place. Palantir has made a version of this case for years, with more commercial success than anyone: put the ontology at the centre and the model at the edge, as a replaceable interface to a governed system of record. That is correct. The disagreement is narrow: in Foundry the ontology is largely modelled by hand, through a substantial services engagement, with a semantic layer whose shape is owned by a vendor. Verificate's bet is that the ontology can be compiled from the customer's own data by machine — scored, deduplicated and bound to the source artifact each fact came from — with the human role shifting from modelling to governing, and the semantic layer staying inside the customer's estate. It may be wrong. But it is the assumption in the incumbent design most worth testing, and it can be tested.

Determinism is an infrastructure property

When an answer is selected from a compiled structure rather than sampled from a model, identical facts produce identical outputs — not because the temperature is zero, but because the decision was never a sampling problem. The output changes when the data changes, and at no other time. Every claim carries its receipt: the source, the timestamp, the artifact a reader can open and check. When the record does not cover the question, the system says so and stops. A sampled system cannot structurally make that offer, because it will always produce something.

Engineers will recognise the difference as the one between a cache and a guess.

The uncomfortable question

If your inference bill is climbing while token prices fall, cheaper tokens will not rescue you. Volume is the problem, and volume is a consequence of when you chose to do the work. The question worth putting to any AI system in production is not which model it runs or how many chips it has. It is simpler:

How much of what it computes today did it already compute yesterday? For most systems now in production, the honest answer is nearly all of it.

Read more

FAQ

Why are enterprise AI bills rising when token prices have collapsed?

Because volume, not unit price, is the problem. Retrieval-augmented generation defers the expensive work — searching a corpus and reasoning over it — to query time, so every question pays the full cost again, on hardware built for throughput but used for latency. Typical enterprise deployments report GPU utilisation of only 15–30%.

What is compile-time inference?

Compile-time inference moves the thinking to the moment data arrives. When a document, a broadcast frame or a transaction enters the system, the facts are extracted, scored, deduplicated and checked against one another, then compiled into a live, cited structure. A query becomes a lookup against that structure instead of fresh reasoning over the archive.

Does compile-time inference need GPUs?

Ingest-time work is batchable, schedulable and not latency-critical, so it can run at high utilisation on cheap or off-peak capacity, including ordinary CPUs. Query-time lookups do not slow down as the archive grows. The bill scales with data ingested — a number that can be forecast, capped and scheduled — rather than with questions asked.

How does compile-time inference differ from Palantir's ontology approach?

Both put the governed representation of the business at the centre and the language model at the edge. The difference is how the ontology is built: Palantir's Foundry ontology is largely modelled by hand through a services engagement; Verificate's bet is that it can be compiled from the customer's own data by machine, scored, deduplicated and bound to source artifacts, with humans governing rather than modelling — and the semantic layer staying inside the customer's estate.

Is the output deterministic?

Yes. When an answer is selected from a compiled structure rather than sampled from a model, identical facts produce identical outputs. Every claim carries its source, timestamp and artifact, and when the record does not cover the question the system says so and stops — something a sampled system cannot structurally offer.

Own the answer. Compile the record.

Verificate Compile-Time Inference is the alternative to RAG: a public fact base compiled for your domain, your records written on top, and every answer resolving to its source — on infrastructure you control.

See Compile-Time Inference Download the brochure (PDF)Talk to us