Adrian Spiridon

Build-Time vs Runtime AI: An Instruction–Data Variability Framework

Note

TL;DR: AI can be useful in two very different places: while building software, or inside the software at runtime. My rule of thumb is simple: if both the instruction and the input space can be specified at build time, use AI to help build the solution, but do not keep it in the runtime path. Runtime AI starts to earn its place when the instruction, the input space, or both remain open-ended until runtime. The 2x2 matrix below is a way of making that decision explicit.

Almost every product seems to be "AI-powered" now.

A search box becomes an AI search box.

A dashboard gets AI insights.

A workflow becomes an AI workflow.

A feature that previously consisted of a few rules and database queries suddenly has an LLM somewhere in the runtime path.

I do not think this is necessarily dishonest. The feature may genuinely call an AI model.

My problem is more architectural:

Just because AI can perform a function does not mean AI should remain part of that function at runtime.

There are two very different questions hidden inside the phrase "use AI":

  • Can AI help us build the solution?

  • Does the solution actually require AI while it is running?

I think we are currently mixing those two questions far too often.

1. Build-time AI is not runtime AI

Suppose the requirement is:

If an invoice is more than 30 days overdue and the amount exceeds EUR 10,000, send it to manual review.

AI can help me understand the requirement, write the code, generate tests, review the implementation and even produce the deployment configuration.

But once the behavior is known, the runtime solution is still something equivalent to:

if overdueDays > 30 && amount > 10000:
    manualReview()

Putting an LLM in that runtime decision would be a regression.

I would be replacing an exact rule with a probabilistic approximation of the same rule, while adding latency, cost, operational dependency and a new failure mode.

The fact that AI helped build the software does not make runtime AI necessary.

That distinction seems obvious when stated this way, yet a surprising amount of today’s "AI-powered" functionality appears to ignore it.

2. Runtime AI is for what remains unknown at build time

Now change the requirement:

Determine whether the customer is disputing this invoice based on whatever material they submitted.

The objective is still clear, but the input may be an email, a PDF, a scan, a chat transcript, several attachments, unusual wording or a language nobody anticipated when the system was implemented.

I can no longer reasonably encode every relevant input pattern as deterministic rules.

Runtime AI starts to make sense.

Or consider:

Analyze this dataset and tell me why churn increased.

Here even the operation itself is partly supplied at runtime.

Tomorrow the user may ask:

Compare premium customers in Q2 with last year, but ignore Germany and separate acquisition decline from reduced spending by existing customers.

Nobody implemented that exact workflow beforehand.

The system is being asked to construct part of the solution from the runtime instruction.

That is where LLMs become genuinely interesting.

So the distinction I find useful is:

Traditional software executes a solution defined beforehand. Runtime AI is most valuable when part of the solution still has to be inferred or constructed at runtime.

3. Instructions and data

While thinking about this, I kept coming back to the two dimensions used in Flynn’s taxonomy: instructions and data.

This is only an analogy, not an extension of Flynn’s taxonomy. Flynn classifies computer architectures according to concurrent instruction and data streams: SISD, SIMD, MISD and MIMD. I am borrowing the same two dimensions to ask a different question: how much variability remains in the instructions and data when the software is actually running?

For an application, ask:

  • How much of the instruction is known at design time?

  • How much of the input space is known at design time?

That produces a simple 2x2 matrix:

                                  INPUT SPACE
                       PREDICTABLE          OPEN-ENDED
                  +---------------------+---------------------+
                  |                     |                     |
 PREDEFINED       | DON'T USE AI        | RUNTIME AI MAY      |
 INSTRUCTION      | AT RUNTIME          | BE JUSTIFIED        |
                  |                     |                     |
                  | Deterministic       | Interpretation /    |
                  | software            | perception          |
                  +---------------------+---------------------+
                  |                     |                     |
 AD-HOC           | RUNTIME AI IS       | STRONGEST CASE FOR  |
 INSTRUCTION      | LIKELY USEFUL       | RUNTIME AI          |
                  |                     |                     |
                  | Intent / planning   | AI-native problem   |
                  +---------------------+---------------------+

The important point is that "predictable data" does not mean the values never change.

A customer ID can be 123 today and 456 tomorrow. A price can move from 100 to 120. That is still a predictable input space if the structure and semantics are known.

What matters is whether the system designer can reasonably anticipate what kind of thing will arrive and what the application is supposed to do with it.

The matrix is therefore not just a classification of AI use cases. It is a runtime-AI decision matrix. The bottom-left quadrant is the important baseline: if both the instruction and the input space are known at design time, AI may still be extremely useful at build time, but there is usually little reason to keep it in the runtime path.

3.1. Quadrant 1: predefined instruction, predictable data

This is ordinary software.

Calculate VAT.

Validate an IBAN.

Apply an accounting rule.

Check permissions.

Route an order according to a known workflow.

Query a table using predefined filters.

These systems can be extremely complicated. Complexity by itself does not make something an AI problem.

If the behavior and input model are known, I would generally prefer deterministic code.

AI may be extremely useful during implementation, but there is little reason to keep it in the runtime architecture.

This is the quadrant where "AI-powered" most often feels like marketing rather than engineering necessity.

3.2. Quadrant 2: predefined instruction, open-ended data

Here the task is fixed, but the possible inputs are difficult to enumerate.

Examples include:

  • Is this arbitrary email a complaint?

  • Is there a pedestrian in this image?

  • What text is present in this scan?

  • Does this document contain evidence of a contractual dispute?

The instruction is stable.

The difficulty is the input space.

This has always been natural territory for machine learning: vision, speech recognition, classification, semantic interpretation and similar tasks.

The AI handles the fuzzy part; deterministic software can handle what happens next.

3.3. Quadrant 3: ad-hoc instruction, predictable data

This is particularly interesting for LLMs.

The data may be perfectly structured and well understood — for example, a relational database with a stable schema.

What changes is the operation the user wants to perform.

Show me whether the decline came mainly from fewer new customers or lower spending by existing ones, excluding France.

The database did not change.

The application did not have a button called exactly that.

The instruction arrived at runtime.

An LLM can interpret the request, create a query or execution plan, and hand the actual execution to deterministic tools.

This is a much stronger use of runtime AI than asking an LLM to recalculate a number the software already knows how to calculate.

3.4. Quadrant 4: ad-hoc instruction, open-ended data

This is the AI sweet spot.

Here are seventeen documents, several emails and two spreadsheets. Work out why this project failed and identify the three most likely causes.

Neither the exact operation nor the shape of the relevant input was completely known when the application was built.

The problem itself becomes partly a runtime object.

This is where agents and LLM-based systems feel fundamentally different from conventional applications rather than merely bolted onto them.

They can receive a problem that did not exist when the software was written and construct a plausible procedure for solving it.

4. Make deterministic what can be deterministic

This leads to the architectural principle I find most useful:

Make deterministic everything that can reasonably be made deterministic. Use runtime AI for the irreducibly open-ended remainder.

A good AI architecture therefore often looks less like:

everything
    |
    v
   LLM
    |
    v
 result

and more like:

open-ended instruction / input
            |
            v
           AI
 interpretation / planning
            |
            v
   structured command
            |
            v
 deterministic software
 rules / queries / transactions
            |
            v
          result

The AI deals with ambiguity.

The deterministic system deals with certainty.

This separation also gives us much better control over testing, security, reproducibility and failure handling.

An LLM may decide that the user intends to refund an order.

It should not therefore invent the refund transaction itself.

Once the intent has been converted into a known command, the existing deterministic system can validate authorization, calculate the amount, enforce limits, execute the transaction and record the result.

5. AI can sometimes eliminate itself

There is another consequence I think is underappreciated.

Sometimes runtime AI is useful because we do not yet know the rule.

After observing enough cases, however, we may discover that what looked fuzzy is actually governed by a stable pattern.

At that point the right architectural move may be to remove AI from that part of the runtime path and encode the discovered behavior directly.

In other words, successful AI usage can sometimes eliminate the need for itself.

AI helped us discover, prototype or specify the solution.

Once the solution became known, deterministic software became the better implementation.

That is almost the opposite of the current tendency to add AI permanently to every layer simply because it is available.

6. The question I would ask before adding runtime AI

So instead of asking:

Can we use AI here?

I would ask:

What remains unknown until runtime?

If the answer is "nothing important," I probably do not want AI in the runtime path.

Use AI aggressively to help design the solution, implement it, test it and improve it.

But if both the instruction and the input space are already known, let the resulting software do what software has always been very good at: execute the known behavior exactly.

If the instruction is ad-hoc, the input space is open-ended, or both, then runtime AI starts to earn its place.

That is the distinction I find much more useful than whether a product can put AI-powered on the feature list.

<< Previous Post

|

Next Post >>