Adrian Spiridon

The Generation–Evaluation Asymmetry

This summer, while watching the World Cup with some friends, I criticized a player’s execution.

The response came immediately:

Would you have scored from there?

Obviously not.

But that was also irrelevant.

I did not need to be a better football player than him to observe that, by the standards of professional football, the execution was poor.

That small argument stayed with me because it points to a much broader phenomenon:

Producing something and evaluating something that has already been produced are different tasks.

And, very often, the second one seems easier than the first.

I will call this the generation–evaluation asymmetry.

It turns out that versions of this distinction appear in several completely different fields — from cognitive psychology to computer science.

And with the arrival of generative AI, it has suddenly become extremely relevant to software development.

TL;DR

  • Producing a candidate and evaluating an existing candidate are different cognitive tasks.

  • Across several disciplines, we repeatedly find related distinctions: recall vs recognition, generation vs evaluation, productive vs receptive knowledge, and search/solve vs verification.

  • P vs NP provides a particularly rigorous computer-science analogy: efficiently verifying a proposed solution does not automatically imply that we can efficiently find one.

  • I have always found code review, codebase understanding, auditing and refactoring more natural than generating equivalent functionality from a blank page.

  • Generative AI changes the software-development workflow by making candidate generation dramatically cheaper.

  • In a Human-in-the-Loop workflow, the developer increasingly spends time on understanding, constraining, evaluating and validating AI-generated candidates.

  • AI does not make validation trivial. It makes generation cheap enough that specification and validation become relatively more important — and potentially the new bottlenecks.

1. You are not solving the same problem

The football objection sounds reasonable at first:

If you could not perform the action yourself, how can you judge someone who can?

But it assumes that execution and evaluation require the same ability.

They do not.

The player has to perceive the situation, decide what to do, coordinate the movement, get the timing right and execute — all in real time.

The spectator receives something the player did not have:

the completed candidate.

We see the situation, the chosen action and its result.

Our task is no longer:

Produce the right execution.

It is:

Evaluate this execution against some standard.

That does not make the spectator automatically correct. We may lack information about what the player saw, the tactical instructions, the physical constraints or the available alternatives.

Evaluation can absolutely be wrong.

But the important point is that being unable to generate an outcome does not imply being unable to discriminate between better and worse outcomes.

The tasks are different.

2. The computer-science version: solving versus verifying

This distinction has an unusually rigorous counterpart in theoretical computer science.

The famous P versus NP problem asks, roughly:

If a solution can be checked efficiently, can it also always be found efficiently?

More precisely, P is the class of decision problems that can be solved in polynomial time by a deterministic algorithm.

NP stands for nondeterministic polynomial time. One equivalent characterization is the class of decision problems whose YES answers have certificates that can be verified in polynomial time by a deterministic algorithm.

Take the Hamiltonian cycle problem.

Given a graph, ask:

Is there a cycle that visits every vertex exactly once and returns to the start?

Finding such a cycle appears difficult.

But if somebody gives me one:

A -> F -> D -> B -> C -> E -> A

checking it is straightforward.

I verify that every vertex appears exactly once, that every required edge exists and that the path returns to the beginning.

The candidate changes the problem.

This does not mean that "P means generation" and "NP means verification." Formally, P ⊆ NP, and whether P = NP remains one of the great open questions in computer science.

What interests me here is the underlying distinction:

problem -> find candidate

versus:

problem + candidate -> evaluate candidate

Most complexity theorists believe that P != NP, although nobody has proved it.

Our everyday experience certainly makes that belief feel intuitive: the world seems full of situations where recognizing a valid solution is considerably easier than discovering one from scratch.

That intuition is not a proof of P != NP.

But it is a useful way of thinking about the difference between search and verification.

3. The same pattern appears elsewhere

Once I started thinking about it, I noticed versions of the same distinction in many other disciplines.

In cognitive psychology there is recall versus recognition.

It is generally easier to answer:

Which one is the capital of Mongolia: Astana, Bishkek or Ulaanbaatar?

than:

What is the capital of Mongolia?

In the first case, the candidate is present.

In the second, you must generate it from memory.

In creativity research, scholars explicitly distinguish idea generation from idea evaluation and selection.

One process creates candidates.

Another judges, selects and refines them.

In language learning, researchers distinguish receptive and productive knowledge.

You may immediately understand a word when you hear it, while being unable to spontaneously produce that same word while speaking.

Sport has its own literature around perceptual-cognitive expertise: recognizing patterns, anticipating events and making judgments are themselves skills, distinct from physically executing the action.

The terminology changes:

generation       <-> evaluation
recall           <-> recognition
productive       <-> receptive
solve / search   <-> verification
execution        <-> judgment

They are not identical concepts.

But they all expose a similar asymmetry:

Having the candidate available can make the cognitive problem fundamentally different from having to produce the candidate yourself.

4. I had already noticed this in programming

Long before generative AI, I had noticed something similar in my own work.

I have always found it easier to enter an existing codebase, understand how it works, identify inconsistencies and reason about how it should evolve than to implement the equivalent amount of functionality from a blank page.

The difference is particularly obvious in code reviews.

Give me an implementation and I can start asking questions almost immediately:

  • Does this belong here?

  • Does this duplicate something that already exists?

  • Is this abstraction actually necessary?

  • What assumption is this code making?

  • What happens on the unhappy path?

  • Does this fit the architecture of the rest of the system?

  • Is the implementation solving the requirement or merely making the tests pass?

  • Will this decision cause problems somewhere else in the repository — or in another repository entirely?

A codebase is a candidate solution.

A pull request is a candidate modification.

An architecture is a candidate decomposition of a problem.

Having something concrete in front of me gives me structure to reason about.

That does not mean writing code is difficult for me. I can produce implementations that satisfy the same standards I apply when reviewing other people’s work.

But I have always felt that analysis, understanding, review and refactoring come more naturally than generation from scratch.

For a long time, that was simply a personal observation about how I worked.

Then generative AI arrived.

5. AI changes which side of the asymmetry we spend time on

Traditional development looked approximately like this:

requirements
    |
    v
understand the problem
    |
    v
design
    |
    v
implement
    |
    v
test
    |
    v
review

The developer participated heavily in both generation and evaluation.

LLMs change the economics of that workflow.

For many tasks we can now do something closer to:

requirements + context
        |
        v
       AI
        |
        v
candidate implementation
        |
        v
human evaluation
        |
   reject / refine
        |
        v
       AI
        |
        v
better candidate
        |
        v
validation

Generation has become dramatically cheaper.

Instead of spending hours constructing the first candidate, we can often get one in seconds.

Or several candidates.

Or ask the model to modify the candidate repeatedly.

The developer therefore spends proportionally more time on the other side of the asymmetry:

understanding, constraining, evaluating and validating.

This is one form of Human-in-the-Loop (HITL) AI: the automated system produces outputs or takes actions, while human judgment remains part of the workflow for guidance, supervision, correction or approval.

The machine generates.

The human remains inside the loop because the generated answer is not automatically trusted.

6. Verification is not one thing

For software, "verification" is also much richer than checking the answer to a mathematical problem.

There are at least two different levels.

6.1. The product as a black box

First I can ignore the implementation entirely and ask:

Does the product actually do what I wanted?

Does the feature behave correctly?

Does the workflow make sense?

Are the edge cases covered?

Does the resulting system solve the original business problem?

At this level, I am evaluating the behavior of the generated artifact.

6.2. The implementation as a white box

Then I can inspect how the result was produced.

The application may appear to work while the implementation is still wrong.

The generated code may:

  • violate an architectural boundary;

  • duplicate existing functionality;

  • misunderstand a domain invariant;

  • introduce a security problem;

  • perform badly at scale;

  • be unnecessarily complicated;

  • make future changes harder;

  • accidentally satisfy the visible tests while implementing the wrong semantics.

This matters because AI-generated code is still generated code.

Recent software-engineering research treats evaluating LLM-generated software artifacts as a difficult problem in its own right, and recent surveys describe a growing need for reliable evaluation methods as code-generation throughput increases.

Tests help.

Static analysis helps.

Automated quality gates help.

AI reviewers themselves may help.

But eventually somebody or something has to establish that the generated artifact corresponds to the real intent.

And intent is often the hardest part.

7. Why AI can feel surprisingly easy to use

This, I think, explains part of the almost magical feeling of modern AI-assisted development.

It is not simply that the AI "codes faster."

It changes which problem the human is solving.

Instead of continuously asking:

How do I construct this?

I can increasingly ask:

Is this what I wanted?

and:

What is wrong with this version?

The blank page disappears.

A candidate exists.

And once a candidate exists, many of us are operating in a mode in which humans are remarkably effective: comparison, recognition, criticism, selection and refinement.

The same developer who might spend an hour deciding how to structure an implementation may need only a few minutes to look at a proposed structure and say:

No. This is the wrong abstraction. Use the existing mechanism instead.

That is a profound change in leverage.

AI is not eliminating engineering judgment.

It is making engineering judgment more central.

8. The bottleneck moves

There is an obvious consequence.

If generation keeps getting cheaper, generation stops being the limiting factor.

The bottleneck moves elsewhere.

Before something can be generated, somebody must establish what should be generated.

After it has been generated, somebody must establish whether the result is actually correct.

In other words, as implementation becomes cheaper, specification and validation become relatively more expensive.

That is where I think the next major transformation of software development will happen.

But that deserves a separate article.

For now, the observation is simpler:

When the cost of generating candidate solutions collapses, the ability to evaluate them becomes disproportionately valuable.

The football player still has to take the shot.

But if somebody else can generate the shots for us, our job increasingly becomes knowing which ones should count.

<< Previous Post

|

Next Post >>