Skip to main content
Back to insights
AI AgentsCost Optimization
Zaid MarjiNOV 18, 202610 min read

Three Decisions That Determine Whether Your AI Product Works

Share:

Three white diamonds on a horizontal line that ends in an arrow, on a coral background. Above the first are three circles of increasing size, the middle one filled. Above the second are two pages of text, one with a dashed line where a line is missing. Above the third is a padlock.

Which model should handle the task? Is the result what we asked for? Should the agent be allowed to act?

Zaid Marji | Fractional AI CTO, Eventum

I recently ran an experiment that changed how I think about the cost of AI agents.

I gave coding agents four small development tickets and measured what happened under different model-routing strategies.

When every ticket went to the top model, the bill was $9.09.

When a specialized decision model evaluated the tickets and selected the appropriate model tier, the bill fell to $3.47. A decision model returns one answer from a fixed set, with a probability. A language model returns text.

Both approaches passed all 51 hidden acceptance tests.

That is a 62% reduction in measured cost. The tickets were the same. Only the assignment changed.

The decision model was TypeSafe’s Jev. Across the four tickets, its routing judgments cost approximately $0.0005.

The routing experiment. Choosing a model tier for each ticket cut the bill from $9.09 to $3.47. Coding agents, four small development tickets. Both configurations passed all 51 hidden acceptance tests. Two bars drawn to one scale. Every ticket goes to the top model: $9.09, 51 of 51 hidden tests passed. A decision model selects the model tier: $3.47, 62% lower, 51 of 51 hidden tests passed. A small test, not a production benchmark: one run per configuration, and the tests were not difficult enough to distinguish the models’ capabilities. Source: the author’s public model-router experiment.

A small test, not a production benchmark. The tests were not difficult enough to distinguish the models’ capabilities. A separate blind review found a bug in the cheaper implementation that the automated tests missed. There was one run per configuration, and costs for the same model moved about 15% between runs. The path that splits one ticket across several agents has never been run end to end.

But the experiment reinforced a question I’ve been thinking about for some time.

How much expensive generative intelligence are we spending on tasks that only require a small, structured decision?

Not every AI call needs to produce text. Many exist to choose between known alternatives: which queue receives a request, whether a condition was satisfied, whether a result needs review, or whether an agent should proceed.

These decisions can be embedded in software as explicit, testable components.

I see three places to apply that idea.

Three small decisions. Where the work goes, whether the plan matches the specification, and whether the agent may act. 1, before the work is assigned: Which model should do the work? Estimate the task’s reasoning, ambiguity and risk. Ordinary code turns the answers into a model tier. Status: one small experiment, with numbers. 2, before coding begins: Is the agent solving the problem we actually gave it? Compare the agent’s plan, its written intent, with the original specification. Status: being built. Not yet measured. 3, before an action runs: Should the agent actually be allowed to act? Judge whether a proposed step deserves scrutiny. The execution layer decides what is permitted. Status: to explore next. A decision model returns one answer from a fixed set, with a probability.

1. Which model should do the work?

The default architecture for many AI applications is straightforward: select a capable model and send everything to it.

The model answers easy questions, difficult questions, routine classification requests, and complex reasoning problems alike.

This is convenient, but it assumes that every request warrants the same level of computational effort.

It doesn’t.

Classifying a support ticket and debugging a race condition are different jobs. Yet an application using one frontier model for everything pays frontier-model rates for both.

Model routing is an attempt to make that distinction explicit.

The idea is older than my experiment. In 2024, researchers at UC Berkeley introduced RouteLLM, a framework for learning when requests should go to stronger or weaker language models. On their evaluated benchmarks, routing reduced costs substantially while retaining a high proportion of the stronger model’s performance.

My experiment at Loom Labs, my studio, approached the problem from another direction.

I used Jev to evaluate seven characteristics of the task, including reasoning difficulty, ambiguity, risk, scope, and verifiability.

The answers were passed into an explicit routing policy.

Three dimensions primarily determined the model tier: reasoning, ambiguity, and risk. The remaining signals are there to decide whether a task needs delegation, parallel work, or an additional verifier.

That separation is important.

The decision model estimates properties of the task. Ordinary code determines what those properties mean for routing.

When a policy changes, the relevant decision can be modified and tested without relying on a language model to reinterpret the entire strategy.

I also compared this approach against a fresh instance of the top model making each routing decision. That configuration cost $6.37 in its separate run, against the previously measured $9.09 all-top-model baseline.

The routing calls themselves cost $1.69 at full price, or approximately $0.14 when counting only marginal tokens rather than the entire context loaded by those fresh instances.

Jev’s routing calls cost about $0.0005.

The cost of making a decision can be a meaningful part of the total cost of the work.

What the routing decisions cost. With the top model as router, the routing calls were $1.69 of a $6.37 run. With Jev, about $0.0005 of $3.47. The same four tickets, routed two ways. Each bar is one run’s total cost, drawn to one scale. Coral is the cost of the routing calls. A fresh instance of the top model makes each routing decision, $6.37 in total: routing calls $1.69, rest of the run $4.68. Jev, the decision model, makes each routing decision, $3.47 in total: rest of the run $3.47, routing calls: about $0.0005. $1.69 is at full price. Counting only marginal tokens rather than the entire context those fresh instances loaded, it is approximately $0.14. Separate runs, one per configuration. Both passed all 51 hidden tests. Source: the author’s public model-router experiment.

Of course, cheap routing is only valuable if the choices are good.

Sending a task to a smaller model can increase retries, introduce defects, or force an expensive escalation later. Model switching can also disrupt prompt caching and continuity across a long-running agent session.

A useful router therefore needs to be evaluated on the complete task.

The routing decision is cheap. The consequences of a bad routing decision may not be.

That’s the question I’m extending into application workloads, including experiments with open-weight models on GPUs: which requests need the strongest model, and how do we identify them before paying for one?

2. Is the agent solving the problem we actually gave it?

There is a second decision that may be even more consequential than model selection.

Before asking whether an AI-generated change is correct, consider whether it is implementing the correct requirement.

Agents can produce functioning code for the wrong interpretation of a specification.

They can satisfy tests while overlooking an exception, misunderstanding a condition, or narrowing the scope of the request.

Sometimes the failure happens before the first line of code is written.

In the development harness I’m building, a specification is converted into an intent file, a written representation of what the implementation is supposed to accomplish. The agents then work against that intent.

This creates an obvious point for an independent check.

Does the intent actually preserve what the original specification required?

Consider a hypothetical refund workflow.

The specification says that a refund may be issued only after the requester’s identity is verified and the transaction is confirmed eligible.

An agent converts that into an implementation plan focused on eligibility checks, but omits the identity-verification requirement.

It can then write perfectly functional code and pass tests derived from that incomplete plan.

The system has checked that the implementation follows its plan. Whether the plan faithfully represents the requirement is still unchecked.

A narrow decision model could compare the intent against the source specification and flag possible omissions or contradictions before coding begins.

The key word is narrow.

Asking a model “Is this software ready to ship?” combines many separate questions into one vague judgment. It asks the model to assess requirements, implementation, test quality, security, and perhaps a dozen other concerns simultaneously.

Asking “Does this intent preserve the identity-verification requirement?” creates something much easier to verify.

A hypothetical refund workflow. The code follows the plan. Whether the plan follows the specification is still unchecked. The specification: a refund may be issued only after the requester’s identity is verified and the transaction is confirmed eligible. An agent converts it into a plan. The agent’s plan, its written intent: eligibility checks, with identity verification omitted. It writes code and tests from that plan. The implementation: functional code that passes the tests derived from the incomplete plan. Checked: the implementation follows its plan. Still unchecked: does the plan faithfully represent the requirement? A narrow question for a decision model, before coding begins: Does this intent preserve the identity-verification requirement? A hypothetical example. The intent check is being added to the author’s development harness. It is not yet a measured production result.

That is the check I’m adding to my harness. It is not yet a measured production result.

To evaluate it, the next step is to assemble examples of faithful and drifted intent files, establish reliable labels, and measure whether the model distinguishes them. Its threshold should reflect the cost of missing a material requirement versus unnecessarily interrupting development.

Even an accurate intent checker would not replace tests, code review, or security controls. It addresses one specific failure mode earlier in the process.

But catching an incorrect interpretation before an agent builds around it can be much cheaper than discovering the same problem after implementation.

The distinction I care about is simple:

Did the agent execute the plan correctly? And was the plan itself correct?

Those are different questions. A reliable development process needs evidence for both.

3. Should the agent actually be allowed to act?

The third decision is where AI stops being a source of recommendations and starts changing the world.

An agent can draft an email. It can also send the email.

It can propose a database migration. It can also execute one.

It can identify an account that appears suspicious. It can also freeze that account.

The transition from producing an answer to taking an action changes the risk.

A mistaken recommendation can be reconsidered. A payment, deleted record, leaked credential, or incorrectly changed permission may be difficult or impossible to undo.

The response cannot simply be to make the model more confident.

The system needs to distinguish between actions it can perform automatically, actions requiring confirmation, and actions it should not be permitted to execute at all.

At Loom Labs, I built AgentWatch, an open-source tool that monitors AI coding agents for operational and security warning patterns.

It watches for behavior associated with loops, repeated tool failures, context degradation, suspicious credential handling, and other signs that an agent may be heading in the wrong direction.

The existing system uses monitoring and detection logic. What I want to explore next is adding narrowly defined model judgments about an agent’s behavior as the run progresses.

For example, a monitor could ask whether the agent is repeating essentially the same unsuccessful action, whether a proposed tool call is inconsistent with the stated task, or whether the current sequence resembles a known failure pattern.

A probability could help determine when to alert an operator, interrupt a run, or request additional inspection.

Detecting a risky action is one job. Authorizing it is another.

A decision model is not an authorization system.

Even a perfectly formatted answer with a high confidence score cannot establish that a user has permission to move funds, expose information, or alter a production environment.

These controls belong in the execution layer. My rule: every agent action is logged, and nothing irreversible runs without a person signing it. Approvals, payments, anything that changes state.

That distinction is consistent with OWASP’s guidance on AI agent security, which calls for least privilege per tool, human-in-the-loop controls on consequential actions, and monitoring.

The decision model can help identify when something deserves scrutiny. It should not be able to overrule the controls that determine what the agent is allowed to do.

Judgment and authority. Detecting a risky action is one job. Authorizing it is another. The decision model helps identify what deserves scrutiny: Is the agent repeating essentially the same unsuccessful action? Is a proposed tool call inconsistent with the stated task? Does the current sequence resemble a known failure pattern? A probability helps determine when to alert an operator, interrupt a run or request inspection. The execution layer determines what the agent is allowed to do: an action runs automatically, requires confirmation, or is not permitted to execute at all. The decision model cannot overrule these controls. Every agent action is logged, and nothing irreversible runs without a person signing it. Approvals, payments, anything that changes state. The rule is the author’s. The model judgments on the left are what he wants to explore next.

This is also why it matters that specialized decision models return structured judgments.

A model that selects from a fixed answer set has a narrower output interface than one that generates arbitrary commands. That can make its integration easier to constrain.

It does not make its judgments immune to misleading inputs, adversarial instructions, or distribution shifts.

The application still owns the boundary between judgment and action.

What turns a promising pattern into production infrastructure?

There is a common thread across all three decisions.

Each begins with a question that the application already answers somehow.

Which model handles this request?

Does this plan preserve the requirement?

Should this proposed action proceed, pause, or escalate?

Today, those choices may be embedded inside a generative prompt, hard-coded as a crude rule, or left to an agent to interpret.

Making them explicit creates an opportunity to evaluate them independently.

But returning a probability is not the same as demonstrating reliability.

For model routing, the evaluation needs to include end-to-end task quality, retries, escalation costs, and latency. A router that saves 50% on inference but doubles the downstream failure rate may be a poor trade.

For intent checking, the important measurement is whether material mismatches are detected without creating an unmanageable number of false alarms.

For agent monitoring, the evaluation needs to establish whether alerts correspond to actual problems and whether intervention happens early enough to matter.

In every case, the most useful operating measurements include the share of decisions handled automatically and the observed error rate within that automated group. The full cost of fallback and review also matters.

A hundred carefully reviewed historical examples can be a useful starting point for discovering failure modes and designing an evaluation. They are not enough to establish very low production error rates.

The path forward is to run these decision checks alongside existing workflows, measure their behavior on representative traffic, and gradually assign responsibility only where the evidence supports it.

This is particularly important when several checks are composed. A model router, an intent checker, and an action monitor can each perform well individually while the overall system still fails because their errors interact.

Reliability is a property of the workflow.

The opportunity is bigger than saving tokens

I launched Moody’s Analytics’ first ML product. It reads borrowers’ financial statements for lenders.

Its models produced confidence scores too. We still had to decide what those scores meant operationally, how to evaluate them, and when human judgment was required.

The core discipline hasn’t changed.

What has changed is the economics and the range of places where model-based decisions can now be introduced.

A specialized model can evaluate many bounded questions cheaply enough to sit throughout an application, sometimes before an expensive model call, sometimes after one, and sometimes immediately before an agent takes an action.

Generative models remain essential for tasks that require creating, interpreting, investigating, or reasoning across complex information. Deterministic software remains preferable wherever the rules are exact and can be applied reliably.

The opportunity is to stop treating every application-level judgment as another request for generated text.

My routing experiment is one example with numbers. The other two checks will get theirs before I rely on them.

All three point toward the same architecture: capable models performing the substantive work, surrounded by smaller, explicit decisions that determine where that work goes, how it is checked, and what the system is allowed to do with the result.

If you want a second pair of eyes on where the small decisions sit in your stack, get in touch.

Disclosure: Eventum is an official technology partner of TypeSafe, and TypeSafe CTO Erik Gafni is an advisor to Eventum. The routing measurements are from my own public experiment, not an independent production benchmark. The intent checker and model-based agent monitor are described as work in progress.

Summarize with AI:

ChatGPTGrokGeminiClaude
Related service

Discovery, architecture, build, evaluation, deployment, handoff. Senior technical ownership end-to-end.

Diagonal halftone representing the AI project delivery flow.