The Most Important AI Design Decision Is Where The Model Stops

—Tom Richards is CTO and co-founder of MoneyMeans, the company behind Aida—

He argues that model performance is only part of the design problem. In regulated sectors, companies must decide what a language model is allowed to do and which outputs it must never produce.

What we are seeing is most companies building with AI are asking the wrong question, they ask how good their model is, but the more important question is where it should stop?

I spent much of the past year working on that question while building Aida. We have now published a white paper explaining how we answered it.

Every company putting a language model in front of customers should ask the same question, particularly in regulated fields. Most do not. That omission is itself a design decision and usually a bad one.

A Good Answer To The Wrong Question

Early on, someone asked Aida whether they should overpay their mortgage. The answer covered the interest they could save, early-repayment charges and the value they placed on owning their home sooner. It was clear and technically defensible. It was also the wrong answer.

An adviser hearing that question would widen it before answering. Is there an employer pension contribution going unclaimed? Does the customer have an adequate emergency fund? What is their marginal tax rate?

Overpaying a mortgage and contributing to a pension compete for the same money over roughly the same horizon, and for many people the pension wins. In our case, the deciding detail was an employer contribution that was not present in the data Aida could access.

The model had reasoned well about the question it was given. The system had allowed it to define that question too narrowly.

Better Models Do Not Close This Gap

The common reaction to a result like this is to wait for the next model. It will be smarter and more careful, so the problem will fix itself. However, in our internal testing, we put the same question to more capable models. They still produced polished answers about mortgage overpayments without first examining the customer’s wider circumstances.

A model answers the question in front of it. An adviser works out which question matters. That second task is an architecture problem, and a bigger model will not solve it.

I think this is where many AI products go wrong. Teams treat the model as the product and everything around it as scaffolding. In fields where an answer can affect someone’s finances, health or legal position, the scaffolding is part of the product.

Deciding What The Model Is Allowed To Do

In Aida, the model does what an adviser would say out loud, the language model manages the conversation. It interprets the customer’s question, , draws out what we need to know, and explains the resulting guidance. The calculations, prioritisation and actions shown on screen are deterministic. They are calculated rather than generated, and the arithmetic is retained for review.

There is a deliberate gap in our architecture where the model could have generated the customer’s next action. That boundary was the most important design decision we made.

We also evaluate every message Aida sends. Most of that evaluation runs alongside the response and does not delay its delivery and costs the user nothing: tone, helpfulness and personalisation are logged and improved in the next iteration. Anything touching the advice boundary is treated like a security issue rather than a bug. It is judged before delivery, and it gets rewritten or blocked. The user waits a few extra seconds, which is a fair price. An improvement can wait for the next release, but once a crossed boundary reaches someone it can’t be undone.

None of this takes people out of the loop; it decides where they sit in it. The real-time checks are automated, because no human team could review every message in the seconds before it is delivered. But every verdict those judges reach is logged with the conversation and summarised into dashboards our team reviews daily and weekly.

Anything that touches the advice boundary is escalated the way a security incident would be: a person looks at what was said, why it was caught and whether a rule, a prompt or the tool flow needs to change. And because we operate as an Appointed Representative of an FCA-authorised firm, the boundary itself is subject to regulated oversight, not just our own judgement.

That is the real role of human insight in a system like this. Models enforce the line on every message, but people drew it, and people decide when it moves.

These controls can add a few seconds to the response time. That is a fair price. A product improvement can wait for the next release. A customer may already have acted by the time an unsafe answer is corrected.

The Question Companies Need To Answer

If you’re building with AI, and particularly in finance, health, law or anywhere else people act on what you tell them, “how good is our model?” will always have a comforting answer, because models keep improving.

“Where does our model stop?” is harder to answer. It makes you decide which parts of your product must be probabilistic, which must never be, and which failures you schedule versus which you block. It makes you design the boundary rather than find it in a complaint or worse, in an outcome someone can’t reverse.

We need both kinds of component, the deterministic and the probabilistic, and neither is enough alone. The companies that earn trust with AI won’t be the ones with the cleverest models. They’ll be the ones who decided early and deliberately, where the model’s job ends.

For us, that wasn’t a limitation we accepted. It was the part of the system we designed first.

—Tom Richards is CTO and co-founder of MoneyMeans, the company behind Aida.—