I received a comment on my recent entry, "AI, the New UI, Not the New API".

The comment reads:

We are already at the point where supermarkets is putting the responsibility for their AI replacing products back on the customer who ordered something... Now multiply that across all domains... And then imagine the cost!

This use-case highlights a real-world symptom of a shortcut, an architectural problem: What happens when you let a probabilistic agent execute concrete system changes without strict constraints?

Tacit Knowledge

In an earlier post, I called humans "entropy reducers." Just as AI is a probabilistic reducer, humans are too. The difference is that humans, knowingly or unknowingly, do not expose a lot of the context they possess. AI simply does not have access to that context to make similar decisions. This touches on a highly debated topic in AI research: tacit knowledge and the hidden state.

The Power of the Hidden State

In the grocery substitution use-case, when a human makes a decision, they are carrying their learned experience and applying it to yield an outcome. A human shopper knows: "The shop is out of whole milk. I will pick toned milk, but never soy milk because my partner hates the taste."

This preference is a hidden state. The human has access to it, but the grocery list lacks it. The AI making the substitution in the supermarket app only has the explicit context window it was given (the user's cart, the store inventory, and a generic similarity matching algorithm). It fails because it lacks the lifetime of multimodal training data the human is carrying around. The AI will catch up eventually once the preference is registered; until then, the user is getting soy milk.

The Context Bottleneck

This creates a massive bottleneck in system design. We want AI to handle ambiguous, probabilistic tasks, but we do not have a frictionless way to transfer our vast human context into the machine's context window yet.

When humans unknowingly fail to expose this context, the AI falls back on generic probabilities. It doesn't make a "bad" decision; it makes the most mathematically likely decision based on a severely restricted dataset.

Humans: The Ultimate Entropy Reducers, Really?

Reflecting on my own entries, it dawns on me that humans are actually more like Context Anchors. We ground our probabilistic decisions in a strict, personalized reality. When an AI hallucinates a bad grocery substitution, the system relies on the human to step in and apply their hidden context to reject the error, thereby collapsing the probability wave into a definitive, deterministic outcome.

But Then There is Context Bloat

Can we fix the user preference problem? Yes, certainly. We can introduce corrective micro-prompts to nudge the AI's decision-making. The problem is that corrective prompts are post facto, and in a stateless conversation, they are absolutely useless. Furthermore, appending them all just makes the context bloated.

Enter RAG (Retrieval-Augmented Generation)

You don't want to overwhelm the AI with context bloat. The simple alternative seems to be: just train it again with the additional context. It will work as expected. But at what cost? Retraining an AI is a notoriously expensive affair, a FinOps nightmare.

This is where RAG helps. Instead of retraining, your frozen model queries a database of user preferences and retrieves only the meaningful bits of context, injecting them just-in-time for the AI to make a better decision.

Tying it Together

In fact, we do not even need an AI example to tie everything together.

The Problem: Mental Model Mismatch

The classic Up/Down buttons in a lobby suffer from a classic UX failure: they require the user to translate their goal into a rigid system command, which often leads to the "wrong instruction". Are those buttons for registering an intent, or are those call buttons for commanding the elevator to your floor? There are no hints, and the UX simply fails to convey the point.

When a user is on Floor 3 and wants to go to Floor 10, they are supposed to press UP (their intended direction of travel). However, if they see the elevator is currently on Floor 15, many users will press DOWN, thinking they need to "call the elevator down to me."

The system's API expects the user's desired trajectory, but the user is trying to issue a mechanical command to the machine. It is a low-context interaction that creates errors.

The Solution: Destination Dispatch

With advancements in technology and enough resources, it is now possible to add individual buttons for all the floors directly in the lobby. The UX simplifies to "Which floor?" No more guessing what the buttons mean.

This solution highlights the fact that UX still matters. Capturing a sanitized intent upfront is exactly what allows an AI to make better, safer decisions.