Every few weeks, a new product announcement crosses my desk about an AI system making decisions in a regulated workflow. A claims processing platform that evaluates medical necessity. An underwriting engine that decides loan approvals. A benefits screener that determines eligibility. A compliance monitor that flags regulatory violations. A logistics dispatcher that routes shipments autonomously.

Many of these products are built well. Some are built carelessly. Almost none of them talk openly about the single most important architectural question in this category: what happens when the system produces a different answer on the same input the next time it runs.

This silence is not because the issue does not exist. It exists in every system built on large language models as the decision-making component. The silence is because naming the issue forces architectural choices that most teams have not made and do not want to make. It is easier to ship a probabilistic system with a disclaimer than to build the deterministic evaluation layer that would make the system actually defensible.

This post is an argument that the deterministic evaluation layer is not optional for consequential decisions. It is an argument that the current industry conversation about AI reliability, auditability, and responsible deployment is missing a specific architectural concept, and that the absence of that concept is producing a lot of systems that will not survive the scrutiny they are about to receive from regulators, auditors, and the people on the receiving end of their decisions.

I have a commercial interest in this argument. My company builds deterministic evaluation infrastructure. I am not a neutral observer. But the argument is worth making independent of any product, because the pattern is what matters, not the vendor, and the pattern is becoming urgent.

The specific reliability gap

Language models are probabilistic by design. When an LLM produces an output, it samples from a distribution over possible tokens, and the output reflects whatever sequence of samples happened to occur during that particular forward pass. Set the temperature to zero and the distribution collapses toward its mode, which reduces variation but does not eliminate it across model versions, inference contexts, or hardware implementations.

This is not a bug. It is a property of how the models work. Probabilistic reasoning is why LLMs can handle unstructured natural language so well. Forcing them to be deterministic would defeat the purpose of using them for the interpretation work they are genuinely good at.

The problem is not that LLMs are probabilistic. The problem is that a probabilistic component is being used for the decision itself rather than for the interpretation of the input. The two functions have fundamentally different reliability requirements, and conflating them produces systems that fail in specific, predictable ways.

What consequential decisions actually require

A regulated decision has four reliability properties that a non-regulated decision does not:

• Reproducibility. Running the same input through the system tomorrow should produce the same output. This is not a nice-to-have; it is often a legal requirement, and it is always an operational necessity for audit, appeal, and dispute resolution.
• Explainability at the individual decision level. Not explainability in the machine-learning research sense of post-hoc feature attribution, but explainability in the sense that a specific person asking “why did I get this answer” can be given an answer that references specific rules and specific evidence.
• Consistency across cases. Two applicants, patients, or claimants with materially identical situations should get materially identical decisions. Not similar decisions. Identical decisions. This is what it means for a rule-based system to be fair in any meaningful sense.
• Versionability. The rules that govern the decision must be versioned, and a historical decision must be reproducible against the specific version of the rules that was in effect at the time. Without this, every rule update invalidates the audit trail of every prior decision.

Probabilistic systems cannot provide any of these properties. Not because the engineering is hard. Because the computation model is incompatible.

A probabilistic system does not produce the same output on the same input. It produces a sample from a distribution over possible outputs. The distribution may be narrow and the samples may be consistent in practice, but consistent in practice is not the same as consistent as a matter of architecture. An auditor cannot verify that a system is consistent by observing a single run. A regulator cannot accept “it usually works” as a compliance statement. A person whose claim was denied cannot be told that the system probably would have denied them again under identical conditions.

The response from the AI industry to this reality has been, broadly, to pretend it is not there. Disclaimers proliferate. Terms of service are updated. Explainability tooling gets layered on top of probabilistic outputs and called auditable. Human review is appended to the end of automated decisions and called oversight. These are all workarounds. They treat a structural architectural problem as a messaging problem, and they are going to produce a wave of regulatory action, lawsuits, and consumer harm as the systems they paper over enter more and more consequential workflows.

Why existing approaches do not work

Before arguing for the deterministic evaluation layer, I want to be specific about why the currently popular alternatives are insufficient. Each of them is built on an assumption about reliability that does not survive examination.

Explainable AI tooling (SHAP, attention maps, counterfactuals)

These techniques attempt to explain what a model did after the fact. They produce visualizations of feature importance, attention weights, or counterfactual comparisons that help a human understand which parts of the input drove the output.

These are useful research tools. They do not solve the decision reliability problem. A SHAP value explains the model's behavior on a specific input but does not guarantee the model will behave identically on the same input next time. An attention visualization shows what the model looked at, not what it would look at if rerun. A counterfactual explanation describes how the output would change with different inputs, but says nothing about whether the output is consistent with the same input.

More fundamentally, explainable AI operates at the wrong level. It explains the model. The thing that needs to be explained is the decision, which is a different object. In a deterministic system, a decision is explained by reference to the specific rule that produced it and the specific evidence the rule consumed. No post-hoc tooling is required, because the explanation is built into the computation. In a probabilistic system, the decision is explained by probabilistic reasoning about the model's behavior, which is a strictly weaker claim.

Guardrails

Guardrail systems filter LLM outputs against safety criteria before they reach the user. They check for prohibited content, policy violations, prompt injection, and similar patterns. Guardrail frameworks (Guardrails AI, NVIDIA NeMo Guardrails, various in-house equivalents) have become standard infrastructure in production LLM deployments.

Guardrails do what they are designed to do. They do not make the underlying decision deterministic. A guardrail-protected LLM is still producing probabilistic outputs; the guardrail is just deciding whether to let the output through. Two runs on the same input produce different outputs, both of which may pass the guardrail checks. The decision itself remains inconsistent across runs, regardless of how many layers of filtering surround it.

Guardrails are an essential tool in the overall safety architecture of LLM deployments, but they are not the tool that provides decision reliability. Using them as if they were is a category error that becomes visible the first time an auditor asks why the same case produced different outcomes on different days.

Constitutional AI and RLHF

Constitutional AI and reinforcement learning from human feedback are techniques for training models to behave more consistently with specified principles or preferences. Models trained this way produce outputs that track their training more reliably than untrained or naively fine-tuned models.

This is a genuine improvement in model quality. It is not a replacement for determinism in the decision path. A constitutionally-trained model produces outputs that are more consistent, but they are still sampled from a distribution. The distribution has shifted, but it has not collapsed. Two runs on the same input can still produce different decisions, even if the distribution of those decisions has become tighter than in a non-trained model.

Constitutional AI is orthogonal to the architectural question of where to draw the probabilistic-to-deterministic boundary in a decision system. It makes the probabilistic component more reliable within its probabilistic regime. It does not change the fundamental property that probabilistic systems cannot provide determinism.

Human in the loop

The most common pattern in production AI systems for consequential decisions is to have a human review the AI's output before it takes effect. The AI produces a recommendation; a human clicks approve or reject. This is called human in the loop, and it is positioned as the solution to AI reliability concerns.

Human review at the end of a pipeline has two failure modes that are now well documented in the academic literature and visible in every real deployment. The first is automation bias: humans who are presented with an AI recommendation are strongly inclined to accept it, especially when they are processing many cases under time pressure. The second is the inability to efficiently re-run the decision with modified inputs; the human can reject the recommendation but cannot easily explore what would have happened with a slightly different framing, which means their judgment is largely constrained to accept-or-reject rather than active decision-making.

Human review at the end of a probabilistic pipeline is better than no review, but it is not a substitute for deterministic evaluation. It is review of a decision that was already made, not construction of a decision under human guidance. And because the underlying pipeline remains probabilistic, the same review run tomorrow on the same input may be reviewing a different recommendation. The human cannot verify consistency by reviewing individual cases, because consistency is a property of the system, not of any single case.

The deterministic evaluation layer

The alternative to each of the above approaches is architectural, not technical. It is a deliberate separation of the decision pipeline into two stages with different reliability requirements, connected by a structured handoff.

In the first stage, probabilistic reasoning is used for what it is good at: reading unstructured natural language and producing a structured intermediate representation. An LLM reads a clinical note, an insurance application, a shipment event log, or a contract clause, and generates a typed data structure that captures the relevant fields. Confidence scores are attached. Low-confidence extractions are flagged. Ambiguity is preserved rather than suppressed.

Between the two stages, a human operator confirms the structured intermediate representation. This is not human review of a finished decision. It is human confirmation of the interpretation that the decision will be based on. Ambiguities are resolved. Uncertainties are corrected. The operator's role is to ensure that the downstream deterministic stage is working from a correct understanding of the case.

In the second stage, a deterministic engine takes the confirmed intermediate representation and evaluates it against encoded rules. Same input produces same output, byte-identical, every run. The engine implements Boolean logic, threshold comparisons, temporal reasoning, and ontology matching, but all of this is pure computation with no probabilistic components in the decision path.

This architecture, which I have written about as the bridge pattern in a separate post, makes each reliability property tractable. Reproducibility follows directly from the determinism of the decision stage. Explainability at the individual level is built into every output, because every rule evaluation can be traced back to the specific rule applied and the specific field consulted. Consistency across cases is provable by inspection of the rules. Versionability follows from treating rule sets as versioned data, independent of the engine's code.

None of these properties require advances in machine learning. None of them require new mathematical techniques. They require an architectural choice to stop using probabilistic systems for decisions and instead use probabilistic systems for interpretation, with a clear handoff to a deterministic evaluation layer that makes the actual call.

Why this is not happening yet

The architecture I am describing is not novel. Rule engines have existed for decades. Drools, Jess, CLIPS, and dozens of commercial business rules management systems have implemented deterministic evaluation over structured inputs for years. What is new is not the idea of a deterministic decision engine. What is new is the ability to bridge unstructured natural language into structured input without requiring an army of humans to type the structured input by hand.

So why is the AI industry not converging on this pattern? A few reasons.

The demo incentive

LLM-only pipelines produce impressive demos. The AI reads the document and makes the decision, all in one apparent step. The demo tells a clean story: the AI is smart, the human is replaced, the system is magical. A bridge pattern demo is slightly less sexy. The AI reads the document, then a human confirms the extraction, then a separate engine applies rules. There is a pause in the middle. The human looks slow compared to the machine.

The demo incentive pushes teams toward architectures that look better in two-minute pitches and perform worse in production. This is a well-known pattern in software, but it is unusually acute in AI right now because the demo-to-production gap is larger than it has been in any prior generation of technology.

The engineering unfamiliarity

Most AI engineering talent today has come up on probabilistic systems. Model training, prompt engineering, RAG, agentic workflows. The concepts and tools in their professional vocabulary are all probabilistic. Deterministic rule engines, typed data structures, versioned rule packs, and the discipline of keeping logic out of machine learning code belong to an older engineering tradition that has fallen out of fashion even where it is needed.

This is a cultural gap, not a technical one. The information required to build deterministic decision layers has been documented for decades. What is missing is the instinct to reach for it when a new AI system is being designed. A team building a prior authorization system today is more likely to start with an LLM-and-a-prompt and add features than to start with a rule engine and add LLM interpretation. The second path produces a more reliable system. The first path is where the talent defaults.

The venture incentive

AI systems that ship with real deterministic evaluation layers are harder to build and harder to sell as pure-AI products. They require encoded domain expertise, which is expensive to produce. They require human operator workflows, which reduce the automation narrative. They require architectural discipline that is hard to demonstrate in a pitch deck.

The companies getting funded right now are the ones that tell clean AI stories. LLM in, decision out, all automated, all magical. The companies building reliable decision infrastructure are, broadly, having harder conversations with investors because the product story is structurally more complex. This selection pressure pushes the industry toward architectures that generate fundable demos and away from architectures that generate defensible production systems.

I think this pressure will reverse as the first wave of AI-in-consequential-decisions products hits the regulatory and legal reality they are not prepared for, but the reversal will take time. Between now and then, a significant amount of AI deployment in consequential workflows is going to happen on architectures that cannot be defended, and people on the receiving end of those decisions will bear the cost.

What regulators are likely to require

A year from now, two years from now, three years from now, the regulatory conversation around AI in consequential decisions is going to converge on a few specific requirements that are already visible in nascent form in the EU AI Act, the NIST AI Risk Management Framework, and state-level AI legislation in California, Colorado, New York, and elsewhere.

These requirements, when they land clearly, will look like this:

First, any AI system that materially affects a consequential decision about an individual must produce a record of the decision that can be reconstructed from the system's inputs and state at the time of the decision. This is a reproducibility requirement. Systems that cannot satisfy it will not be permitted to operate in regulated domains.

Second, any individual affected by an AI-driven decision must be able to obtain an explanation of the decision that references the specific factors that produced the outcome. This is an individual explainability requirement. It is stricter than machine-learning explainability and cannot be satisfied with SHAP values.

Third, any AI system that makes decisions in regulated domains must be able to demonstrate consistency across similar cases, which means that two individuals in materially identical situations receive materially identical decisions. This is a consistency requirement. It is already implicit in existing anti-discrimination law and will become explicit in AI-specific regulation.

Fourth, any AI system operating under a versioned policy or rule set must be able to reproduce decisions made under prior versions of those policies, for purposes of audit, appeal, and retrospective compliance review. This is a versioning requirement. It maps directly onto how rule packs are handled in deterministic evaluation systems and does not map at all onto how model versions are handled in probabilistic systems.

Meeting any of these requirements requires, at the architectural level, a deterministic evaluation layer. Meeting all of them requires the bridge pattern or something very close to it. Systems that attempt to satisfy these requirements with probabilistic decision components and layered explainability tooling will fail the first serious regulatory audit they face, and the companies that built them will have to rebuild from the ground up.

The strategic implication

For any team building AI into consequential workflows right now, the strategic question is whether to design for the current permissive regulatory environment or the future stricter one. The permissive environment allows probabilistic systems with disclaimers. The stricter environment will require deterministic evaluation layers.

Teams that build deterministic evaluation layers now will have an architecture that survives the regulatory transition. Teams that skip the deterministic layer to move faster will reach a point where their architecture no longer complies, and they will have to rebuild. Rebuilding a deployed production system is expensive. Rebuilding one with real users who are already dependent on its outputs is much more expensive.

The cost of building the deterministic layer up front is high but known. The cost of not building it is low now and potentially catastrophic later. A team that bets on the current regulatory environment lasting is making a bet against a trend that is visible across every major jurisdiction.

For platform companies in the AI infrastructure space (Anthropic, OpenAI, Oracle, Google, AWS, and others), the question is whether to provide deterministic evaluation capability as part of their platform offering. Currently, most of these platforms focus on making the probabilistic component more reliable, more steerable, more aligned. None of them offer a deterministic evaluation layer as a first-class component of their platform. This gap is filled by third parties like us, which is a perfectly reasonable division of labor, but it means that platform companies are implicitly sending the message that the deterministic layer is someone else's problem. That is true for now. It will not remain true.

My prediction is that within three years, every serious AI platform will ship with, or require its customers to ship with, a deterministic evaluation layer for any decision surface that touches a regulated domain. The platforms that build this capability natively will have an easier time serving enterprise and regulated customers. The platforms that treat it as optional will face customer churn as regulators force the issue.

What should happen

The AI industry conversation is currently dominated by debates about model capability, alignment, safety, and governance. These are important debates, but they are happening at the wrong level of abstraction for the reliability problem that actual production deployments are facing.

The conversation that needs to happen, and has not yet, is architectural. Where, in a consequential-decision AI system, should the probabilistic components stop and the deterministic components begin? What does the handoff between them look like? What does a well-designed structured intermediate representation contain? How are deterministic rule sets encoded, versioned, and maintained? How do these architectures behave under audit, appeal, and regulatory review?

These are engineering questions with engineering answers. They are not philosophical questions about the nature of AI. They are the kinds of questions that get answered through case studies, reference architectures, open source implementations, and accumulated practice. The industry has not yet accumulated that practice at scale because the demo-driven and venture-driven incentives have pushed in a different direction. But the practice is accumulating, slowly, in the teams that are building real systems for real regulated workflows. It will become mainstream. The only question is how much damage happens between now and then to the people on the receiving end of probabilistic decisions that cannot be defended.

For anyone building in this space, the practical advice is specific. Identify the decisions your system makes. For each one, ask whether a wrong answer would require the system to defend itself to an auditor, a regulator, or a person affected. For every decision where the answer is yes, the probabilistic component should be upstream of the decision, not the decision itself. A structured intermediate representation should mediate the handoff. A deterministic engine should produce the actual decision. Human operators should confirm the interpretation at the handoff boundary. An audit trail should capture the rule trace for every decision made.

This architecture is not more expensive to build than the alternatives. It is differently expensive. It pays most of its cost up front in the careful design of the intermediate representation and the encoding of the rule set, and it pays almost nothing in ongoing operational cost or regulatory risk. Probabilistic-only architectures pay little up front and pay increasingly large amounts over time as reality catches up. The question is just when the cost comes due.

Closing

The AI industry is building a lot of systems that make decisions it cannot defend. Some of these systems will be fine. Some of them will cause real harm to real people before they are either rebuilt or shut down. The harm will be concentrated in the populations least equipped to challenge decisions they cannot understand.

The architectural response to this situation is not mysterious. It is the deterministic evaluation layer. It has been available for decades; the specific innovation of the current moment is the ability to bridge unstructured natural language input into structured form without human labor, which makes the deterministic layer practical at scale for the first time.

Teams that understand this are building differently than teams that do not. The systems they build will be more reliable, more auditable, more defensible, and more resilient to the regulatory transition that is already under way. The sooner the architectural pattern becomes common knowledge, the fewer bad systems get built in the intervening years.

For any engineer, product leader, or investor thinking about where AI is headed in consequential decisions, the question to sit with is not whether models will get better. They will. The question is whether the architecture surrounding the model is prepared to hold the decisions it is now being asked to make. For most current deployments, the answer is no. It should become yes. That is the work.