For nearly as long as the web has existed, web development has wrestled mightily with the right way to connect components over the network. This is the question of the remote API. It influences every aspect of the software we build. We sort of arrived at a tolerable compromise with JSON APIs. While these have their limitations, you have to appreciate their underlying simplicity.

But the advent of AI-enabled endpoints that can mediate intent is changing the basic workings of the internet. This change is gradually reawakening an old dream, the service-oriented architecture (SOA). This time around, with luck, we’ll finally gain the flexible, discoverable, and maintainable automated service discovery we’ve longed for. Fingers crossed.

Why old-school SOA failed

Let’s call this burgeoning influence of AI on web architecture SOA 2.0.

To understand why SOA 2.0 is different from SOA 1.0, we have to remember the trauma of the 2000s. (This may be painful but also cathartic.) The original dream of SOA was beautiful: a world where disparate business services—inventory, billing, shipping, you name it—could automatically discover each other, understand capabilities, and orchestrate complex tasks without human intervention.

To achieve this, we built a monument to complexity. We had SOAP (Simple Object Access Protocol) for messaging, WSDL (Web Services Description Language) to define contracts, and UDDI registries for service discovery. At the center of it all sat the Enterprise Service Bus (ESB), a massive piece of middleware that was supposed to route everything gracefully, seamlessly. In case you young’uns are confused, that is all based on XML.

By the time you were done understanding the infrastructure well enough to know how to do something, you had forgotten what you set out to do.

It failed. It was egregiously heavy. Just to do some simple thing like create a “New Item” endpoint, you immediately had to begin scaling a wall of rigid definitions.

Because computers historically required absolute, deterministic perfection, if a single XML tag in a SOAP envelope was missing, or if a service updated its WSDL without every client re-generating its stubs, the entire multi-million-dollar pipeline would violently unravel. Some of us may be familiar with a similar challenge in containerized microservices (like Kubernates), where trying to determine where in the mesh a problem originated is… awkward.

Classic SOA was a house of cards, too brittle to survive the fuzzy reality of the internet.

The typical JSON API of today is a reaction against SOA. (It may be an overreaction.) We abandoned SOA for the relative simplicity of REST, giving up on the dream of autonomous service orchestration in exchange for manual integrations that just work.

The new intention-to-execution middleware

A sea change is already happening with app-level architecture. 

The effect of AI endpoints in an app’s service profile goes beyond just a new capability. It changes how the rest of the services work together. The overall effect is something like the app gaining an understanding of itself, and what it can do. This is not dissimilar to what WSDL was supposed to accomplish. But instead of a hard-coded descriptor, where some person had to keep what was available and what was described in sync, you now have a layer that can accept dynamically produced descriptors and unite them with fuzzy user intention and produce meaningful action.

You tie in AI endpoints to bridge between what the user is trying to accomplish, with the various strict capabilities available. These capabilities may exist within the app at the back end, at the front end, or at another service layer. The main thing is that there is a flexible AI layer that mitigates the need to hard-code the links between services.

In classic SOA, the contract was a rigid, unforgiving WSDL document. In modern common practice, the contract is a strongly coupled RESTful endpoint. In SOA 2.0, the contract has a hitherto unknown degree of flexibility, thanks to the natural language capabilities of an LLM.

When a user or a system expresses an intent—say, “Provision a new staging environment for the billing service”—the AI middleware doesn’t look for a hard-coded, point-to-point integration. Instead, it digests the intent and performs semantic routing, consulting a registry and selecting the appropriate tools. That registry, rather than a heavy UDDI, might be a vector database of available internal API endpoints, or a collection of available functions.

Modern LLMs equipped with function-calling capabilities act as the ultimate dynamic orchestrators. They read the JSON schema of a target REST API, understand its parameters, and dynamically map the user’s fuzzy, unstructured intent into a perfectly formatted JSON payload. If a field is missing, the LLM can either infer it from context or pause execution to ask the user for clarification.

The brittleness of SOA 1.0 is replaced by a shock absorber. If the target API changes a parameter name from customerID to clientId, the AI middleware can read the updated schema and adjust its mapping on the fly. No client code needs to be recompiled. No stubs need to be regenerated. The multi-million-dollar pipeline survives.

When software becomes smart

These are not just abstract ideas. I recently did my taxes, using a popular mainstream service that I will not name. I had several unusual and grumpy areas to deal with, including the new crypto regulations. It was not pretty.

But what I was most struck by was how dumb the software was, compared to the AI chatbot I was using to help guide me. I wanted to be able to tell the (stupid) software what I was trying to do. Such as “Carry my NOL from last year!” Or “I don’t know if I need a schedule K, you tell me!”

I don’t want another chatbot. I mean, I already have a good chatbot. I want the application to be well-integrated with AI services that understand the app, understand my current situation within the app, and meet me at the level of intent, applying the lessons learned by others who have used the same tools.

This kind of targeted, intelligent leveling up of intention is, from all I can see, the next stage of software development, and it is going to be massive.

Latency, non-determinism, and other challenges

We are trading the deterministic brittleness of classic SOA for the probabilistic fuzziness of SOA 2.0. And that trade is going to be demanded with ever more insistence by users. But it comes with a new set of trade-offs.

First, there is the latency tax. The old enterprise service bus was heavy to configure, but at run time, the messaging was just routed XML. Injecting an LLM into the critical path of an application adds hundreds of milliseconds, if not seconds, of latency. For asynchronous tasks or complex orchestrations, this is a welcome trade-off. For real-time, high-throughput microservices, it is a deal-breaker.

Second, there is the problem of non-determinism. We spent decades training ourselves (and our systems) to expect that given input A, a system will always produce output B. That deterministic equation was our bottom line faith. The intent layer doesn’t work that way. An LLM might route a request beautifully 99 times, then hallucinate a parameter on the 100th. Or it might choose an entirely different execution path based on a subtle shift in the user’s phrasing.

A third fly in the ointment is the so-called non-functional requirements, or NFRs. These are your pesky sidebar issues that refuse to be ignored, like security and reliability. 

Security concerns are magnified by model capabilities like function calling (or “function passing”). If you pair a user’s desires with what the AI can do, and you then let the AI decide, what happens next is clearly an act of faith unless guardrails are put in place. These guardrails must go beyond typical web security (i.e., make sure important function calls are hardened on the server, not exposed on the client) and must be internalized by the AI or (more likely) imposed from a layer outside the AI. There are a number of ways to do this, varying in degree of power and complexity. 

We certainly will continue to use standard practices (like RBAC and SSO) to enforce authentication. We will continue to implement standard authorization techniques (like OAUTH and JWT). But we will bring these to bear in the context of that intent layer and its capabilities.

Reliability is another challenge. For example, I recently hit a snag with Google’s Imagen API. Everything was working beautifully, then suddenly, some of the images stopped generating. There were no errors in the client or server logs; however, there were 500 errors in the network. Upon deeper examination, the prompting had morphed (between app context and user content) to include what the Imagen API rules deemed to be dangerous content. This was not obviously flagged prompting. It was fairly pedestrian creative writing, along the lines of “A dark, surreal, and glitchy cyberpunk landscape with menacing figures….” That kind of thing.

These are some of the ways that even simple, direct use of LLM APIs can surprise you. The question I am mulling is, what will be the unexpected outcomes on software writ large?

Dawn of a probabilistic web

Since its inception, the unpredictable, probabilistic nature of the internet came primarily from the humans using it (and background radiation flipping transistors, network failures, geopolitical effects on the ground, and the like). But AI-mediated APIs introduce an intentional, semantically controlled form of probability.

As developers, we will naturally discover the techniques that make consuming AI endpoints more effective. Here I am thinking about practices like structured responses and function calling. But the larger question is, what will the nature of software become?

In a world of binary states, strict protocols, and rigid URIs, if you send a GET request to a specific endpoint, you expect an exact, predictable response. We have spent the last 40 years treating the web like a vast, unimaginably complex state machine.

But as LLM-mediated APIs permeate our architecture with stochastics, the very fabric of the internet begins to change. By injecting AI into the routing and discovery layers, we are introducing a massive dose of probability into the foundation of our networks. When a request is no longer a hard-coded URI call but a natural language intent parsed by an LLM, the connection between node A and node B ceases to be a rigid wire. It becomes a weighted probability.

In essence, we are remaking the internet to mirror the architecture of the AI models we are deploying. Just as a neural network relies on the probabilistic firing of synapses rather than deterministic if/then statements, the next iteration of the web will rely on fluid, semantic discovery. Services won’t just “link” to each other; they will gravitate toward one another based on the conceptual proximity of their capabilities within a shared latent space.

This alters the character of software engineering. We lose (the illusion) of being entirely in control. Its strange paradox is that engineering using explicitly probabilistic components may make for a more resilient system. There is a longstanding debate about the best metaphor for software development. For the longest time, the construction of a building always seemed to be an apt analogy, or perhaps the mechanics of a vehicle. But these days, the gardening or cultivation metaphor is looking ever more relevant.

Despite the challenges posed by inserting AI in the stack, we are finally circling back to the original promise of the early 2000s. This time, fingers crossed, we are equipped with the right tools for the job.

We tried to build autonomous service discovery using rigid logic and deterministic XML, and it collapsed under its own weight. Now, we are building it with neural networks that understand the “intent” behind the integration. We are still building middleware, but instead of an enterprise service bus, we are building an enterprise reasoning bus.

The era of manually hard-coding every integration between every microservice may be coming to a close.