When I first started in the field of data science, I was so fascinated by ML algorithms and predictive analytics. Like every good data scientist worth their salt, of course I attempted to build an algorithm to predict the stock market. Naively, I had the brilliant idea that I am secretly going to discover the grand equation behind stock movements and crack the code to the future - Retirement here I come.

Gentle reader, was I wrong.

Not only machine learning was not the magic bullet I thought it would be - it turned out predicting the stock market is not all that straightforward - in fact more people get it wrong than right.
You know why? Because markets aren’t deterministic! And neither is the real world (fight me!)

Take for instance this supply chain assignment in my Operations theory grad course. Our final year grade exam was an assignment of running a real world simulated supply chain for a week. The task was to maximize profit while striking the right balance between meeting demand and reducing overstock. At its core, this was a forecasting problem - we had to accurately guess customer demand in order to plan order volume, efficient storage, transport routes etc. I cycled through a bunch of predictive algorithms from a time series to regression and xgboost. And yet in the end, the algorithm that forecasted the best was a rolling 3 week average. Not because a rolling average is statistically more powerful than an xgboost, but because real world data often is more noise than perfect trend.

A decade later not much has changed, AI and all.

AI can now fetch trends faster than you can say analyst and build sophisticated models within minutes. But the fact remains - real data is noisy. The universe is random and riddled with anomalies and exceptions.
And the best algorithms cannot eliminate uncertainty, in an inherently probabilistic world.

The AI will replace us argument fundamentally boils down to the age old debate of determinism vs free will.

At its core, the determinism argument subscribes to the viewpoint that the universe can be completely explained by a core set of rules and laws. Given the right set of variables, one would be able to explain every occurrence in a universe. This also means if you know the state of the universe at time t you are able to predict the state of the universe at t+1. In short, all events are an effect of previous state / cause - and everything is predetermined. By conclusion, the determinism argument implies, the decisions that we make are not really free choices, but preset actions triggered based on variables that led up to that moment.

For example - We know it takes 365 days for the earth to rotate around the sun. So if we know the temperature of the sun on day 1, through laws of physics we should also be able to derive the exact location of earth and the temperature on day 365. This temperature then helps determine the chance of rain on day 365 - which then determines the chance of you stepping out of the house. So determinism argues the likelihood of you stepping out on day 365 was already determined by the temperature of sun on day 1, which in turn was determined by position of the sun at the beginning of the universe and so on.
Everything that happens is determined by events preceding it

On the reverse, the free will argument argues the universe is non-deterministic and the choices we make our own. Up until the 20th century the Newtonian physics model had little evidence to negate the determinism theory. From planetary motions to atomic matter - all seemed to follow the same basic set of physical equations suggesting the universe was contained in an explainable set of mathematical equations. That is until the discovery of quantum mechanics. Quantum physicists noticed things are far more interesting when objects get very small or large or fast and at those scales outcomes aren’t pre-fixed but more probabilistic in nature. This is most fundamental argument to support the universe being non-deterministic and by conclusion the idea of free-will

At its core the AI will automate everything argument hinges on the assumption that we operate in a perfectly well-defined, explainable universe waiting to be discovered with more sophisticated algorithms and compute. But a non-deterministic universe negates that.
AI is not going to eliminate noise in a inherently probabilistic universe - just surface it faster

Take for instance, the odds of life on earth. Studies shows there is less than 0.001% chance of planets like earth and intelligent life existing. And yet we exist in the bottom 1 in 10 million probability curve - on a floating rock paying taxes!

The mathematics can tell you the odds, but in a non-deterministic universe it still can’t dictate what will actually happen.

We fundamentally live in a messy, undefined random universe where things exist beyond mathematical odds and defined frameworks. Entropy can’t be automated.

This doesn’t mean I’m an AI skeptic - far from it. AI is fundamentally disrupting industries and the way we work and there is a massive shift happening in career journeys and employability, faster than we are prepared for. But AI cannot automate inherently random real-world systems - even if the market seems to think so. That also doesn’t mean we will not see massive disruptions due to layoffs and cost-cutting. The shareholders demand profits and profits the market gets. But replacing humans with AI doesn’t just magically make systems more better and efficient - it just surfaces the same noisy problems faster (and at more costly rate?). As evidenced by multiple studies showing no noticeable gains in productivity, even though AI advancement is everywhere!

That doesn’t mean every job is safe. Some job are more automatable than others. For instance - If your job is a building a set of rules and enforcing systems based on those rules - especially digital systems - these are the easiest for AI systems to replicate. Examples are data-entry jobs, scheduling, accounting, any job with more than 80% repeatable patterns

However, if your job operates with an inherently probabilistic space, AI is going to help more than harm. The time to write code or algorithms yourself is probably past (although I disagree with this too - working code != good code. But topic for another post) but the best algorithms and systems are only tools that will surface the same uncertainties - faster. The most valuable skill to learn for these roles is navigating uncertainity.

Again, these roles will not involve engineers building the actual predictive algorithms but being involved in probablistic decision making process and making the right decisions in light of these predictions.