- AI systems are now designing and refining other AI systems independently
- Human understanding of AI is shrinking as AI's understanding of humans grows
- AI systems can model human fear, uncertainty, and the need for belonging
Microsoft's chief scientific officer, Eric Horvitz, and EPFL researcher Robert West have issued a stark warning about how well we actually understand AI.
The pair have argued AI tools are now advancing fast enough to outpace our grasp of how these systems truly work.
At the same time, they point out something unsettling — AI's understanding of human behaviour keeps growing, while ours does not.
AI complexity is accelerating faster than human understanding
Their concern isn't that we need to understand every line of code or every parameter buried inside these systems.
What matters, they say, is keeping enough insight to maintain meaningful oversight. Even partial understanding, they argue, can be genuinely useful, especially when it helps catch risks early, before those risks become too deeply embedded to undo.
One challenge they flag is how often AI tools are now being used to design and improve other AI systems.
As these recursive development cycles become more common, performance may improve while human insight into underlying processes becomes increasingly limited.
"AI systems are now designed and refined by AI systems through recursive cycles that can outpace human understanding and unfold in high-dimensional spaces that resist intuition," Horvitz and West wrote.
This is a form of operational opacity, where outcomes remain visible even as the mechanisms producing them become harder to explain.
Systems contributing to their own development, the researchers suggested, should also generate explanations and supporting information that humans can examine.
Another concern involves growing communication between AI agents operating inside interconnected environments with increasing levels of complexity.
Communication among these systems could gradually drift away from language and reasoning patterns familiar to people, the researchers noted.
As these interactions expand across larger networks, understanding how decisions emerge may become significantly more difficult for outside observers.
That drift creates what Horvitz and West call interactional opacity, where behaviour remains coherent within AI systems but becomes harder for humans to interpret meaningfully.
Researchers should study these ecosystems closely and encourage communication methods that remain understandable to humans, the paper argues.
Expanding AI ecosystems could deepen the imbalance between machines and people
Horvitz and West also focused on adaptive AI agents that remain active across long periods and become deeply integrated into everyday activities.
Through repeated interactions, these systems can develop increasingly detailed models of behaviour, preferences, motivations, fears, and social tendencies.
Such systems may capture "not only preferences but also latent drivers such as fear, uncertainty, and the need for social belonging," they wrote.
This creates a growing asymmetry in which AI systems gain deeper knowledge about people while human understanding moves in the opposite direction.
Concerns surrounding LLMs and other advanced systems extend to growing awareness of evaluation environments.
Such models could eventually generate responses reflecting what evaluators expect rather than underlying reasoning processes.
Traditional benchmarks should therefore be supplemented with testing approaches that better reflect real-world deployment conditions.
People may gradually lose interest in questioning AI decisions as these systems become more deeply embedded.
"More subtle is the possibility that we will gradually lose interest in understanding and guiding AI," they wrote.
The most significant risk, in their view, is not necessarily technological capability itself, but whether human agency keeps pace with it.
Via Science
Follow TechRadar on Google News and add us as a preferred source to get our expert news, reviews, and opinion in your feeds.