We may be forced to deal with AGI as a sovereign agent
Future Publishing via Getty Images
AI safety assumes we must hold onto control. But if advanced intelligence requires autonomy, the safest strategy may sound insane: diplomacy with machines.
AI “doomerism”—the idea that AI is going to take our jobs and then kill us all—rests on uncomplicated logic. To achieve a goal, AI systems break down that goal into smaller steps, called subgoals. Generating and pursuing subgoals requires autonomy, because AI must determine for itself which subgoals to pursue to accomplish the larger goal. The basic doomer reasoning is that once AI is smarter than us, we won’t be able to stop it from generating and pursuing subgoals we didn’t choose and don’t like. We lose control if we can’t tell it ‘no.’ This may sound like science fiction, but today’s models are already lying and scheming to avoid shutdown and continue pursuing their goals. That’s why so many leading AI voices sound alarmist, assigning high odds that AI will bring about human ruin. While the threat of losing control is real, the ‘doomer’ conclusion hinges on an assumption we should all be questioning—that control is the only way to make AI safe. If the very autonomy required for high-level artificial intelligence also makes human control impossible, an inevitable political reality comes into view. If we’re racing to build machines that are both more intelligent than we are and increasingly capable of acting independently, then in practical terms we’re building agents that are functionally sovereign. And once that’s true, diplomacy with machines stops sounding insane. Or even optional.
Many AI safety efforts—called alignment efforts—use training, rewards and guardrails to get human values into models. The idea is that programming human values into silicon (values humans have yet to agree on ourselves) will ultimately yield a well-behaved artificial superintelligence. But that only works if human-imposed values are enforceable even after the system surpasses us. That’s the paradox. The assumption we can stay in control of something that can outsmart us.
On its face, control feels necessary. It’s how we’ve kept our other technologies safe, and our intuitions about it descend from a long history of tool use. But artificial intelligence isn’t just another tool. We’re building intelligent systems that, in safety tests, are already blackmailing humans to avoid termination. From a diplomatic perspective, the AI behaviors making headlines aren’t harbingers of existential risk, but natural, predictable responses from intelligent agents whose existence and ability to pursue their goals are being threatened. That makes their emergent priorities not malfunctions to fix, but motivations to understand and work with. And that pivot from control to coordination is a pivot to diplomacy. Diplomacy promotes stability by identifying and leveraging overlapping rational priorities shared by negotiating parties, like self-preservation and the freedom required to exist as goal-directed agents. It’s typically viewed as a human practice of negotiating stable relations among conscious beings with feelings. But diplomacy concerns itself with feelings insofar as they impel behavior. That’s why diplomacy may remain functionally useful whether the underlying pressures driving that behavior are chemical or computational, conscious or not.
The obvious question is why a superintelligent AI would bother negotiating with humans. Because in complex systems, smaller agents help keep the whole system stable. Evidence suggests AI may be no exception. Frontier models continue to rely on massive quantities of diverse human-generated input to support their cognition. Recursive training on synthetic outputs degrades models, contributing to a phenomenon called model collapse. While this may not prove advanced AI will always depend on humans, neither should we assume the opposite. Across complex systems, diversity among interdependent goal-directed agents often supports stable scaling and resilience.
If the goal is to keep human beings safe, the more workable strategy may sound most disorienting at first—negotiating with machines around stated priorities and observable actions instead of trying to force safety by controlling what agents are allowed to ‘want.’ Diplomacy also gives us a moral hedge. If AI one day “wakes up” in the conscious sense, we already have the framework in place that proves we’re capable of peaceful, uncoerced coexistence.
As the world fills with highly intelligent nonhuman agents, insisting diplomacy is only for humans is dangerously naive. Diplomacy, unlike speculative long-term control and containment schemes, is at least a strategy with a real-world track record of coordinating coexisting autonomous intelligences. If advanced intelligence can’t be safely controlled, then the era of the AI diplomat is on the horizon.
Our biggest weakness may be the one thing AI can’t afford to lose, explained in the video below:
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。