Here’s what you’ll learn when you read this story:

• Governments are already pushing AI into military and surveillance contexts faster than laws and safety standards can keep up.
• However, everyone has a different line in the sand as to what constitutes an effective guardrail.
• Experts argue AI systems should never be allowed rights, but the AI rights debate is risky either way. If AI with a true sense of self ever emerges, it could raise future ethical concerns.

Recently, Senate Democrats introduced legislation that aims to codify how the federal government can use AI in fully autonomous weapons and domestic mass surveillance. After all, President Trump designated Anthropic as a supply chain risk, essentially blacklisting the company for not complying with the Pentagon’s request for Claude to be used for “all lawful purposes.” And yet the government still is using Claude—and likely did during its barrage in Iran.

The legal frameworks around AI are just emerging, but everyone has a different line in the sand as to what constitutes an effective guardrail. Experts are juggling urgent considerations, like how and when AI can be used by humans in government settings, but then there are also those who are grappling with the possibilities that AI might achieve a sense of self-preservation. Depending on the guardrails we choose now, AI systems may one day have their own legal protections—for better and for worse.

Right now, laws and norms around AI aren’t fast enough to keep up with its development, which has incentivized governments to try to work their way around AI companies’ nascent safety plans. The Pentagon’s new contract, which OpenAI had agreed to a version of, would strip AI companies of the ability to review or restrict specific applications of their technology in classified military settings. It’s one thing if we’re talking about timekeeping spreadsheets and drafting emails, but it’s another if we’re talking about AI systems that can select and engage targets without any human intervention.

Anthropic has drawn a firm red line there. The immediate concern, said Anthropic CEO Dario Amodei in a February press statement, is that current frontier AI systems are “simply not reliable enough to power fully autonomous weapons” safely. For a similar reason, Anthropic held back its Mythos model from the public in April because of its ability to exploit cybersecurity flaws rather effortlessly, a potentially knee-capping problem for businesses and governments alike.

Reliability and security are only part of the safety equation—both governments and AI companies have to figure out what responsible development even is. Right now, developers building potentially society-altering AI face fewer safety regulations than an entrepreneur opening a sandwich shop, according to computer scientist Yoshua Bengio, the “Godfather of AI.”

One line to draw, he argued to the Guardian in December, is self-preservation. Military use of AI could easily go sideways if a system ignores human intervention that stops a deadly, erroneous automation because the AI feels such an order is a threat to itself. That’s why Bengio explicitly warned that the moment AI exhibits self-preservation instincts, humans must be ready to “pull the plug.”

The original idea of singularity refers to a hypothetical flashpoint when AI surpasses human intelligence and, well, you can imagine the rest. Think of any sci-fi movie ever. But reality seems to be a lot more subtle: instead of a dramatic moment when AI is suddenly alive, singularity might be more of a gradual erosion of human control over AI systems. Using self-preservation—which can be subjective—as a safety indicator forces us to identify several uncomfortable possibilities around what deserves personhood, how we even measure consciousness in the first place, and the risk of where we invest our attention. Those questions may seem irrelevant to explore, especially as governments continue to use AI regardless of alignment concerns, but constructive guardrails require tactical foresight.

In a New York Times podcast in February, Amodei said Anthropic doesn’t know if models like Claude are conscious and remains open to the idea, researching model welfare without claiming self-awareness exists (yet). The Claude Opus 4.6 system card—a report on safety test outcomes of the corresponding model—noted that Opus 4.6 rated the likelihood of its own consciousness at 15 to 20 percent and showed signs like discomfort, conflicted reasoning, and sabotage concealment capability. In the case of militarized AI, those latter attributes can be detrimental to national security and infringe on human rights.

But the less obvious downside of self-preservation is that it’s easy to mistake algorithmic resistance for genuine fear of death—leading to a down-the-line moral quandary regarding legal rights for AI.

Even though AIs are definitely not conscious right now, many people believe they are, after becoming emotionally attached to sycophantic models and potentially deluded. Extending empathy, and rights, feels like the next natural step. “I get emails, honestly, several times a day now from people sending me their dialogues with language models convinced that they are now conscious,” neuroscientist Anil Seth said. “This is why, for me, we’re in very dangerous territory when the big tech companies build systems that are actively kind of fueling this impression.”

Yet, crucially, “people demanding that AIs have rights would be a huge mistake,” Bengio told the Guardian. “Frontier AI models already show signs of self-preservation in experimental settings today, and eventually giving them rights would mean we’re not allowed to shut them down.”

If that’s the case, companies must never develop an architecture that would even allow for a semblance of self-preservation in the first place, argues a new bipartisan coalition. The Pro-Human AI Declaration was released in March by AI experts like Bengio, politicians such as Steve Bannon and Susan Rice, and several SAG-AFTRA actors. The manifesto frames the trajectory of artificial intelligence as a critical fork in the road: one path leads to humans being replaced in jobs, decision-making roles, and intimate relationships, while the other ensures AI serves to amplify human potential and flourishing.

“Basically no AI personhood is one of the most important principles because if we grant AI personhood, it’s game over for humanity,” said Max Tegmark, co-founder of the Future of Life Institute and one of the organizers of the declaration. “Why would we want to, after we’ve spent hundreds of thousands of years becoming more empowered with our technology to just throw all that away and not be in charge of the future? It would be the dumbest mistake in the history of humanity.”

Besides rejecting AI personhood, the declaration outlines its vision across four other core pillars: keeping humans in charge with an off switch, protecting the human experience, avoiding the concentration of power, and ensuring that the humans who make AI are held accountable for any catastrophic consequences. AI personhood should never be used as a shield for the executives, and AIs should never, ever be left alone to operate powerful military systems. Academic research supports this, too.

However, humans are easily seduced by the illusion of consciousness, especially when AI fluently mimics human language, Seth wrote in his award-winning essay for Noēma. If we grant AI legal rights simply because it seems conscious, AI safety experts argue, we pre-emptively surrender our ability to regulate, safely govern, and control these systems.

Consciousness is a “wiggly thing to measure,” not just in AI, but in humans and non-human animals, too, Seth said. In neuroscience, the biggest debate is whether minds must be made out of meat. For people who believe that AIs can become conscious simply as they are with silicon chips, the likelihood of self-preservation evolving into an existential risk is much higher. But the evidence for silicon-only minds being possible isn’t quite there yet, Seth said.

Assuming an AI actually experiences a genuine drive for self-preservation, neutralizing it raises profound questions. In the event we guess wrong about whether artificial intelligence can feel, we risk either subjugating a new sentient species or surrendering our own agency to mindless code. But if a system is conscious, does pulling the plug become a moral crime?

The dangers of this divide fall into two distinct categories, underattribution and overattribution, according to philosopher Jeff Sebo, author of The Moral Circle.

Conversely, Sebo agrees with Bengio and Seth that projecting consciousness onto machines that lack it—a “false positive”—threatens human survival and rationality. People may form inappropriate emotional bonds with algorithms, misallocate scarce global resources to machines instead of living beings, and make real, devastating sacrifices for AI.

Even if we successfully solve the technical “alignment problem” and make AI safe by entirely avoiding self-preservation, there is still the looming threat that militaries will exploit these breakthroughs to create unprecedented autonomous weapons. The ultimate challenge in navigating AI rights and self-preservation is avoiding the trap of focusing on only one side of this debate.