Should the AI-powered drones of the future have a licence to kill? The question is becoming ever more pressing as governments and the defence industry acknowledge that drone systems will play an increasingly crucial role in future warfare.

With drones being deployed in huge numbers in the Ukraine war and AI being used to assist bombing missions in the Iran conflict, there is an expectation among some observers that weapons will have to operate with increased operational autonomy, which means they will need something approximating a moral framework.

Last year Mustafa Suleyman, chief executive of Microsoft’s AI arm and a co-founder of the UK-based DeepMind, was unequivocal about the issue of machines making moral decisions. He said: “AIs cannot be people – or moral beings.”

David Omand, the former head of the UK spy agency, GCHQ, has told the Guardian he believes AI can create a “moral” configuration for unmanned weapons, while the UK armed forces minister, Al Carns, told the Financial Times recently there must be an option to “take the human out of the loop” in decision-making. This brings into focus the ethical and technological challenge of whether morality can be programmed into an autonomous weapons system.

Zee Talat, an academic specialising in machine learning at the University of Edinburgh’s school of informatics, argues that large language models – the technology that underpins modern generative AI systems such as chatbots – are fundamentally incapable of moral decision-making.

AI systems are trained on vast amounts of data that allow them to build a probabilistic model of what is the next most likely word or sentence in a sequence. This is not how humans make moral and ethical decisions, says Talat, even if the model has ingested all the philosophical tracts known to humanity.

“If you have a machine that’s probabilistic by nature it will veer towards the most likely answer in a situation. Do we think that morality follows probabilistic notions?” says Talat, who has recently co-authored a paper that argues ethical evaluation is an “open-ended, debate-based, sociopolitical process”, which AI cannot keep up with.

Andrew Rogoyski, of the Institute for People-Centred AI at the University of Surrey, says AI systems have become much more sophisticated since the arrival of ChatGPT in 2022 – as the emergence of so-called “reasoning” models shows. Nonetheless, can they replicate the nuance of moral decision-making?

“Morality is deeply complex, contested, culturally shaped, and something most humans never fully resolve, even for themselves,” he says. “Perhaps the real question is whether we understand morality well enough to codify it. Until we do, we cannot expect machines to embody something we ourselves cannot clearly articulate.”

This leads to the question of whether it is possible to have a universally recognised moral code for autonomous weapons. Programming morality into drones is problematic, according to Jessica Dorsey, an assistant professor of international law at Utrecht University in the Netherlands. Her concerns include determining whose morality the drone is following, a difficult process given the United Nations is still trying to achieve a global consensus on autonomous weapons governance.

There is also the issue of how to programme an AI system to distinguish between a combatant and a civilian. Article 57 of the Geneva conventions – a series of international treaties regulating the conduct of war – states that combatants should do “everything feasible to verify that the objectives to be attacked are neither civilians nor civilian objects … but are military objectives”.

If you don’t get the law right you will end up repeating flawed decisions on a vast scale if AI-powered drones are deployed en masse, says Dorsey.

“War is filled with so many variables and it is a given that things will go wrong. And when that happens at AI-like speed, it is difficult to unravel,” she says.

But some experts argue that giving drones greater autonomy, and programming rules of engagement and morality into them, will be a necessity if other nation states continue to develop and deploy similar technology at pace.

“For any military to compete effectively against other high-end militaries it is going to need a large amount of systems that will be required to take decisions on their own,” says Nicholas Wright, a neuroscientist and author of Warhead, a book on the human brain and war.

AI-powered drones are still a nascent technology, and, despite tech industry hype, there are only limited examples of their deployment on the battlefield. More than 100 startups across the US and Europe are now building drones and drone software platforms. These range from light, low-altitude drones designed for imaging and surveillance to heavier craft designed to carry weapons. But the companies building them – and envisioning the warfare of the future – have sharply conflicting views on what their decision-making might look like in future wars.

“Morality is the province of human beings, which is why AI-assisted weapons systems need to be built in a way that extends the judgment and decision-making of the operator rather than wholesale replacing it,” says Olaf Hichwa, the co-founder of Neros, a US drone startup.

For Hichwa, drones will not replace human decision-makers, but enhance the abilities of their human pilots.

“A lot of people are looking at autonomy somewhat incorrectly. They’re not looking at it through the eyes of the user. They’re saying, let’s make fully autonomous weapon systems,” he says.

“But the reality of warfare is that it’s still very much a human-on-human interaction.”

One use of AI in warfare, he says, might be to reduce the cognitive burden of the people who have to operate craft such as FPV (first-person view) drones – small craft heavily used in the Ukraine conflict which transmit a view of the battlefield to their human pilots via a video link. In one scenario, a drone could guide itself for the “last mile” of a mission.

“One example of autonomy is … right now, pilots require a lot of very precise training to get the drone to travel along the right angle. Some autonomy could be the user or the human sets a general path and the drone follows that path.”

Jon Gruen, the chief executive of Fortem Technologies, which builds systems to defend against drones, has a different view. He points out that the US already uses autonomous defence systems to intercept incoming drones and missiles, and that AI is used “all the way through” in many of these instances, in identifying targets and, at times, launching a weapon at them.

“When you detect something, you then need to identify it. That identification is done autonomously now for the most part.”

The same technology can now be used to deploy a weapon, he says, including when to fire a weapon and then guiding it to its target.

Alex Fink, the US chief executive of Swarmer, a US-Ukraine startup that develops software for autonomous drones, says there could be a future scenario where humans might not select individual targets at all but instead designate a “kill box”, in which anything in a given area at a given time is fair game for an autonomous system to target.

“A human certifies this area, there are no friendlies, and I guarantee there will be no friendlies in the next 15 minutes. Any vehicle in this area is pre-approved for this period of time,” says Fink.

“If a human is able to take that responsibility upon themselves, that they know there will be no friendlies, no bystanders in this area, then that type of mission is possible.”

But if the technology is there, the moral and legal consensus is not.