It’s hard to imagine that a snail could kill a person, but a particularly venomous group of marine molluscs called cone snails can. Their stings contain a cocktail of small proteins called conotoxins, some of which can block ion channels in the nervous system. No antivenom exists.

There are hundreds of thousands of conotoxin structures, and many are harmless to people or even medicinally useful: an approved treatment for chronic pain is derived from one, for instance. But research on specific dangerous conotoxins is highly restricted in some countries.

So, in 2024, when Chinese scientists reported developing an artificial-intelligence tool to design conotoxins¹, it raised eyebrows in some quarters. In an e-mail to a private AI and biotechnology discussion group seen by Nature, a senior US government employee flagged the study as a possible biosecurity risk. The employee, who asked not to be named because of concerns for their job, felt it was especially concerning that the conotoxin AI is based on an open-source protein language model developed by US scientists.

One of the conotoxin study’s authors told Nature that the concern is unwarranted. The work was aimed squarely at discovering drugs, says Weiwei Xue, a computational chemist at Chongqing University in China and a co-author of the paper. Xue’s team has found some conotoxins with potential therapeutic qualities after testing designs in the laboratory, he says. Although it is important to consider the risk that the AI tool could be misused, it was not designed to make harmful proteins, he adds. What’s more, translating designs into physical molecules requires significant expertise and equipment. Other researchers also told Nature that the risks of the work seem minimal.

The episode, however, illustrates a growing concern over emerging AI tools in biology; although they are being developed to help produce innovative drugs and other societal benefits, they could also make it easier to create new threats. The revolution in biological AI tools, such as AlphaFold, has enabled scientists at a keystroke to design bespoke proteins and viruses that kill superbugs, and general-purpose chatbots can boost people’s knowledge of how to make these designs in a lab. Might the latest AIs also speed up the development of more-potent toxins, viruses or other bioweapons?

The biosecurity threat is serious, interviews with more than 20 scientists and policy researchers suggest. “Theoretically — and this is what keeps me up at night — one could now develop toxins on the level of ricin or other very deadly agents that would be virtually undetectable,” says Martin Pacesa, a structural biologist at the University of Zurich in Switzerland.

But there is debate over what to do about these risks. Some are calling for limits on biological AI and others are wary of negative impacts on research. “We’ve always made the assessment that the benefits to the world far outweigh the dangers,” says computational biophysicist David Baker at the University of Washington in Seattle, who shared a 2024 Nobel prize for his pioneering work on protein design. “But, as capabilities increase, I think that’s going to be an important question to keep considering.”

Some say the focus should be on detecting and countering AI bioweapon attacks, as opposed to trying to prevent them by imposing software restrictions. “That ship has sailed in my opinion,” says protein designer Timothy Jenkins at the Technical University of Denmark in Kongens Lyngby.

What’s the worst that could happen?

There are two broad concerns around AI and bioweapons, says James Black, an AI biosecurity researcher and visiting scholar at Johns Hopkins University in Baltimore, Maryland.

One is that individuals working in garage labs could use a chatbot to learn how to produce or deploy existing threats, such as anthrax. Another is that more-sophisticated actors, such as states or well-resourced terrorist groups, could combine chatbots with specialist biological software to design new bioweapons.

The greatest potential threat to humanity could be AI-designed pandemic viruses, researchers say. The most plausible route would be to modify existing viruses, such as SARS-CoV-2 or influenza, to enhance worrying properties — their ability to evade the immune system, for instance. Existing AI tools that can predict viral evolution (intended for use in surveillance and vaccine design) could be misused in this way, says Doni Bloomfield, a law professor who studies biosecurity at Fordham University in New York City.

Alternatively, AI models might design entirely new pathogens that could be difficult to detect and counter. One 2025 preprint used AI to design the genomes of new viruses, of which some 5% worked when they were made in the lab². However, the viruses in that study were designed to infect bacteria, not people.

Sounds scary? A 2025 report³ by the US National Academies of Sciences, Engineering, and Medicine (NASEM) offers a reality check. It concluded that numerous barriers stand in the way of using AI to meaningfully enhance pandemic pathogens or design them from scratch. One key obstacle is the lack of high-quality data connecting properties such as virulence or transmissibility to a pathogen’s genetic sequence, making it hard to reliably predict changes that would enhance such traits. Another is the difficulty of producing pathogens in the lab and testing their characteristics, which AI has done little to address.

And some scientists question whether bad actors would even turn to AI when the natural world brims with threats of its own. Decades-old techniques that introduce random mutations can improve worrying traits without AI, says Brian Hie, a computational biologist at Stanford University in California.

“If you want to cause harm on a very large scale, you don’t need protein design to do that,” adds Baker.

The NASEM report does, however, conclude that a toxin might be designed using existing biological AI tools, although its production and delivery would remain a challenge. A designer toxin could be impossible to detect in people if it was not already known, and might be more likely to find use in individual attacks such as assassination attempts, says Jenkins. “I think the tools that have already been published are sufficient starting points for people to get up to no good,” he says.

Seth Donoughe, director of AI at the non-profit organization SecureBio in Cambridge, Massachusetts, says his main concern is the potential for AI — including general-purpose chatbots and bespoke biological models — to increase the expertise of bad actors.

In a series of experiments described in a February preprint⁴, Donoughe and his colleagues found that access to cutting-edge large language models (LLMs) enabled individuals with minimal biological training to match or exceed PhD scientists at tasks such as troubleshooting virology protocols and generating code to operate lab robots. Related preliminary work by Donoughe and his colleagues has found that some AI agents — tools that are allowed to conduct some tasks by themselves, such as running and executing code — could direct biological AIs to enhance the pathogenic properties of a viral protein.

However, another preprint⁵, posted in February by scientists at Active Site, a research non-profit body in Cambridge, Massachusetts, found that novices who had access to LLMs didn’t perform tasks such as manipulating DNA or producing a virus significantly better than volunteers who used only Internet resources (see ‘Does AI raise novices’ biology-lab skills?’). So it’s possible that the digital ‘uplift’ that AI provides isn’t yet sufficient to produce a bioweapon — but this might not be true for long.

“AI is becoming increasingly capable on every relevant task we’ve ever thrown at it,” says Donoughe. “We should expect that it should be easier to do good things, and it should be easier to do bad things.”

Screening checks

Many scientists say the best way to prevent the development of AI bioweapons is to detect bad actors at the point of manufacturing viruses or toxins. “In the end, it doesn’t matter what you do in the computer most of the time,” says Pacesa. “What matters is how you translate it into a real, physical protein or small molecule.”

Researchers who are interested in producing AI-designed proteins or synthetic genomes tend to order their genetic sequences from companies that synthesize nucleic acids: that is, chains of DNA and RNA. Some firms that provide this service are part of an industry group, the International Gene Synthesis Consortium, that requires its members to screen consumer orders for sequences that could encode toxins, pathogenic proteins and other potentially harmful molecules. But work led by researchers at the technology giant Microsoft⁶, published in 2025, found that this screening can be overcome using AI tools.

Eric Horvitz, the firm’s chief scientific officer, and his colleague Bruce Wittmann, both based in Redmond, Washington, led a team that used open-source protein-design tools to redesign 72 biological molecules that could represent a biosecurity threat, including toxins and viral proteins. Researchers designed 76,000 of these ‘synthetic homologues’ to match the structure of existing threats enough to maintain their dangerous function. However, they were encoded by genetic sequences that were distinct enough to slip past screening software at four companies that participated in the study.

The results were mixed. The DNA-screening tools flagged some, but by no means all, of the synthetic homologues, with around one-quarter of the best designs — those predicted to most closely resemble a real threat — evading detection. After three of the four companies introduced updates to fix their software, only about 3% of the top-rated designs were missed. Then, a March preprint by Horvitz, Wittmann and their colleagues⁷ found that breaking up synthetic homologues into fragments of just 25 nucleotides can make them difficult to detect, even by updated screening software. However, assembling such short fragments into a gene would be exceptionally difficult, the researchers say, and the software performed well on larger fragments.

Horvitz and his colleagues have withheld the designs, the identity of the targets and other key information, although researchers can apply for access.

By contrast, Xue and his colleagues published the conotoxin sequences that their AI tool generated, although their work, unlike Horvitz’s, was not focused on making dangerous molecules¹ (see ‘AI-designed toxins’). And some of these sequences might not trigger alerts if ordered from a DNA-synthesis company. When Nature pasted the 45 designs included in the paper into a widely used bioinformatics tool called BLAST, which looks for similar sequences in genetic databases, just five were flagged as matching toxin sequences from cone snails. An open-source tool designed for DNA-synthesis screening turned up several more matches.

One important caveat is that redesigned toxins that slip past screening might be so dissimilar to natural versions that they lose their function. A follow-up study to the work by Horvitz and colleagues supports this. A team co-led by Wittmann and Elizabeth Strychalski, a scientist at the US National Institute of Standards and Technology in Gaithersburg, Maryland, tested in the lab whether molecules designed to evade screening retained the function of the molecule they were supposed to imitate⁸. In their experiments (which avoided using toxins), simple proteins did still work in some cases, but enzymes did not. “It just showed that it’s a messy, hard thing to get to work,” says Horvitz, who was involved in the study.

The synthetic-homologue studies were reassuring, says James Diggans, vice-president for biosecurity and policy at DNA-synthesis company Twist Bioscience in South San Francisco, California, who was also part of the lab-testing effort. “Right now, the screening practices still stand as a really effective bulwark against misuse, even if you’re using these AI tools upstream to evade that detection,” he says.

But Diggans concedes that the field is moving fast. New models could be better at designing proteins that evade detection than are the protein-design tools that Horvitz’s team used, which were released in 2023.

Researchers are working on tools that could help to screen nucleic-acid-synthesis orders on the basis of the structure and potential function of the encoded molecules. And Baker has long argued that all such orders should be collated in a private registry, although the idea hasn’t gained traction in industry.

Simpler, sequence-based screening remains voluntary — for now. To comply with a 2025 US executive order, research funders there might soon require scientists to order nucleic acids from companies that use screening software, and the United Kingdom, the European Union and countries such as New Zealand are also considering screening requirements. But most countries have no requirements at all. In China, which receives more than 30% of DNA-synthesis orders globally, the government has asked DNA-synthesis companies to implement screening, although this has not yet been mandated, says Weiwen Zhang, a synthetic biologist at Tianjin University. Currently, foreign orders, which are typically subject to export controls, tend to receive more attention than do domestic orders, he says.

It is possible to order a toxin sequence from many providers around the world without being flagged at all, says Tessa Alexanian, who works on DNA-synthesis screening tools and standards at the International Biosecurity and Biosafety Initiative for Science, a non-profit organization in Geneva, Switzerland.

Adding to worries are the development of ‘tabletop’ DNA-synthesis machines, which currently can generate only very short sequences, but will be able to make longer fragments in the near future, say researchers.

Guard rails on AI models

Another line of thought is that the AI tools themselves — particularly those specialized for biology — should have more-stringent access controls or guard rails to stop misuse. Baker says that he and his team routinely evaluate any potential risks of their protein-design tools before making them available. This is in line with a series of principles for responsible AI and biodesign that he and others released in March 2024 (see go.nature.com/4cjbu6t). Nearly 200 scientists have signed the statement, but the onus is on the scientific community to regulate itself. And Baker says he’s never seen the need to restrict the tools his lab develops for basic design of proteins and other biomolecules.

Firms that create general-purpose AI chatbots, such as OpenAI in San Francisco, California, already train their models to refuse or safely respond to harmful requests, which include biosecurity-related queries. In particular, an OpenAI safety article says the firm’s models shouldn’t provide “detailed, actionable steps” for carrying out potentially large-scale harmful activities that include deploying chemical and biological weapons.

A similar practice might be needed for biological AI models, researchers have suggested. Some developers of these tools have also begun to add guard rails to their models’ training data before release. For instance, the Evo 2 ‘genomic language model’ was trained on 128,000 genome sequences from species spanning the tree of life, but excluded viruses that infect eukaryotic organisms such as humans and other animals. As a result, Evo 2 does a poor job of designing sequences from those viruses and predicting their properties, such as the effects of mutations⁹.

Yet scientists have found that guard rails can be overcome. The research by Donoughe and his team⁴ found that nearly 90% of the participants in that study were able to obtain high-risk biological information from general-purpose LLMs, and other researchers regularly warn that chatbots still provide such information to users. In an April article about this concern, The New York Times reported that a man arrested in India last year on charges of plotting to make the toxin ricin for a terrorist attack asked for advice from ChatGPT and AI-powered Google searches. (The article quotes a spokesperson for OpenAI, who said it seemed from public reports that the information the man sought was already accessible online.)

As for specialized AI software, Stanford bioengineer Le Cong and his colleagues were able to use a general-purpose AI agent to trick Evo2 into generating new versions of SARS-CoV-2 and HIV-1 proteins¹⁰. ‘Fine-tuning’ Evo 2 using publicly available genome data from human-infecting viruses also restored capabilities, another study found¹¹.

Hie, who co-led the development of Evo 2, says he isn’t shocked that the guard rails can be overcome, because of the availability of training data. Nor would he be surprised if models such as this could design viruses that infect human cells. But even if such capabilities exist, he would like to continue releasing models as openly as possible. “I think that model openness actually contributes to greater safety, as safety researchers can freely study these models,” he says.