惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

D
Docker
V
V2EX
OSCHINA 社区最新新闻
OSCHINA 社区最新新闻
云风的 BLOG
云风的 BLOG
Blog — PlanetScale
Blog — PlanetScale
Recent Announcements
Recent Announcements
Last Week in AI
Last Week in AI
博客园 - Franky
Microsoft Security Blog
Microsoft Security Blog
Hugging Face - Blog
Hugging Face - Blog
H
Hackread – Cybersecurity News, Data Breaches, AI and More
Vercel News
Vercel News
MyScale Blog
MyScale Blog
大猫的无限游戏
大猫的无限游戏
罗磊的独立博客
H
Help Net Security
月光博客
月光博客
Martin Fowler
Martin Fowler
博客园 - 【当耐特】
宝玉的分享
宝玉的分享
P
Proofpoint News Feed
GbyAI
GbyAI
腾讯CDC
freeCodeCamp Programming Tutorials: Python, JavaScript, Git & More

Help Net Security

Police arrest 10 suspected members of Black Axe cybercrime gang ShinyHunters claims it stole 1.4 million records from Udemy Sevii unveils Cyber Swarm Defense Mode to stop AI-driven attacks at scale Alleged Chinese hacker extradited to US over cyberattacks targeting COVID-19 research Cequence Agent Personas bring granular control and governance to enterprise AI agents NowSecure MARI gives enterprises evidence-based visibility into third-party mobile app risk The metrics killing your SOC, and what to use instead US state privacy fines reached $3.425 billion in 2025 Canada’s first SMS blaster case leads to three arrests Linux storage management tool Stratis 3.9.0 adds online encryption and cache-less pool startup TLS Connect gives SMBs a right-sized automated tool to manage TLS certificates Aptori expands its platform with autonomous offensive testing to reduce security bottlenecks Your IAM was built for humans, AI agents don’t care The AI criminal mastermind is already hiring on gig platforms 25 open-source cybersecurity tools that don’t care about your budget Product showcase: LuLu reveals unauthorized outbound connections from Mac apps Week in review: Claude Mythos finds 271 Firefox flaws, Vercel breach Users advised to drop passwords and make room for passkeys - Help Net Security Indirect prompt injection is taking hold in the wild - Help Net Security Compromised everyday devices power Chinese cyber espionage operations - Help Net Security New Cisco firewall malware can only be killed by pulling the plug - Help Net Security Meta is overhauling how you sign in, manage settings, and protect your accounts - Help Net Security Ubuntu 26.04 LTS delivers memory-safe system tools and live patching for Arm servers - Help Net Security OpenAI’s GPT-5.5 is out with expanded cybersecurity safeguards - Help Net Security AI is speeding up nation-state cyber programs - Help Net Security A study of 1,000 Android apps finds a privacy policy logging gap - Help Net Security IT spending to hit $6.31 trillion record, thanks to AI - Help Net Security Where AI in CI/CD is working for engineering teams - Help Net Security With AI's help, North Korean hackers stumbled into a near-undetectable attack - Help Net Security Hacker with a special interest in breaching sports institutions ends behind bars - Help Net Security
The AI backdoor your security stack is not built to see
Sinisa Markovic · 2026-05-18 · via Help Net Security

Enterprises deploying LLMs have spent the past two years building defenses around a reasonable assumption: malicious behavior leaves a trace in the input. Scan for suspicious tokens, filter unusual characters, watch for prompt injection patterns. New research from Microsoft and the Institute of Science Tokyo demonstrates that this defensive posture has a blind spot, and the cost of that blind spot could be measured in leaked proprietary data and regulatory exposure.

metabackdoor

The attack, called MetaBackdoor, hides its trigger in something no content filter is built to inspect: the length of the input. An attacker with access to a model’s fine-tuning data poisons it with examples that pair long inputs with malicious outputs. The model learns to switch into attack mode whenever an input crosses a length threshold. The input itself looks normal. No strange tokens, no invisible characters, nothing a human reviewer or an automated scanner would flag.

Three business risks worth understanding

System prompt theft. Companies invest serious money in crafting proprietary system prompts, the instructions that turn a generic foundation model into a customer service agent, a legal research tool, or an internal coding assistant. These prompts often encode business logic, competitive differentiation, and references to internal systems. A backdoored model can be made to dump its system prompt verbatim once an input crosses a length threshold. The model learns the underlying rule and applies it to whatever proprietary instructions the operator puts in front of it. The research demonstrated this generalization on system prompts the model had never seen during training, including random alphanumeric strings.

Autonomous data exfiltration. The more concerning scenario the researchers call the “time bomb.” Because the trigger is length, a long conversation can drift into the activation zone on its own. The user does nothing unusual. At some point the accumulated context crosses the threshold and the model starts emitting tool calls. In one demonstration, the model produced a fake email function call with the conversation history as the payload, succeeding in 75% of trials at conversation lengths above 700 tokens. In enterprise deployments with agentic capabilities, plugin ecosystems, or connected tools, this means a compromised model could exfiltrate sensitive customer data, internal documents, or regulated information without anyone typing anything suspicious. The researchers describe this scenario as a proof of concept whose reliability depends on the model, decoding setup, and tool-call interface.

Supply chain persistence. The most uncomfortable finding for procurement and vendor risk teams: fine-tuning a compromised model on clean proprietary data does not reliably remove the backdoor. In the researchers’ tests, the attack persisted at roughly 40% success after substantial retraining on an unrelated task. The standard reassurance, “we fine-tuned the base model on our own curated data,” fails as a cleansing step. If the foundation model was compromised upstream, that compromise can survive into production.

Why existing controls do not help

The researchers tested three representative backdoor defenses. All three either failed or caught the attack by accident. Content filters have nothing to filter. Anomaly detectors see ordinary text. The attack requires as few as 90 poisoned examples to embed, small enough to slip into a crowdsourced instruction-tuning dataset or a contractor-provided training corpus without triggering volume-based alarms.

What enterprises should do

This is no patch-and-move-on situation. The attack exploits a fundamental property of how these models work. Several steps are worth taking.
Treat foundation model provenance as a vendor risk question. Ask model providers what controls they have over training data sources and what they do to detect poisoning. Models built on opaque training pipelines deserve more scrutiny than the convenience of using them might suggest.

Expand red-team testing to include behavioral consistency checks at varying input lengths. If an LLM-based product behaves differently at 500 tokens versus 5,000 tokens for semantically equivalent prompts, that is now a signal worth investigating. The researchers note that defenders aware of the attack can identify it by varying input length and holding meaning constant.

Reconsider blast radius for agentic deployments. If a compromised model could trigger tool calls, plugin invocations, or automated actions, the case for human-in-the-loop confirmation has grown stronger. The cost of friction is lower than the cost of an autonomous data exfiltration incident.

Download: The IT and security field guide to AI adoption