惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

D
Docker
Apple Machine Learning Research
Apple Machine Learning Research
宝玉的分享
宝玉的分享
博客园 - 叶小钗
酷 壳 – CoolShell
酷 壳 – CoolShell
博客园 - 司徒正美
OSCHINA 社区最新新闻
OSCHINA 社区最新新闻
博客园 - Franky
爱范儿
爱范儿
罗磊的独立博客
IT之家
IT之家
钛媒体:引领未来商业与生活新知
钛媒体:引领未来商业与生活新知
N
Netflix TechBlog - Medium
云风的 BLOG
云风的 BLOG
P
Proofpoint News Feed
U
Unit 42
Engineering at Meta
Engineering at Meta
WordPress大学
WordPress大学
博客园 - 三生石上(FineUI控件)
T
Tailwind CSS Blog
H
Help Net Security
博客园_首页
freeCodeCamp Programming Tutorials: Python, JavaScript, Git & More
人人都是产品经理
人人都是产品经理

Help Net Security

ChatGPT advanced account security adds passkeys and hardware keys Week in review: High-severity LPE vulnerability in the Linux kernel, cPanel 0-day exploited for months Automating Pentest Delivery: A Step-by-Step Guide - PlexTrac Open-source privacy proxy masks PII before prompts reach external AI services Shadow AI risks deepen as 31% of users get no employer training Identity is the control plane for distributed infrastructure AI traffic is getting bigger, louder, and less predictable New infosec products of the month: April 2026 cPanel zero-day exploited for months before patch release (CVE-2026-41940) Cisco releases open-source toolkit for verifying AI model lineage Met Police face criticism for using AI to spy on their own officers Nine-year-old Linux kernel flaw enables reliable local privilege escalation (CVE-2026-31431) Hacker with a special interest in breaching sports institutions ends behind bars - Help Net Security IP Fabric MCP server adds governance and control to enterprise AIOps workflows - Help Net Security Aqua Compass MCP server enables real-time investigation and containment of runtime threats - Help Net Security Google brings instant email verification to Android, no OTP needed - Help Net Security If cyber espionage via HDMI worries you, NCSC built a device to stop it - Help Net Security Apple fixes iPhone bug that let FBI retrieve deleted Signal messages(CVE-2026-28950) - Help Net Security GopherWhisper APT group hides command and control traffic in Slack and Discord - Help Net Security OpenAI tackles a bad habit people have when interacting with AI - Help Net Security A year in, Zoom's CISO reflects on balancing security and business - Help Net Security Scenario: Open-source framework for automated AI app red-teaming - Help Net Security GDPR works, but only where someone enforces it - Help Net Security Ransomware, fraud, and lawsuits drive cyber insurance claims to new peaks - Help Net Security Google’s Workspace Intelligence promises privacy while running on your data - Help Net Security Cyberattack on French government agency triggers phishing alert - Help Net Security Claude Mythos finds 271 Firefox flaws, Mozilla believes zero-days are numbered - Help Net Security Prove Identity Platform connects verification, authentication, and fraud prevention - Help Net Security New Mirai variants target routers and DVRs in parallel campaigns - Help Net Security Acronis GenAI Protection gives MSPs control over AI usage and data risks - Help Net Security
The AI backdoor your security stack is not built to see
Sinisa Markovic · 2026-05-18 · via Help Net Security

Enterprises deploying LLMs have spent the past two years building defenses around a reasonable assumption: malicious behavior leaves a trace in the input. Scan for suspicious tokens, filter unusual characters, watch for prompt injection patterns. New research from Microsoft and the Institute of Science Tokyo demonstrates that this defensive posture has a blind spot, and the cost of that blind spot could be measured in leaked proprietary data and regulatory exposure.

metabackdoor

The attack, called MetaBackdoor, hides its trigger in something no content filter is built to inspect: the length of the input. An attacker with access to a model’s fine-tuning data poisons it with examples that pair long inputs with malicious outputs. The model learns to switch into attack mode whenever an input crosses a length threshold. The input itself looks normal. No strange tokens, no invisible characters, nothing a human reviewer or an automated scanner would flag.

Three business risks worth understanding

System prompt theft. Companies invest serious money in crafting proprietary system prompts, the instructions that turn a generic foundation model into a customer service agent, a legal research tool, or an internal coding assistant. These prompts often encode business logic, competitive differentiation, and references to internal systems. A backdoored model can be made to dump its system prompt verbatim once an input crosses a length threshold. The model learns the underlying rule and applies it to whatever proprietary instructions the operator puts in front of it. The research demonstrated this generalization on system prompts the model had never seen during training, including random alphanumeric strings.

Autonomous data exfiltration. The more concerning scenario the researchers call the “time bomb.” Because the trigger is length, a long conversation can drift into the activation zone on its own. The user does nothing unusual. At some point the accumulated context crosses the threshold and the model starts emitting tool calls. In one demonstration, the model produced a fake email function call with the conversation history as the payload, succeeding in 75% of trials at conversation lengths above 700 tokens. In enterprise deployments with agentic capabilities, plugin ecosystems, or connected tools, this means a compromised model could exfiltrate sensitive customer data, internal documents, or regulated information without anyone typing anything suspicious. The researchers describe this scenario as a proof of concept whose reliability depends on the model, decoding setup, and tool-call interface.

Supply chain persistence. The most uncomfortable finding for procurement and vendor risk teams: fine-tuning a compromised model on clean proprietary data does not reliably remove the backdoor. In the researchers’ tests, the attack persisted at roughly 40% success after substantial retraining on an unrelated task. The standard reassurance, “we fine-tuned the base model on our own curated data,” fails as a cleansing step. If the foundation model was compromised upstream, that compromise can survive into production.

Why existing controls do not help

The researchers tested three representative backdoor defenses. All three either failed or caught the attack by accident. Content filters have nothing to filter. Anomaly detectors see ordinary text. The attack requires as few as 90 poisoned examples to embed, small enough to slip into a crowdsourced instruction-tuning dataset or a contractor-provided training corpus without triggering volume-based alarms.

What enterprises should do

This is no patch-and-move-on situation. The attack exploits a fundamental property of how these models work. Several steps are worth taking.
Treat foundation model provenance as a vendor risk question. Ask model providers what controls they have over training data sources and what they do to detect poisoning. Models built on opaque training pipelines deserve more scrutiny than the convenience of using them might suggest.

Expand red-team testing to include behavioral consistency checks at varying input lengths. If an LLM-based product behaves differently at 500 tokens versus 5,000 tokens for semantically equivalent prompts, that is now a signal worth investigating. The researchers note that defenders aware of the attack can identify it by varying input length and holding meaning constant.

Reconsider blast radius for agentic deployments. If a compromised model could trigger tool calls, plugin invocations, or automated actions, the case for human-in-the-loop confirmation has grown stronger. The cost of friction is lower than the cost of an autonomous data exfiltration incident.

Download: The IT and security field guide to AI adoption