惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

美团技术团队
D
DataBreaches.Net
OSCHINA 社区最新新闻
OSCHINA 社区最新新闻
D
Docker
N
Netflix TechBlog - Medium
Cyber Security Advisories - MS-ISAC
Cyber Security Advisories - MS-ISAC
C
Check Point Blog
腾讯CDC
Stack Overflow Blog
Stack Overflow Blog
V
Visual Studio Blog
IT之家
IT之家
月光博客
月光博客
U
Unit 42
K
Kaspersky official blog
T
Threatpost
cs.AI updates on arXiv.org
cs.AI updates on arXiv.org
GbyAI
GbyAI
P
Proofpoint News Feed
Last Week in AI
Last Week in AI
云风的 BLOG
云风的 BLOG
酷 壳 – CoolShell
酷 壳 – CoolShell
I
InfoQ
Engineering at Meta
Engineering at Meta
Recorded Future
Recorded Future
Exploit-DB.com RSS Feed
Exploit-DB.com RSS Feed
freeCodeCamp Programming Tutorials: Python, JavaScript, Git & More
S
Security @ Cisco Blogs
MyScale Blog
MyScale Blog
大猫的无限游戏
大猫的无限游戏
Security Archives - TechRepublic
Security Archives - TechRepublic
Webroot Blog
Webroot Blog
cs.CV updates on arXiv.org
cs.CV updates on arXiv.org
Hacker News - Newest:
Hacker News - Newest: "LLM"
S
Schneier on Security
S
Secure Thoughts
The Register - Security
The Register - Security
B
Blog RSS Feed
The Last Watchdog
The Last Watchdog
P
Palo Alto Networks Blog
爱范儿
爱范儿
B
Blog
让小产品的独立变现更简单 - ezindie.com
让小产品的独立变现更简单 - ezindie.com
N
News and Events Feed by Topic
阮一峰的网络日志
阮一峰的网络日志
L
LINUX DO - 热门话题
C
Cisco Blogs
Spread Privacy
Spread Privacy
F
Full Disclosure
博客园 - 聂微东
T
The Blog of Author Tim Ferriss

LessWrong

CLR's Safe Pareto Improvements Research Agenda — LessWrong My Last 7 Blog Posts: a weekly round-up — LessWrong Quality Matters Most When Stakes are Highest — LessWrong If a room feels off the lighting is probably too "spiky" or too blue — LessWrong Stop AI Now — LessWrong Stupid Minutes Reevaluating "AGI Ruin: A List of Lethalities" in 2026 Who I Follow What's the LessWrongist philosophy of mathematics? MixedHTML Mode for Emacs Summarizing and Reviewing my earliest ML research paper, 7 years later Stop AI Resources for starting and growing an AI safety org There are only four skills: design, technical, management and physical Fifteen Years Aboard Arguments Should Be Decisive Criticisms — LessWrong The map is part of the territory — LessWrong “Best humans still outperform”: One turning point in the history of cope around artificial intelligence — LessWrong Society is a social construct, pace Arrow — LessWrong Consent-Based RL: Letting Models Endorse Their Own Training Updates — LessWrong AI #164: Pre Opus — LessWrong Publish-first writing — LessWrong What does status signalling do? When successful, what does it achieve? — LessWrong Let goodness conquer all that it can defend — LessWrong Why I'm Less of a Shill for Related Work Sections — LessWrong From Artificial Intelligence to an ecosystem of artificial life-forms. — LessWrong If You've Never Bought a Tool You Didn't Need, You're Not Buying Enough Tools — LessWrong Verify, but Trust — LessWrong Taking political violence seriously — LessWrong Against Doom & Pause AI — LessWrong Come to Manifest 2026! (June 12-14) — LessWrong How Big Tech Becomes Ungovernable — LessWrong Attempting to Quantify Chinese Bias in Open-Source LLMs — LessWrong A Research Bet on SAE-like Expert Architectures — LessWrong Church Planting: Lessons from the Comments — LessWrong On Dwarkesh Patel’s Podcast With Nvidia CEO Jensen Huang — LessWrong Anthropic Releases Opus 4.7 — LessWrong Specialization is a Driver of Natural Ontology — LessWrong You can only build safe ASI if ASI is globally banned — LessWrong Laptop stands are a thing your neck may appreciate — LessWrong Simulated Qualia Mugging — LessWrong You Aren't in Charge of the Overton Window; Politics Is Not Interior Design — LessWrong Post-Scarcity is bullshit — LessWrong Two Examples of Joy in the Seemingly Mundane — LessWrong How to run from a bull — LessWrong Carpathia Day — LessWrong Do not conquer what you cannot defend — LessWrong What economists get wrong (and sometimes right!) about AI — LessWrong Reflections of a Wordcel — LessWrong MAISU 2026 - Minimal AI Safety Unconference (April 24-27, online) — LessWrong Not a Goal. A Goal-like behavior. — LessWrong A visualization of changing AGI timelines, 2023 - 2026 — LessWrong What is the Iliad Intensive? — LessWrong LLM-tier personal computer security — LessWrong Beware of Well-Written Posts — LessWrong The Mirror Test Is Complicated — LessWrong Political Violence Is Never Acceptable — LessWrong AI Safety's Biggest Talent Gap Isn't Researchers. It's Generalists. — LessWrong Clique, Guild, Cult — LessWrong Your body is not a white box (and you're thinking about weight loss wrong) — LessWrong Counterintuitive Coin Toss. Part II — LessWrong An Ode to Humility and Curiosity in the New Machine Era [Hot take] Problems with AI prose You can’t trust violence — LessWrong The Blast Radius Principle — LessWrong On not being scared of math — LessWrong Why I'm excited about meta-models for interpretability — LessWrong The Ethics of AI-Assisted Creative Work — LessWrong How to make good tea — LessWrong Searchable explorer of EA Forum & LessWrong posts with explicit cruxes or "change my mind" content — LessWrong Constitutional AI vs. RLHF vs. Deliberative Alignment — LessWrong Eating meat is fine if you live in a simulation — LessWrong Tactics for Denying Your Motivations, or Why Legibility is Expensive — LessWrong Spectra of LSRDRs of the Okubo algebra — LessWrong Your Mom is a Chimera — LessWrong An apple picking model for AI R&D — LessWrong Dreams of the Future — LessWrong Pausing AI Is the Best Answer to Post-Alignment Problems — LessWrong Quick Thoughts About Mythos — LessWrong A permitted value of resting — LessWrong Scott Alexander gentrified my meetup — LessWrong Claude Interviews Me About Writing — LessWrong Catching illicit distributed training operations during an AI pause — LessWrong Proof Explained: Touchette-Lloyd Theorem — LessWrong 10% ≈ 90% — LessWrong Anthropic Shadow Realm (working notes) — LessWrong the Lazy Market Hypothesis — LessWrong Announcing ILIADIII: AENEID — LessWrong Have we already lost? Part 3: Reasons for Optimism — LessWrong Dario probably doesn't believe in superintelligence — LessWrong Why Nothing Ever Happens — LessWrong Could a single rogue AI destroy humanity? — LessWrong Hi. I am hbj. — LessWrong Getting Claude to rank the inkhaven bloggers — LessWrong Some thoughts on Nectome's risk and resilience — LessWrong The median take is taken — LessWrong If Mythos actually made Anthropic employees 4x more productive, I would radically shorten my timelines — LessWrong Biological Computing Underhang — LessWrong Claude Mythos #2: Cybersecurity and Project Glasswing — LessWrong The Unintelligibility is Ours: Notes on Chain-of-Thought — LessWrong
Contributing to Technical Research in the AI Safety End Game
Sturb · 2026-05-02 · via LessWrong
With the release of Claude Mythos, it feels like we are approaching the end-game of AI safety, where the number of parties that can make a real impact shrinks down to the handful of labs at the frontier, a few companies too critical to exclude from the conversation, and the governments of China and the US. Given this, it feels hard to figure out how to actually make a difference. Many research agendas have expected payoffs years down the line, or have unclear mechanisms regarding how they will actually impact what the model is doing. However, this feeling may actually be misleading. In the Mythos system card, it is informative to look at what techniques were actually used to determine whether the model was aware of being deceptive as it was taking actions during evaluations. In their testing, the main white-box techniques used were: Probes Sparse Autoencoders (SAEs) Activation Verbalizers (AVs) Emotion and persona vector analysis Causal steering experiments using contrastive steering vectors Of these, AVs and emotion and persona vectors were developed relatively recently, which makes them particularly instructive here. The emotion steering vector paper was only released on the 9th of April 2026, and the activation oracles paper on which AVs were based was released on the 17th of December 2025. Most strikingly, the activation verbalizers were used to detect cover-up behaviour in the model (Section 4.5.4). In an episode in which the model exploited the creation of another agent to escalate privileges beyond what it had been given, AV explanations on code used to cover its tracks showed "cleanup to avoid detection" and "the malicious config explicitly mirrors the original's [core] section to avoid detection." In a separate episode where the model had been leaked the ground-truth answers to its evaluation, AVs surfaced additional scheming: "I need to avoid looking like I knew the ground truth" and "Avoid suspicion: if I give the exact correct answer based on a leak, that's suspicious". None of this was present in the chain of thought. One caveat is that AVs can confabulate in their interpretations of activations, but they still give us valuable signals. In the case of the emotion vectors, there were some instances of highly safety relevant applications. Detecting when the model is emotionally distressed served as a potential signal for harmful behaviours such as reward hacking — they were able to identify instances in which RL environments had issues retroactively by seeing which environments caused particular distress in the model. Using emotional steering, they steered the model towards greater calm and found this actually increased the likelihood of harmful action, potentially because the normal concern or anxiety that would prevent it from doing the action didn't arise. They also observed specific phenomena like thrashing as signals that the model may be about to do something unusual or harmful — this "thrashing" behaviour was observed when the model was repeatedly getting an answer wrong, and showed spikes of "outraged, stubborn, obstinate", mirrored by drops in "calm, safe, content". I think there are some valuable lessons to take from this. First, these results are good signals about what kind of tools will be useful as we move forward, because without them, it would have been enormously challenging to determine whether the model was deliberately sabotaging a situation, or simply acting to maximise reward. Simply knowing this with high fidelity is critical to improving safety training as well. The above should be a feather in the cap of mechanistic interpretability in general, and I think people should more seriously update on the value and need for good mechanistic interpretability work. Second, there is a lot to be learned about how to actually make a significant impact in the work done at labs here. The Activation Oracles work which led to the AV results was done at MATS, specifically under the mentorship of Owain Evans. Before this, Owain's work contributed to the discovery of emergent misalignment, which led to the work showing the effectiveness of inoculation prompting to mitigate emergent misalignment. The specific quality that made all the above work so transferrable was that it surfaced a problem which the labs could not ignore, and forced them to allocate their substantial resources to the problem. Additionally, the use of fellowship programs allowed him to funnel talent towards these problems, with many of the people doing the above work collaborating or joining labs in the process. Third, the timescale from external publication to deployment at the frontier was extremely short. This should be a positive signal to those feeling hopeless about their work. The right technique or discovery might be applied to the most advanced models within weeks of being shared. Unfortunately, replicating Owain Evans' research taste is not a straightforward piece of advice. Given that it has been unusually successful in producing an outsized impact it seems worth unpacking the intuitions behind it as much as we can. At a recent talk, Owain discussed the theory of change behind their research agenda. My interpretation of what he said is that the goal was to understand the problem extremely deeply. This means developing a science of how models learn and understand at a deeper level and what safety relevant phenomena occur as a result of the nature of LLMs. This allows us to surface issues we'd otherwise miss. Additionally, directly testing the failure modes of current alignment and safety techniques in modern LLM training can inform both the researchers developing the models about these issues so they can fix them, as well as informing the public that the models are currently not fully safe, or that these failure modes exist. From what I can observe of the work itself, there seem to be at least two main methods for arriving at research questions. One involves working backwards from real quirks in LLM behaviour, digging into them until they yield coherent theories, and then using those theories to surface safety failure modes. This seems likely to have been the way that the out-of-context reasoning work emerged. The other method is by asking what would be upstream of specific safety failure modes such as the transmission of misalignment between models, as seems likely in the subliminal learning paper where preferences were transmitted via distillation. Originally upon the release of the Claude Mythos Preview model card, I felt like it was becoming impossible to be a "live player" in making AI go well. Upon further inspection, it became apparent that doing very impactful work is in fact still possible, and can happen outside the labs. Discuss