惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

N
Netflix TechBlog - Medium
G
Google Developers Blog
H
Hackread – Cybersecurity News, Data Breaches, AI and More
T
The Blog of Author Tim Ferriss
Microsoft Azure Blog
Microsoft Azure Blog
GbyAI
GbyAI
L
LangChain Blog
云风的 BLOG
云风的 BLOG
OSCHINA 社区最新新闻
OSCHINA 社区最新新闻
aimingoo的专栏
aimingoo的专栏
P
Proofpoint News Feed
Cyber Security Advisories - MS-ISAC
Cyber Security Advisories - MS-ISAC
小众软件
小众软件
WordPress大学
WordPress大学
A
About on SuperTechFans
大猫的无限游戏
大猫的无限游戏
C
Check Point Blog
月光博客
月光博客
Stack Overflow Blog
Stack Overflow Blog
美团技术团队
Jina AI
Jina AI
T
Tailwind CSS Blog
Google DeepMind News
Google DeepMind News
D
Docker

Hacker News: Best

madhadron - The seven programming ur-languages GitHub - smol-machines/smolvm: Tool to build & run portable, lightweight, self-contained virtual machines. I Measured Claude 4.7's New Tokenizer. Here's What It Costs You. Introducing Claude Design by Anthropic Labs It Is Time to Ban the Sale of Precise Geolocation The creative software industry has declared war on Adobe Isaac Asimov: The Last Question Newly unsealed records reveal Amazon’s price-fixing tactics, California attorney general claims Clojure - Documentary Android CLI and skills: Build Android apps 3x faster using any agent Qwen3.6-35B-A3B on my laptop drew me a better pelican than Claude Opus 4.7 Codex:全能型助手 Introducing Claude Opus 4.7 Qwen Studio The Future of Everything is Lies, I Guess: Where Do We Go From Here? Virginia Bans Sale of Geolocation Data YouTube now lets you turn off Shorts Burgers | マクドナルド公式 ChatGPT for Excel Ask HN: Who is using OpenClaw? Live Nation illegally monopolized ticketing market, jury finds Google Broke Its Promise to Me. Now ICE Has My Data. Open Source Isn't Dead. The Future of Everything is Lies, I Guess: New Jobs Unexpected €54k billing spike in 13 hours: Firebase browser key without API restrictions used for Gemini requests IPv6 – Google Your Backpack Got Worse On Purpose Good sleep, good learning, good life Fixing a 20-year-old bug in Enlightenment E16. Does Gas Town 'steal' usage from users' LLM credits & paid services to improve itself?
Do Transformers Need Three Projections? Systematic Study ...
Ali Kayyam, Anusha Madan Gopal, M Anthony Lewis · 2026-06-02 · via Hacker News: Best

Transformers have become the standard solution for various AI tasks, with the query, key, and value (QKV) attention formulation playing a central role. However, the individual contribution of these three projections and the impact of omitting some remain poorly understood. We systematically evaluate three projection sharing constraints: a) Q-K=V (shared key-value), b) Q=K-V (shared query-key), and c) Q=K=V (single projection). The last two variants produce symmetric attention maps; to address this, we also explore asymmetric attention via 2D positional encodings. Through experiments spanning synthetic tasks, vision (MNIST, CIFAR, TinyImageNet, anomaly), and language modeling (300M and 1.2B parameter models on 10B tokens), we discovered that our transformers perform on par or occasionally better than the QKV transformer. In language modeling, Q-K=V projection sharing achieves 50% KV cache reduction with only 3.1% perplexity degradation. Crucially, projection sharing is complementary to head sharing (GQA/MQA): combining Q-K=V with GQA-4 yields 87.5% cache reduction, while Q-K=V + MQA achieves 96.9%, enabling practical on-device inference. We show that Q-K=V preserves quality because keys and values can occupy similar representational spaces and attention operates in a low-rank regime, whereas Q=K-V breaks attention directionality. Our results systematically characterize projection sharing as an underexplored instance of weight tying in attention, with direct, quantifiable inference memory benefits, particularly valuable for edge deployment. The code is publicly available at https://github.com/Brainchip-Inc/Do-Transformers-Need-3-Projections