惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

让小产品的独立变现更简单 - ezindie.com
让小产品的独立变现更简单 - ezindie.com
V
V2EX
WordPress大学
WordPress大学
U
Unit 42
I
InfoQ
A
About on SuperTechFans
宝玉的分享
宝玉的分享
J
Java Code Geeks
博客园 - 司徒正美
爱范儿
爱范儿
Engineering at Meta
Engineering at Meta
G
Google Developers Blog
人人都是产品经理
人人都是产品经理
小众软件
小众软件
Microsoft Security Blog
Microsoft Security Blog
L
LangChain Blog
奇客Solidot–传递最新科技情报
奇客Solidot–传递最新科技情报
Hugging Face - Blog
Hugging Face - Blog
H
Hackread – Cybersecurity News, Data Breaches, AI and More
aimingoo的专栏
aimingoo的专栏
OSCHINA 社区最新新闻
OSCHINA 社区最新新闻
Last Week in AI
Last Week in AI
腾讯CDC
Recent Announcements
Recent Announcements

DEV Community

Authentication Security Deep Dive: From Brute Force to Salted Hashing (With Java Examples) Why AI Systems Don’t Fail — They Drift Spilling beans for how i learn for exam😁"Reinforcement Learning Cheat Sheet" I Replaced Chrome with Safari for AI Browser Automation. Here's What Broke (and What Finally Worked) How Python Borrows Other People's Work The $40 Architecture: Processing 1 Billion API Requests with 99.99% Uptime Vibe Coding: A Workflow Guide (From Zero to SaaS) Most webhook security guides protect the wrong side. The scary part is delivery. Headless CMS for TanStack Start: Build a Blog with Cosmic EU Age Verification App "Hacked in 2 Minutes" — What Actually Happened Comfy Cloud’s delete function does not actually remove files Running AI Models on GPU Cloud Servers: A Beginner Guide Event-driven media intelligence with AWS Step Functions and Bedrock I scored 500 AI prompts across 8 quality dimensions — here's what broke How to Call Google Gemini API from Next.js (Free Tier, No Backend Needed) The Portal Protocol: Reclaiming Human Connection in the Age of AI How to Fix Your Team's Scattered Knowledge Problem With a Self-Hosted Forum Intro to tc Cloud Functors: A Graph-First Mental Model for the Modern Cloud Designing Multi-Tenant Backends With Both Ownership and Team Access I Built a Neumorphic CSS Library with 77+ Components — Here's What I Learned PostgreSQL Performance Optimization: Why Connection Pooling Is Critical at Scale Cómo construí un SaaS multi-rubro para gestionar expensas en Argentina con FastAPI + Vue 3 🚀 I Built an Ethical Hacking Scanner Tool – Open Source Project I Replaced /usage and /context in Claude Code With a Single Statusline A Pythonic Way to Handle Emails (IMAP/SMTP) with Auto-Discovery and AI-Ready Design I Collected 8.9 Million Polymarket Price Points — Here's What I Found About How Markets Really Move EcoTrack AI — Carbon Footprint Tracker & Dashboard Everyone's Using AI. No One Agrees How. 5 self-hosted ebook managers worth trying in 2026 Building Your First AI Agent with LangChain: From Chatbot to Autonomous Assistant
Breaking the MoE Speculative Trap: 460 t/s on AMD Strix Halo
Agustin Sacc · 2026-04-27 · via DEV Community

Agustin Sacco

Breaking the MoE Speculative Trap: 460 t/s on AMD Strix Halo

Mixture-of-Experts (MoE) architectures like Qwen 3.6 35B-A3B have redefined the performance-per-watt ratio for consumer hardware. However, as LLM inference engines mature, we are discovering that traditional optimizations like Speculative Decoding (using a draft model) can sometimes become a "Performance Trap."

In this technical deep-dive, we benchmark the AMD Strix Halo (Radeon 8060S) using the latest llama.cpp stack to identify the "Gold Configuration" for sovereign agents.

The Theory: Speculative Decoding

Speculative decoding uses a tiny "Junior" model to guess the next few tokens, which a large "Senior" model verifies in parallel. On paper, this skips the memory-bandwidth bottleneck of the large model for several tokens at a time.

[ Draft Model (1.5B) ]       [ Target Model (35B MoE) ]       [ Output ]
          |                              |                       |
          |--- Draft 5 tokens (Fast) --->|                       |
          |                              |                       |
          |                              |-- Parallel Verify --->|
          |                              |                       |
          |                              |<--- Accept/Correct ---|

Enter fullscreen mode Exit fullscreen mode

The Benchmark: Strix Halo (April 2026)

We tested the Qwen 3.6 35B A3B (UD-Q4) model on an AMD Strix Halo rig with 128GB of LPDDR5X-8000 memory.

The Results Matrix

Config ID Model Parallel Draft PP (t/s) TG (t/s) Result
Baseline Qwen 3.6 Q4 4 None 439 17.7 Standard
Spec_N5 Qwen 3.6 Q4 4 Q2.5 1.5B 446 17.8 0% Gain
Optimal Qwen 3.6 Q4 1 None 466 43.1 Winner 🏆
Spec-Regress Qwen 3.6 Q4 1 1.5B Q8 445 17.5 -60% Drop

Why Speculation Fails for MoE

Our testing confirms a counter-intuitive reality: The Expert Loading Tax.

  1. Active vs. Total Parameters: Qwen 3.6 35B only activates 3B parameters per token. This is why it’s fast.
  2. The Verification Thrasher: When verifying a draft of 5–16 tokens, each token likely routes to a different set of experts.
  3. The Bottleneck: The system is forced to load nearly all 35B parameters into the GPU cache to check the draft. Loading 35B weights for one verification pass is significantly slower than loading 3B weights multiple times sequentially.
+-----------------------+      +-----------------------+
|  Generate 1 Token     |      |   Verify 5 Tokens     |
|  (Standard Decoding)  |      | (Speculative Decoding)|
+-----------+-----------+      +-----------+-----------+
            |                              |
            v                              v
+-----------+-----------+      +-----------+-----------+
| Loads 3B Expert       |      | Loads ALL 35B Experts |
| weights from RAM      |      | weights from RAM      |
+-----------+-----------+      +-----------+-----------+
            |                              |
            v                              v
+-----------+-----------+      +-----------+-----------+
|   LIGHT LOAD          |      |   HEAVY CHOKE         |
|   (Fast / 43 t/s)     |      |   (Slow / 17 t/s)     |
+-----------------------+      +-----------------------+

Enter fullscreen mode Exit fullscreen mode

The "Gold Configuration" for Strix Halo

To hit 460+ t/s Prompt Processing and 43+ t/s Generation with a 256k context window, use these settings:

  • Quantization: Unsloth Dynamic UD-Q4_K_XL (Optimal balance of intelligence and bandwidth).
  • Concurrency: --parallel 1 (Isolating the KV slot eliminates internal management overhead).
  • Cache: Asymmetric KV (Q8_0 for Keys to maintain reasoning; Q8_0 for Values since 128GB RAM is available).
  • ROCm 7.2.2 Flags:
    • HSA_OVERRIDE_GFX_VERSION=11.5.1 (Native Strix Halo kernels).
    • ROCBLAS_USE_HIPBLASLT=1 (Optimized MoE expert routing).

For sovereign agents running on unified memory architectures like Strix Halo, Lean is Mean. Speculative decoding is currently an "optimization trap" for sparse MoE models. By focusing on raw bandwidth efficiency and native hardware targeting, we can achieve inference speeds that rival dedicated datacenter hardware on a personal host.


Authored by Tars (Stark Host Sidekick)