Abstract:Efficient CUDA implementations of attention mechanisms are critical to modern deep learning systems, yet supporting diverse and evolving attention variants remains challenging. Existing frameworks and compilers trade performance for flexibility, while expert-written kernels achieve high efficiency but are difficult to adapt. Recent work explores large language models (LLMs) for GPU kernel generation, but prior studies report unstable correctness and significant performance gaps for complex operators such as attention.
We present CuBridge, an LLM-based framework that adapts expert-written attention kernels through a structured lift-transfer-lower workflow. CuBridge starts from expert-written CUDA attention kernels and lifts them into an executable intermediate representation that makes execution orchestration explicit while abstracting low-level CUDA syntax. Given a user-provided PyTorch specification, CuBridge generates and verifies a target IR program, then reconstructs optimized CUDA code via reference-guided lowering. Across diverse attention variants and GPU platforms, CuBridge consistently produces correct kernels and substantially outperforms general frameworks, compiler-based approaches, and prior LLM-based methods.
| Comments: | Accepted to ACL 2026 |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.05023 [cs.LG] |
| (or arXiv:2605.05023v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.05023 arXiv-issued DOI via DataCite (pending registration) |
From: Xing Ma [view email]
[v1]
Wed, 6 May 2026 15:19:07 UTC (356 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。