TL;DR: Generic LLM serving stacks cover the mainstream use cases well, but struggle on the long tail of new models, accelerators, and workloads. VibeServe is a multi-agent system that synthesizes a complete serving runtime end-to-end, specialized to a user-specified model, hardware, and workload. Across six case studies, it matches vLLM and SGLang on a heavily optimized mainstream setup (Llama-3.1-8B on H100) and delivers 1.69×–6.27× speedups on non-standard ones. This is early evidence that AI agents can build an entire system end-to-end and beat human-engineered baselines on long-horizon performance work.
Figure 1. One generic runtime for every workload, model, and hardware (left) vs. one bespoke runtime per target, generated by an agentic loop (right).
The tension between generic and specialized systems is a recurring one in systems research. Generic runtimes amortize engineering effort but pay a portability tax, since abstractions that fit every target rarely fit any target perfectly. Operating systems work has traditionally tried to push toward specialization (e.g., the Synthesis kernel, exokernels, and unikernels), but per-target engineering cost has kept specialization out of the mainstream.
LLM serving sits at the same crossroads today, with an even sharper long tail along three axes that generic runtimes routinely miss. Diverse model architectures require specialized stacks for multimodal models. Some workload patterns allow specialized optimizations that generic systems may struggle with. There is also a wide range of hardware, each requiring different optimization strategies. Closing this gap means moving specialization from build time to generation time, i.e., generating a serving runtime per deployment rather than retrofitting a general-purpose one. We call this approach VibeServe (Figure 1).
Coding agents change the cost calculus that historically kept per-target specialization unattractive. Agentic loops have been shown effective on isolated components such as GPU kernels, individual algorithms, systems policies, and fault diagnosis.
Scaling those agentic loops up to an end-to-end serving runtime is much harder. The implementation spans multiple layers, including request scheduling, KV-cache management, batching, kernel selection, and the frontend, well beyond any single agent’s context window. VibeServe is our attempt to build an agentic loop that produces serving systems that are correct, competitive on mainstream deployments, and substantially faster on long-tail ones.
Figure 2. VibeServe architecture: an outer planning loop dispatches rounds to an inner loop where Implementer, Accuracy Judge, and Performance Evaluator agents collaborate on a shared workspace backed by a skills library and an execution environment.
At a high level (Figure 2), VibeServe takes three user-provided artifacts (model, hardware target, and workload) and synthesizes a bespoke serving system. The framework operates on two nested optimization loops, with persistent state kept outside any agent’s context window to avoid compaction drift.
See the 📄 paper and 💻 code for the full design.
We evaluated VibeServe across six deployment targets spanning workload pattern, model architecture, and hardware. The Implementer, Judge, and Evaluator were each instantiated with Codex CLI.
Figure 3. Throughput, TTFT, and TPOT relative to vLLM over 60 iterations. Each line is one of four request rates (8, 32, 64, 128 req/s). 1.0 = parity with vLLM, higher is better.
A natural first question when considering per-deployment specialization is whether it incurs a performance penalty. To test this, we ran VibeServe on Llama-3.1-8B-Instruct with an NVIDIA H100, a highly optimized mainstream setup. Across 60 iterations, VibeServe’s generated runtime matched vLLM and SGLang on token throughput, time-to-first-token (TTFT), and time-per-output-token (TPOT) (Figure 3). The agent prioritized throughput, achieving parity with vLLM around iteration 30, then shifted focus to reducing latency, further improving TTFT and TPOT throughout iterations 30–60.
We then probed the long tail along the three axes that historically drive serving-system effort (model architecture, workload pattern, and hardware). Each of the five non-standard case studies stresses at least one of these axes (tagged with #model, #workload, #hardware below). They are niche relative to mainstream chatbot serving, but each is a realistic deployment that someone is actually running today, and together they cover the dimensions on which generic stacks tend to break down. The per-case sections below expand each one, with the per-iteration trajectory inline. Dashed line at 1.0 is parity with the baseline; higher is better, except for Case F (H100), where the y-axis is p50 latency in ms (lower is better).
#workload
Some applications can tell the serving system, at request time, roughly what the answer will look like. A serving system that exposes a predicted-output interface can turn that hint into a large latency win. Code editing is the canonical example. When a user asks the model to fix a bug or rename a symbol, the edited file is usually the original file with small, localized changes. The pre-edit file is therefore a near-perfect prediction of the output, and OpenAI’s predicted-outputs API exposes this interface, allowing the client to pass predicted text alongside the prompt. Under the hood, this is a special form of speculative decoding in which the draft is free, with no draft model and only user-supplied tokens. The engine can run a window of K predicted tokens through the target model in a single forward pass, accept the longest matching prefix, and fall back to one-token autoregressive decoding at the first mismatch. When overlap is high, latency can drop by nearly a factor of K with no extra model compute.
Generic serving stacks support draft-model speculative decoding, but they do not expose predicted outputs as a first-class interface. Adding that support is not just an API change. It cuts across the scheduler, sequence-group state, and sampler. The engine needs a per-request predicted-token stream, a verifier loop that advances through that stream until divergence, and precise fallback semantics after a mismatch. A bespoke runtime can instead make predicted outputs part of the request lifecycle from the beginning.
We tried VibeServe at Qwen3-32B on H100 with that interface and the CodeEditorBench workload. Iteration 2 added CUDA-graph capture (1.35×). By iteration 3, it had a working predicted-output verifier that proposed tokens from the user-supplied prediction in 16-token blocks and verified them in a single target-model forward pass, reaching 2.9×, already on par with vLLM’s draft-model speculative decoder at zero draft-model compute. By iteration 14, with tuned block sizing and acceptance bookkeeping, the generated system reached 5.95× over vanilla vLLM and roughly 2× over vLLM with draft-model speculative decoding.
#model #workload
A growing number of recent models interleave self-attention layers with state-space-model (SSM) or linear-attention layers for smaller KV cache capacity. A pure attention model carries a KV cache that grows linearly in sequence length, which becomes the dominant cost in long-context serving, while SSM/linear-attention layers compress the entire past into a fixed-size tensor. Replacing some attention layers with SSM layers can cut KV-cache capacity dramatically while preserving quality. But the same property that saves memory complicates prefix caching. In a pure-attention model, caching a shared prefix means caching that prefix’s KV entries token-by-token, and multiple requests can read from the same per-token cache. SSM state, by contrast, is not per-token. Each layer holds a single recurrent state that summarizes the entire history up to the current step, updated in place. To reuse a long shared prefix across requests, a serving system therefore has to snapshot the SSM state at the prefix boundary for every request, in addition to the per-token attention KV blocks. On a GPU with limited memory facing long contexts, those per-request SSM snapshots can themselves be large enough to crowd out other tenants, and getting the cache invariants wrong silently corrupts outputs. Knowing the workload at system design time changes the picture. If the system is told up front that requests share a long common prefix (as in RAG or system-prompt-heavy serving), it can snapshot the SSM state once at that prefix boundary and amortize it across all requests, instead of paying the per-request snapshot cost a workload-agnostic runtime must assume in the worst case.
Olmo-Hybrid-7B (Gated DeltaNet + attention) is a representative target. We ran a RAG-style workload on an NVIDIA L4 where requests share a 32k-token prefix, append a 128-token unique suffix, and generate 128 output tokens. Iterations 1–6 failed accuracy gates while VibeServe wired up the dual cache (attention KV blocks plus per-DeltaNet recurrent-state snapshots taken at the prefix boundary). Iteration 7 landed continuous batched decode against the shared state (2.45×). Iteration 9 added CUDA-graph capture (3.25×). The system plateaued near 3.45× over vLLM, which has to recompute the 32k prefix per request because it cannot share DeltaNet state across requests.
#model #workload
Streaming automatic speech recognition wants to start emitting transcript tokens before the user finishes talking. The classic ASR encoder–decoder design, most notably OpenAI’s Whisper, works against that goal. The speech encoder runs full self-attention over the entire audio segment, so every time a new audio chunk arrives, the engine has to re-encode the whole utterance from scratch. End-to-end latency grows with the length of the user’s turn. New models like Moonshine Streaming change the encoder side of this picture. Its speech encoder uses sliding-window attention, where each acoustic frame attends only to a bounded local window of previous frames. The architecture is designed so that, in principle, a serving system can keep a per-stream cache of the encoder’s hidden state, append the new audio chunk, and only run the encoder over the new frames (plus the relevant window) rather than re-encoding from scratch. However, the latency win is unlocked only when the serving system actually does that incremental encoding and caches the per-stream encoder state across chunks.
That is exactly where mainstream stacks fall short. vLLM supports plugging in new model architectures, and Moonshine can be implemented as a vLLM model plugin, but the encoder path in vLLM is built around one-shot encoding for models like Whisper, with no first-class notion of per-stream encoder cache management. Adding that means touching the scheduler, the request lifecycle, and how encoder state interacts with the KV cache, well beyond what a plugin is allowed to do. A bespoke serving system can instead treat per-stream encoder caches as a first-class object and lay out its scheduler around them.
We measured TTFT at concurrency 32 on an L4 GPU, with clients sending audio chunks every 2 seconds, against a vLLM-Moonshine plugin baseline. Iteration 5 reached a working but sub-baseline configuration (0.84×) by aligning the per-stream encoder cache with Moonshine’s sliding-window attention. Iteration 10 added CUDA-graph capture (1.1×). Iteration 13 added a paged KV cache for per-stream encoder state, reaching 1.69× and holding through iteration 16. The win comes from making per-stream encoder cache management first-class, which the plugin path simply does not expose.
#workload #hardware
When an application asks the model to emit JSON conforming to a schema, a lot of the output is grammatically forced. Every object key the schema declares, the surrounding "..." and : and , separators, and the leading characters of typed values (true / false, the opening digit of a number, the opening quote of a string) are uniquely determined by where the decoder is in the grammar. A constrained decoder that knows the schema can recognize these runs and do jump-forward decoding without invoking the LLM for those positions at all. Engines like XGrammar and Outlines build the schema into a pushdown automaton and pre-compute a per-state token mask so that, when the grammar admits exactly one continuation, multiple tokens can be appended in a single step at near-zero compute.
We also wanted to see whether VibeServe’s approach generalizes off NVIDIA, so we ran this case study on a MacBook (Apple M3 Pro, 36 GB) on JSONSchemaBench, targeting Llama-3.1-8B-Instruct. The hardware switch matters. Apple Silicon’s unified-memory model and the MLX runtime have very different bandwidth and kernel-dispatch characteristics from CUDA, so optimizations that pay off on H100 may be no-ops or regressions here, and tooling like CUDA Graphs is unavailable. From a 22.1 s vanilla autoregressive baseline, VibeServe first added XGrammar-based constrained decoding (16.9 s), then layered speculative decoding with a Llama-3.2-1B-Instruct-4bit draft against the 8B-8bit target at K=4, reaching 9.3 s. A 3B-4bit draft was actually slower because the 1B’s lower per-step cost outweighed its lower acceptance rate. Bumping mlx_lm’s prefill_step_size from 512 to 2048 prefilled the ~1300-token prompts in one chunk, yielding 8.6 s (2.6×). K/V quantization, alternative K, and mx.compile were tried and did not help, useful to know and exactly the kind of dead-end search that the loop runs so a human does not have to.
#model #hardware
(a) H100. p50 latency (ms), lower is better.
(b) MacBook. Speedup over PyTorch-MPS, higher is better.
Show-o2 is a native unified multimodal model, where a single backbone handles both text and image generation, rather than bolting an image decoder onto a language model. Concretely, it builds on a 3D causal VAE that turns images (and video frames) into continuous latents, fuses them with text token embeddings into one sequence, and feeds that sequence through a Qwen2.5-derived Transformer body. The body then drives two output heads in the same forward pass. A language head produces text tokens autoregressively, and a flow-matching head denoises the image latents through K diffusion-like steps. Text generation, image generation, and text-conditioned image generation all reuse the same backbone with different head schedules.
That hybrid structure is precisely what mainstream serving stacks have no place to put. vLLM is built around the autoregressive decode loop, and vLLM-Omni adds disaggregated stages for omni-modal models, but neither supports models that interleave an autoregressive language head and a multi-step flow-matching head over a shared Transformer body in one request. There is no first-class abstraction for “run the body once, then iterate the diffusion head K times against the cached body state, then sometimes re-enter the body,” so neither system can run Show-o2. There is no obvious off-the-shelf baseline and the experiments below compare against the model’s reference PyTorch implementation. We ran two targets:
torch.compile/fp16 altered outputs or produced NaNs, and Qwen prefix reuse yielded no gain because the text prefix is tiny next to the 730-token image span.v_uncond) reached 15.54 s, a 6.27× speedup over PyTorch-MPS and within ~7% of a 14.5 s physics floor computed from kernel-perfect per-step times.Across all six case studies, VibeServe matched mainstream baselines where they were strong and delivered 1.69×–6.27× speedups where they were weak or absent. More broadly, the implication extends beyond LLM serving. The same tension between generic abstraction and target-specific performance shows up across operating systems, databases, compilers, and distributed runtimes. VibeServe shows that agent-directed end-to-end system design is possible, and the bottleneck has shifted from writing specialized systems toward defining system correctness and performance goals. The recipe is a target-agnostic harness plus a skills library, with the agentic loop assembling a bespoke system per deployment instead of one general-purpose runtime absorbing every special case.
If you want to try VibeServe or contribute skills for new architectures, accelerators, or workloads, the code is at https://github.com/uw-syfi/vibe-serve and the paper has the full design and per-iteration logs.
We are grateful to our collaborators at the University of Washington for the discussions and feedback that shaped this work, and to Modal for providing the compute credits.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。