Abstract:Serial LLM inference backends process requests sequentially under First-Come-First-Served (FCFS) admission, causing Head-of-Line Blocking (HOLB) under mixed workloads: short factual queries can be delayed by minutes behind long generation jobs. While cloud-scale deployments mitigate HOLB via continuous batching (e.g., vLLM, Orca), these solutions require tens of gigabytes of VRAM for concurrent KV-caches, rendering them infeasible for memory-constrained edge and local deployments that rely on serial request dispatch. We present Clairvoyant, a drop-in sidecar proxy for serial OpenAI-compatible backends (e.g., Ollama, this http URL) that implements predictive Shortest-Job-First (SJF) admission. Clairvoyant predicts response length using 19 lightweight lexical features via an ONNX-exported XGBoost classifier, achieving 0.029 ms per-request latency. Because admission scheduling relies on relative ranking rather than exact token prediction, Clairvoyant captures over 95% of the ranking fidelity of fine-tuned transformers at a fraction of the computational cost. We also uncover a critical dataset bias: curated instruction datasets are degenerate training sources for length prediction, as GPT-imposed brevity constraints reduce Long-class representation to under 0.02% of examples, establishing natural conversation logs as the only viable training signal. End-to-end evaluations demonstrate substantial latency reductions across diverse hardware regimes: a 70-76% short-request P50 latency reduction on an RTX 4090, a 69.7% reduction on Apple M1 edge hardware, and an 83.6% reduction in Time-To-First-Token (TTFT) on a GCP NVIDIA L4 real-world trace replay (rho = 0.80). Clairvoyant is open-source, requires zero modifications to the inference backend, and provides a low-overhead mechanism to eliminate HOLB in edge LLM environments.
From: Aravind Sundaresan [view email]
[v1]
Fri, 5 Jun 2026 13:19:05 UTC (190 KB)
[v2]
Thu, 30 Jul 2026 08:24:42 UTC (221 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。