Since its initial release in 2022, Text-Generation-Inference (TGI) has provided Hugging Face and the AI Community with a performance-focused solution to easily deploy large-language models (LLMs). TGI initially offered an almost no-code solution to load models from the Hugging Face Hub and deploy them in production on NVIDIA GPUs. Over time, support expanded to include AMD Instinct GPUs, Intel GPUs, AWS Trainium/Inferentia, Google TPU, and Intel Gaudi.
Over the years, multiple inferencing solutions have emerged, including vLLM, SGLang, llama.cpp, TensorRT-LLM, etc., splitting up the overall ecosystem. Different models, hardware, and use cases may require a specific backend to achieve optimal performance. However, configuring each backend correctly, managing licenses, and integrating them into existing infrastructure can be challenging for users.
To address this, we are excited to introduce the concept of TGI Backends. This new architecture gives the flexibility to integrate with any of the solutions above through TGI as a single unified frontend layer. This change makes it easier for the community to get the best performance for their production workloads, switching backends according to their modeling, hardware, and performance requirements.
The Hugging Face team is excited to contribute to and collaborate with the teams that build vLLM, llama.cpp, TensorRT-LLM, and the teams at AWS, Google, NVIDIA, AMD and Intel to offer a robust and consistent user experience for TGI users whichever backend and hardware they want to use.
TGI is made of multiple components, primarily written in Rust and Python. Rust powers the HTTP and scheduling layers, and Python remains the go-to for modeling.
Long story short: Rust allows us to improve the overall robustness of the serving layer with static analysis and compiler-based memory safety enforcement: it brings the ability to scale to multiple cores with the same safety guarantees more easily. Leveraging Rust’s strong type system for the HTTP layer and scheduler makes it possible to avoid memory issues while maximizing the concurrency, bypassing Global Interpreter Lock (GIL) in Python-based environments.
Speaking about Rust… Surprise, that's the TGI starting point to integrate a new backend - 🤗
Earlier this year, the TGI team worked on exposing the foundational knobs to disentangle how the actual HTTP server and the scheduler were coupled together.
This work introduced the new Rust trait Backend to interface current inference engine and the one to come.
Having this new Backend interface (or trait in Rusty terms) paves the way for modularity and makes it possible to actually route the incoming requests towards different modeling and execution engines.
The new multi-backend capabilities of TGI open up many impactful roadmap opportunities. As we look ahead to 2025 we are excited to share some of the TGI developments we are most excited about:
optimum-nvidia quantize/build/evaluate TensorRT compatible artifacts alongside TGI+TRT-LLM to easily deploy, execute, and scale deployments on NVIDIA GPUs.We are confident TGI Backends will help simplify the deployments of LLMs, bringing versatility and performance to all TGI users. You'll soon be able to use TGI Backends directly within Inference Endpoints. Customers will be able to easily deploy models with TGI Backends on various hardware with top-tier performance and reliability out of the box.
Stay tuned for the next blog post where we'll dig into technical details and performance benchmarks of upcoming backends!
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。