llm-d is an open source solution for managing high-scale, high-performance Large Language Model (LLM) deployments. LLMs are at the heart of generative AI – so when you chat with ChatGPT or Gemini, you’re talking to an LLM.
Simple LLM deployments – where an LLM is deployed to a single server – can suffer from latency issues, even with just one user. This can be because of lack of memory-bandwidth on the server, or because of KV cache pressure on system memory. This means that you’re kept waiting for the LLM to respond to your question or instruction, which can really drag, and nobody likes to be kept waiting.
Llm-d tries to solve this by splitting up the LLM deployment (disaggregating the deployment) and separating out different components of the architecture onto dedicated hardware. This means that the various parts of the system can be managed and scaled independently. Llm-d is a cloud-native system and uses Kubernetes as an orchestration engine for all of this, so that managing the necessary resources can be done automatically – using Kubernetes’ automation features.
For some organizations, sovereignty (that is, keeping things under your own control, governance and oversight) is imperative. That’s true for sensitive data, and also for sensitive data processing, like the things folks do with LLMs, such as building Retrieval Augmented Generative AI (RAG) systems or agentic workflows. So for those organizations, there’s no question that they’re going to want to run an LLM service under their own watch, on their own systems, even in their own data center. With open weight large language models like Kimi-2.5 and GM-5 that can hold their own against Gemini Pro, Claude Sonnet and Grok becoming available, it’s never been a better time to run a sovereign AI Factory. And that’s where llm-d comes into its own.
At a high level, llm-d is composed of four major components, all of which run on a Kubernetes cluster:
Llm-d is designed to work with the very high-performance hardware these setups need, like servers with enterprise-grade GPUs and Infiniband network switching – which is an alternative networking solution to the classic ethernet.
I find that the best way to learn about something more deeply is to work with it. So to that end, I put together some Juju charms for Ubuntu, to get a better understanding of how llm-d works for myself. They can enable you to deploy an LLM to an llm-d setup in a clean, straightforward way – without needing to be a Kubernetes guru.
I’ve made the source code available in GitHub, along with some instructions about how to build the code and get things up and running. Note that I’ve just been playing around with Juju charms when building these; there may be bugs and they are not supported by Canonical, so use them at your own risk.
The diagram below illustrates how the various Juju charms that manage the system are integrated.
Juju charms offer a clean approach to devops, and the system has primitives for both cloud infrastructure and Kubernetes. If you’d like to dig in and learn more about how to develop or use Juju charms, head over to our Juju page.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。