The latest release of qvac-fabric-llm.cpp, the inference engine of the QVAC Fabric LLM, features TurboQuant integration for resource management in long-running inference sessions. Tether adopts the technology as a path to better efficiency when running large

language models on devices with limited compute resources.

TurboQuant is Google’s response to the Key-Value (KV) Cache’s capacity expansion during routine inference, which can reach up to 8GB for a 262,000-token context session using a 4B-parameter large language model.

Tether takes the stage as the first AI research team to ship the KV Cache compression algorithm to a publicly available local AI model. The Turboquant integration will be included in the latest version of the QVAC SDK (v0.12.0) and will be available through Fabric, the inference and fine-tuning engine of the SDK.

This enables developers to serve intelligent models via the qvac-fabric-llm.cpp and compute inferences with negligible loss in precision, while consuming up to 5x less VRAM regardless of context size.

Why does this matter?

When you use an AI assistant, the model records the results from your previous prompts in temporary memory on your device. This record is known as the Key-Value Cache. The KV Cache is akin to a notepad for jotting down key points in a discussion or while reading a book.

For an AI model, it serves as a reference point each time you ask a follow-up question. This makes it easy for the model to follow up on your conversation without having to re-run the whole thread, which would waste a lot of time.

Transformer-based AI models build their KV Cache by storing the “key points” and their identifier, token-by-token (equivalent to “word for word”) in square grids. When you ask a follow-up question, the model sorts the key point using their precise location in the grid and computes new inferences based on your new input (prompt/question).

The KV Cache is a memory optimization technique and ensures a smooth run throughout each usage session. However, as you continue your conversation with the AI assistant, the KV Cache grows larger and consumes more memory on your device.

For instance, a few hours of conversation that grows into a 262,000-token session could consume up to 8GB of VRAM, which is hardly available in user-grade devices. Despite being temporary, KV cache overload could limit how an AI application can be used, especially when running models locally on devices with limited compute capacity, which is the case for the majority of users. KV Cache bloat is a major bottleneck for local AI, pushing users to cloud-based AI.

As a solution, TurboQuant strategically reduces KV Cache memory bloat by converting high-precision data vectors into lower-bit integers. Similar to reducing the size of the handwriting on the notepad or using signs instead of plain words, it shrinks the space the KV cache occupies.

How Turboquant compresses KV Cache memory

TurboQuant drastically reduces the memory consumed per cached token by using Polar quantization (PolarQuant) and Quantized Johnson-Lindenstrauss (QJL) techniques to bypass traditional quantization methods that require storing full-precision constants for small blocks of data.

It pairs PolarQuant’s structural efficiency with QJL’s zero-overhead error correction to compress caches up to 3 bits, delivering up to 5x improvement in memory management.

PolarQuant is what reduces the handwriting on the notepad or converts it into signs that consume less memory. It maps the KV Cache data onto a fixed circular grid and uses polar coordinates rather than the standard Cartesian (X, Y, Z) coordinates to locate key points.

This limits the details required to locate data to Angle (meaning of the data) and Radius (weight or importance of the data), rather than the full locational layout. It avoids the expensive data-normalization steps by replacing square grids with circular grids, simplifying vector representation and data location. This is similar to rewriting “Add 7 apples, then add 4 apples” as “Add 11 apples total.”

When compressing KV Cache data with PolarQuant, there is a risk of reducing the data’s weight score (importance rating). This is where the QJL comes in; it acts as a mathematical error-checker that corrects for a possible loss in attention score (the importance given to the data) during quantization. QJL uses signed bits (+1 or -1) to balance quantization errors. This way, it keeps the attention score perfectly (or nearly) accurate by balancing low-precision data with high-precision queries.

TurboQuant on QVAC SDK: More possibilities for Local AI

TurboQuant is a major breakthrough for local and cloud AI, but especially for local AI, where computing overhead is a major bottleneck for routine use. Tether recognizes the technological brilliance of the algorithm and its potential for models built to operate on tight resources. By compressing what would normally consume 8GB of VRAM down to 1.6GB, TurboQuant frees up resources for your inference machine, expands your bandwidth, and imagination of what can be done with a local superintelligent setup.

The TurboQuant integration via qvac-fabric-llm.cpp is supported by the Vulkan backend. This offers important compatibility and performance advantages attributable to Vulkan’s agnosticism and TurboQuant’s direct GPU execution.

Vulkan support bridges the advantages that TurboQuant offers to a wider range of user-grade devices and vendors outside the NVIDIA ecosystem (AMD and NVIDIA are currently supported, with mobile GPUs planned). It enables users and developers to run highly optimized, compressed local inferences on a wide range of platforms, including personal computers and mobile device GPUs.

TurboQuant’s KV Cache compression happens directly on the device’s GPU and aligns with how a computer naturally handles operations. This means the maths is done on the GPU’s fastest, closest memory, ensuring that models served with Fabric achieve the full 5x reduction in KV cache size while maintaining performance and precision. This lets users run much longer contexts (over 262,000 tokens) without running out of VRAM capacity.

TurboQuant lets you do more, with fewer resources, and in everyday environments. From simple follow-up queries to reviewing files that run in multiple gigabytes on your personal computers or mobile phones, it expands the scale of what can be done with an AI application. In QVAC SDK, it complements other optimization techniques inherent in Tether’s AI framework to power native intelligent systems that support an infinite number of users and autonomous agents. In a ten-billion-strong society, such systems will form a secure, viable, and unstoppable foundation for building the most complex superintelligent units for everyday use, biotechnology, and more.

From a macro perspective, compression techniques that reduce the operational resources required by AI models are the industry standard. The ability to develop and integrate such techniques will significantly impact the success of local AI models and infrastructure.

Tether is committed to building AI solutions that run on any setup and let choose their own biases. Follow the QVAC revolution and contribute to Tether’s drive for open source AI.

---