Optimizing compiler · MLX
Run large language models locally on your Mac, from M1 to M5. Per-layer sensitivity analysis for mixed-precision weights. LoRA fine-tuning that respects the bit budget. A server that speaks both OpenAI and Anthropic APIs (point Claude Code at your local quant). Send it an image, not just text, on any vision-capable model. No GPU cluster, no API key.
Per-layer bit allocation · sample LLM
$ pip install mlx-optiq
3.1×
avg compression vs bf16
+1.4×
decode via MTP / drafter
+13.6
best Capability Score gain
140k+
HF downloads / month
OptIQ Lab — your Mac, your models, no cloud
pip install away: chat with sandboxed tools, compare models, fine-tune, and serve. Here a 4-bit OptIQ Qwen3.5 answers at 90 tok/s, fully offline.01 Pre-built models
Sixteen production mlx-optiq-quantized LLMs on Hugging Face. Nemotron 3, MiniCPM5, Qwen3.5, Qwen3.6 and Gemma-4 families, from 1 B dense to 35 B-A3B mixture-of-experts. They load directly into stock mlx-lm. No special runtime.
02 Quickstart
Each step is reversible and works with stock MLX tools. mlx-optiq is additive. Skip any of these and you still have a working pipeline.
i
Pure Python. Pulls in mlx, mlx-lm and huggingface-hub. Python 3.11+ on Apple Silicon.
terminalbash
$ pip install mlx-optiq
ii
Pre-built mlx-optiq quants load with stock mlx-lm. Per-layer bit assignment is recorded in the model metadata. No special loader required.
generate.pypython
from mlx_lm import load, generate model, tok = load("mlx-community/Qwen3.5-9B-OptiQ-4bit") out = generate(model, tok, prompt="Explain mixed-precision quantization.", max_tokens=200) print(out)
iii
The KV cache is its own sensitivity problem. optiq kv-cache measures it once per model; optiq serve serves with the resulting per-layer config behind an OpenAI-compatible API.
terminalbash
# 1-2 min, once per model $ optiq kv-cache mlx-community/Qwen3.5-9B-OptiQ-4bit \ --target-bits 5.0 -o ./kv # OpenAI + Anthropic compatible server on :8080 # /v1/chat/completions (OpenAI) # /v1/messages (Anthropic; works with Claude Code, anthropic SDK, etc.) $ optiq serve --model mlx-community/Qwen3.5-9B-OptiQ-4bit \ --kv-config ./kv/kv_config.json \ --port 8080
Where to next Each model family has a getting-started guide with model-specific sampling defaults and recommended use cases. Building an agent? Drop llms.txt into your IDE. It's the entire library reference in one Markdown file.
03 What it does
A single per-layer KL-divergence pass drives weight, KV-cache and LoRA-rank allocation. The rest of the toolkit (hot-swap adapters, multi-protocol serving with five tested client integrations, image input on the vision models, and the OptIQ Lab GUI for quantize, fine-tune, dataset, and chat workflows) sits around that core.
i
Per-layer KL on calibration data picks the bits. Sensitive layers stay high-precision, the rest go low, at the same average size as uniform-4.
Higher accuracy at the same disk size as uniform-4
ii
A separate sensitivity pass on the KV cache. Layer 0 is often 56× more sensitive than average, so uniform 4-bit KV is catastrophic; mixed-precision is not.
Faster long-context decode without breaking quality
iii
Fine-tune with adapter rank scaled by each layer's bits, then keep N adapters mounted on one base and switch them per request, no reload.
Sensitivity-aware rank · hot-swap mounted adapters
iv
Run text models, and send pictures to the ones that take vision. The vendored vision tower rides in a bf16 sidecar, so one repo loads text-only under mlx-lm or full image+text under OptIQ. Vision docs.
VLM + LLM · one artifact, no mlx-vlm dependency
vi
A web UI for the whole workflow: quantize wizard, SFT/DPO fine-tuning with a dataset designer, and chat with sandboxed tools (web search, Python, terminal) and image upload.
Quantize, design data, fine-tune, and chat in the browser
04 How it works
Uniform 4-bit quantization treats every layer the same, but layers are not the same. mlx-optiq measures, then allocates.
For each layer, simulate-quantize just that layer at each candidate bit-width. Forward-pass calibration data. Measure KL divergence between the perturbed logits and the reference logits. Repeat for every layer; you now have a (layer, bits) → quality cost table.
Greedy knapsack on the table: start every layer at the lowest bit-width,
then greedily upgrade the layer that buys the most KL-reduction per extra bit
until the average bit-budget is exhausted. Layers like lm_head
and the first/last attention blocks are protected at 8-bit by default.
Hand the per-layer bit map to mlx_lm.convert as a quant
predicate. The output is a standard MLX checkpoint that loads anywhere
stock mlx-lm loads, with sensitivity metadata stashed on
the side for downstream LoRA training.
convert.shbash
# Auto-routes between bf16 and uniform-4-bit reference # based on available RAM. $ optiq convert Qwen/Qwen3.5-9B \ --target-bpw 5.0 \ --candidate-bits 4,8 \ --reference auto \ -o optiq_output/Qwen3.5-9B
runs in 1–2 min on a 9 B model · longer for 27 B+
Why this scales
A single calibration-driven sensitivity path. --reference auto
picks bf16 if it fits, otherwise falls back to a uniform-4-bit baseline
with bf16-streaming probes, so 27 B+ models still get a calibration-driven
signal on a 36 GB Mac. The full methodology lives in
our research write-up →
05 How it compares
A snapshot of how the popular paths stack up on the things that actually move quality and speed on Apple Silicon. None of these are wrong; they're optimizing different axes.
| mlx-optiq | mlx-lm | llama.cpp | |
|---|---|---|---|
| Per-layer mixed-precision weights | Yes, calibration-driven | Uniform 4-bit | Block-wise K-quant |
| Per-layer mixed-precision KV cache | Yes | Uniform 4 / 8 / fp16 | Group-wise int8 only |
| Sensitivity-aware LoRA fine-tuning | Rank scaled by per-layer bits | Constant rank LoRA | Inference only |
| OpenAI and Anthropic compatible server | One process, both | OpenAI only | llama-server (OpenAI shim) |
| Text and image input | Yes | Text only | Image via separate build |
| Sandboxed tool support for chat | Three tools: web search, Python, terminal | — | — |
Reading the table mlx-optiq is the only path on this list that uses calibration-driven, per-layer bit allocation for both weights and KV cache, in the native MLX runtime, with serving, fine-tuning, and image input on the vision models in the same package. The others are great at what they target. They just target different things.
06 FAQ
Yes. mlx-optiq runs large language models natively on Apple Silicon, from M1 to M5, using Apple's MLX framework. Install it from PyPI with pip install mlx-optiq, then quantize, fine-tune, and serve models entirely on your Mac, fully offline.
No. There is no PyTorch and no discrete GPU in the path. mlx-optiq is MLX-native and uses the unified memory of Apple Silicon directly, so a MacBook, Mac mini, or Mac Studio is enough. No CUDA, no cloud, no API key.
It depends on the model. A 4-bit OptIQ quant of a 4B model needs roughly 3 GB; a 9B needs about 6 GB; larger mixture-of-experts models need more. Mixed-precision 4-bit quantization is what lets bigger models fit in a Mac's memory while staying close to full-precision quality.
An MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon. Its core is data-driven mixed-precision quantization: it measures each layer's sensitivity and assigns per-layer bit-widths, so quants keep more quality than uniform 4-bit at the same size. It also ships a local web UI (OptIQ Lab) and an OpenAI and Anthropic compatible server.
Get started
Pick a model, get a snippet, ship it. The docs cover every supported family, fine-tuning recipes, and the OpenAI-compatible serving stack.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。