i

$ pip install mlx-optiq

ii

from mlx_lm import load, generate

model, tok = load("mlx-community/Qwen3.5-9B-OptiQ-4bit")
out = generate(model, tok, prompt="Explain mixed-precision quantization.", max_tokens=200)
print(out)

iii

# 1-2 min, once per model
$ optiq kv-cache mlx-community/Qwen3.5-9B-OptiQ-4bit \
    --target-bits 5.0 -o ./kv

# OpenAI + Anthropic compatible server on :8080
# /v1/chat/completions  (OpenAI)
# /v1/messages          (Anthropic; works with Claude Code, anthropic SDK, etc.)
$ optiq serve --model mlx-community/Qwen3.5-9B-OptiQ-4bit \
    --kv-config ./kv/kv_config.json \
    --port 8080

i

Mixed-precision weights

Per-layer KL on calibration data picks the bits. Sensitive layers stay high-precision, the rest go low, at the same average size as uniform-4.

Higher accuracy at the same disk size as uniform-4

ii

Mixed-precision KV cache

A separate sensitivity pass on the KV cache. Layer 0 is often 56× more sensitive than average, so uniform 4-bit KV is catastrophic; mixed-precision is not.

Faster long-context decode without breaking quality

iii

LoRA, two ways

Fine-tune with adapter rank scaled by each layer's bits, then keep N adapters mounted on one base and switch them per request, no reload.

Sensitivity-aware rank · hot-swap mounted adapters

iv

Text and images, one stack

Run text models, and send pictures to the ones that take vision. The vendored vision tower rides in a bf16 sidecar, so one repo loads text-only under mlx-lm or full image+text under OptIQ. Vision docs.

VLM + LLM · one artifact, no mlx-vlm dependency

vi

OptIQ Lab, a local GUI

A web UI for the whole workflow: quantize wizard, SFT/DPO fine-tuning with a dataset designer, and chat with sandboxed tools (web search, Python, terminal) and image upload.

Quantize, design data, fine-tune, and chat in the browser

1. Measure

For each layer, simulate-quantize just that layer at each candidate bit-width. Forward-pass calibration data. Measure KL divergence between the perturbed logits and the reference logits. Repeat for every layer; you now have a (layer, bits) → quality cost table.

2. Allocate

Greedy knapsack on the table: start every layer at the lowest bit-width, then greedily upgrade the layer that buys the most KL-reduction per extra bit until the average bit-budget is exhausted. Layers like lm_head and the first/last attention blocks are protected at 8-bit by default.

3. Convert

Hand the per-layer bit map to mlx_lm.convert as a quant predicate. The output is a standard MLX checkpoint that loads anywhere stock mlx-lm loads, with sensitivity metadata stashed on the side for downstream LoRA training.

# Auto-routes between bf16 and uniform-4-bit reference
# based on available RAM.
$ optiq convert Qwen/Qwen3.5-9B \
    --target-bpw 5.0 \
    --candidate-bits 4,8 \
    --reference auto \
    -o optiq_output/Qwen3.5-9B

Why this scales A single calibration-driven sensitivity path. --reference auto picks bf16 if it fits, otherwise falls back to a uniform-4-bit baseline with bf16-streaming probes, so 27 B+ models still get a calibration-driven signal on a 36 GB Mac. The full methodology lives in our research write-up →

Can I run LLMs locally on a Mac?

Yes. mlx-optiq runs large language models natively on Apple Silicon, from M1 to M5, using Apple's MLX framework. Install it from PyPI with pip install mlx-optiq, then quantize, fine-tune, and serve models entirely on your Mac, fully offline.

Do I need a GPU or PyTorch to run LLMs on a Mac?

No. There is no PyTorch and no discrete GPU in the path. mlx-optiq is MLX-native and uses the unified memory of Apple Silicon directly, so a MacBook, Mac mini, or Mac Studio is enough. No CUDA, no cloud, no API key.

How much RAM do I need to run an LLM on a Mac?

It depends on the model. A 4-bit OptIQ quant of a 4B model needs roughly 3 GB; a 9B needs about 6 GB; larger mixture-of-experts models need more. Mixed-precision 4-bit quantization is what lets bigger models fit in a Mac's memory while staying close to full-precision quality.

What is mlx-optiq?

An MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon. Its core is data-driven mixed-precision quantization: it measures each layer's sensitivity and assigns per-layer bit-widths, so quants keep more quality than uniform 4-bit at the same size. It also ships a local web UI (OptIQ Lab) and an OpenAI and Anthropic compatible server.