The Mac Studio M3 with 512GB of memory is an absurd amount of computer. When I bought one to run 600+ billion parameter language models at home, I knew I was signing up for adventure.

I was already an AI devotee — multiple paid subscriptions, dozens of experiments with Llama and other “small” LLMs. Over time, the possibility of running frontier-class models on my own hardware simply became irresistible.

Still, every review I read about the Mac Studio repeated the same warning: prompt processing on Apple Silicon is too slow for serious LLM work.

Either those reviewers were too early, or I was too stubborn, because I found the bottleneck to be surprisingly negotiable... if you don’t mind sacrificing a few weekends.

Somewhere in the depths of Reddit threads, academic papers, and sparsely documented code, I glimpsed my white whale — Claude Code, powered by local, frontier-class models, running at usable speeds. If local models can do this, they graduate from “expensive hobby” to “actually useful.”

This post is a breadcrumb trail for anyone foolhardy curious enough to try this themselves. Whether it represents victory or Stockholm syndrome, you can decide.

Apple is rarely the cheapest option in any category. But in this fringe use case, I found that a Mac Studio was the only viable option under $10k USD.

Alternatives exist: $50k multi-GPU workstations, DIY Frankenstein servers built from scavenged 3090s, mini-clusters of NVIDIA DGX Sparks.

But uniquely, the Mac Studio is configurable up to 512GB memory1, runs cool and quiet, and won’t trip your circuit breakers at full load.

As for performance, there are broadly three separate questions:

1. What’s the smartest (read: largest) model that it can run?

2. How long does it take to respond? (prompt processing or “prefill”)

3. How fast does it respond? (token generation or “decode”)

On the first dimension, the 512GB Mac Studio is unparalleled, because it can comfortably house the best open weight models currently in existence. Even Kimi K2.5‘s one trillion parameters, compressed optimally, fits with room to spare.

Benchmarks often claim that open models are near parity with the frontier, but in my experience, the gap is slightly wider. To me, the best open models are comparable to API models from 6 to 12 months ago. Still impressive, but 6 months is an eternity in this space!

On the second and third points, I’ll let the numbers speak. Below are results that approximate my day-to-day use cases:

To put these numbers into perspective, even with my slowest model:

• Simple chat messages get replies within seconds.

• Asking for edits on this post (~7000 tokens) takes ~30 seconds.

• Running Claude Code with a moderate system prompt (~16000 tokens) takes ~90 seconds.

And in each case, the response streams at more than 3x my reading speed.

Slower than the APIs? Yes. Unusable? Hardly.

Enough preamble — let’s get concrete. Starting with package management:

• homebrew for Mac

• uv for Python

• pnpm for Node

These are technically optional, but they make life easier:

Then hf to manage model downloads from Hugging Face:

And finally, Claude Code itself:

By default, macOS caps the GPU to 75% of total memory. That’s a reasonable default, but on a 512GB machine, that wastes ~120GB!

Empirically, I’ve found that 6GB is enough for operating system overhead. I also set iogpu.wired_lwm_mb to 10GB to encourage memory reclamation and compression when things get tight, but it’s unclear whether that makes a difference:

sysctl alone doesn’t persist across reboots and modern macOS silently ignores /etc/sysctl.conf. So we need a Launch Daemon to run this automatically on startup:

Save to /Library/LaunchDaemons/local.increase-gpu-memory.plist and then run:

There are many different apps and open-source servers for running LLMs on Macs. But almost all of them wrap the same two foundational projects: mlx-lm for language models and mlx-vlm for multimodal.

If you’re spending money on this hobby, you might as well stay close to the source. It gives you more room to optimize and a much better sense of how models actually work.

It’s also dead simple with uvx:

This installs the latest mlx-lm, starts a server on localhost, and downloads my Qwen 3.5 quantization from HuggingFace. By default, models are saved to ~/.cache/huggingface/hub.

A couple years ago, llama.cpp would have been a solid option as well, but in 2026, MLX is consistently 10-25% faster.

“Models” plural, because apps like Claude Code increasingly expect multiple models under the hood: a large one for complex tasks, a medium one for when speed matters, and a small one for narrow, one-off utilities.

When selecting models, the important rules of thumb are:

• More parameters correlates with more intelligence.

• Speed is proportional to active parameters, not total.

This makes Mixture-of-Experts (MoE) architectures especially attractive. These keep the full model weights in memory (which the Mac Studio has in abundance) while activating only a subset per token (which keeps inference speed usable).

This is how Kimi K2.5‘s massive 1 trillion parameter footprint translates into surprisingly decent speeds — activating only 32 billion at a time.

GLM 5 is a smaller, equally capable model, weighing in at 744 billion. But because it activates 40 billion, it actually runs 20% slower than Kimi.

Qwen 3.5 397B activates 17 billion, roughly half of Kimi’s, and as you might expect, runs roughly twice as fast.

Predictable, once you see the pattern.

Before downloading, there’s one more decision to make — which “quant” to choose?

At a high-level, quantization is simply compression: reducing full precision floats into smaller, more efficient formats. Like other forms of lossy compression, the goal is to get as small as you can without destroying quality.

At a low-level, quantization is PhD territory — obscure acronyms, formal math, endless tradeoffs, and superstitious lore.

My practical takeaways:

• Hardware matters. For example, older generation M1 and M2 Macs choke on BF16 weights. Future generations may support even more formats.

• 4-bit is the sweet spot on Apple Silicon. It’s GPU-optimized and a good balance of size and quality.

• 8-bit is another optimized peak. In most cases, nearly identical quality to full precision, but at half the size.

• Dynamic or mixed precision is essential for optimization. This means keeping important weights at high precision, while compressing the less sensitive ones more aggressively.

• Larger models can remain stable at lower quants (< 4-bit), while smaller models need to stay at 4 or above. Quantization-Aware Training (QAT) can also support lower quants.

• Prefer quants with published, verifiable benchmarks. Measuring quality is impossible with “vibes” alone.

The process for creating and evaluating quants is worth its own post, but you can use my mixed-width MLX quants as a starting point.

mlx_lm.server already has logic for switching between models, but what we actually want is multiple models loaded in parallel, served behind a single API endpoint.2

For this, I use llama-swap. It provides:

1. A management layer for multiple AI server instances.

2. A web UI for debugging logs and raw requests.

3. Centralized settings for generation parameters (ex: temperature, enable_thinking).3

4. Model aliases and convenience shortcuts.

First, install it:

Then, create a YAML config. The following is my core setup:

• One small, medium, and large model in parallel.

• A vision model (accessible via a vision alias).

• Default to non-thinking mode.

• *:think aliases to enable thinking mode.

• Recommended parameter overrides for each model and mode.

And then, to run it:

Visit http://localhost:8080 to see your new API gateway in action.

At ~465GB total and 10GB of KV cache set aside for each model, this is close to the maximum for a 512GB Mac Studio!

Stop here and you’d already have a general purpose AI server capable of handling OpenAI-compatible /v1/chat/completions requests. This covers 99% of the apps out there.

But since Claude Code uses Anthropic’s /v1/messages API format, we need a translation layer.

The easiest path is claude-code-router4.

Save the following to ~/.claude-code-router/config.json to wrap our llama-swap API in an Anthropic-flavored proxy:

If you like control, you can start the router and Claude Code independently:

Or if you prefer convenience, just use the built-in shortcut:

Type in a prompt or two, watch the traffic flow through llama-swap, and pour yourself a drink. You now have Claude Code running locally!

Of course “running” and “running at usable speeds” are two different standards. And our naive initial setup fails the speed test.

The first culprit is Claude Code’s sprawling, dynamic system message. Including skills, commands, and tool definitions, it’s often over 20,000 tokens long!

You might think that this system message represents some kind of hyper-optimized prompting masterpiece. But what I found instead was verbose and repetitive.

After several rounds of iteration, I was able to distill it into something much, much leaner:

This is paired with a shell script that appends local environment and context:

And for further gains, we can save thousands more tokens by only passing in tools we actually need:

You can browse the entire tool catalog here, but I’ve found Bash with a few search and editing shortcuts is more than enough.

Combined, these changes take us from 20k tokens to less than 8k. This cuts processing time and stretches your effective context window too.5

I use this slimmed-down version even with cloud Claude — it simply works better.

The textbook solution for expensive operations like prompt processing is caching — calculate once, reuse later. mlx-lm handles this automatically by default.

In the ideal case, each chat turn builds exactly upon the previous one, and we only need to process the new messages (bolded):

Turn 1

system: You are Arthur, King of the Britons. You are carrying: sword, crown, coconuts
user: What is your quest?

Turn 2

system: You are Arthur, King of the Britons. You are carrying: sword, crown, coconuts
user: What is your quest?
assistant: To seek the Holy Grail.
user: What is the airspeed velocity of an unladen swallow?

Turn 3

system: You are Arthur, King of the Britons. You are carrying: sword, crown, coconuts
user: What is your quest?
assistant: To seek the Holy Grail.
user: What is the airspeed velocity of an unladen swallow?
assistant: What do you mean? An African or European swallow?
user: Huh? I don’t know...

But even a single changed character forces us to recalculate the message history from the point of divergence.

And in practice, Claude Code is not a well-behaved app!

For reasons unclear to me, it changes the order of tools and tool arguments in between turns. Or in the example above, “sword, crown, coconuts” might become “crown, coconuts, sword” or “coconuts, crown, sword.”

Since tools are defined at the beginning of each chat, this leads to slow, full reprocessing on every single message. The longer the chat, the longer the wait.

There are multiple ways to fix this, but the easiest place is in the model’s chat template. For example, here’s my edit to the official Qwen 3.5 template, which sorts tools and arguments into a consistent order:

A similar fix could be configured globally on the server or proxy, to avoid needing to patch every model.

With Claude Code tamed, performance is now nearly acceptable.

“Nearly,” because the ability to break prompt caching is not limited to the unruly clients. Models themselves can also create tricky caching dilemmas.

For example, Qwen 3.5 uses dynamic <think> tag placement. Because these are automatically added and removed from history, they always prevent the previous turn’s assistant response from matching:

Turn 1

system: You are Arthur, King of the Britons.
user: What is your quest?

Turn 2

system: You are Arthur, King of the Britons.
user: What is your quest?
assistant: <think>Ah, the classic Bridge of Death scene.</think>To seek the Holy Grail.
user: What is the airspeed velocity of an unladen swallow?

Turn 3

system: You are Arthur, King of the Britons.
user: What is your quest?
assistant: To seek the Holy Grail.
user: What is the airspeed velocity of an unladen swallow?
assistant: <think>The answer is 11 meters per second, but I should stay in character.</think>What do you mean? An African or European swallow?
user: Huh? I don’t know...

Notice how the first assistant message is processed (bolded) twice? That’s because the <think> tag shifted in turn 3.

Redoing the previous message is better than restarting from the beginning, but still a major drag over the course of a multi-turn chat. Unlike fixing tool order though, this issue is inherent to the model’s chat template logic. Changing it would mean deviating from the model’s chat training — not a good idea.

I spent a few days diving into the mlx-lm implementation and eventually came up with a workaround — speculative prompt processing.

As soon as the model responds, calculate how the thinking tags would shift next turn, and immediately start processing up to the point of the new user message.

The extra work still happens, but it gets a head start. So while I’m reading the model’s previous message and drafting up my own response, the server makes use of those seconds to reduce future wait time.

This feature isn’t available in the official mlx-lm release yet, but feel free to comment on the PR or give my fork a try:

Here, deep in the weeds of experimental fixes and unmerged PRs, our guided tour draws to a close. Beyond this lies the true frontier.

By this point, you have a blueprint for setting up a powerful local AI server. Fully-private, no rate limits, no vendor lock-in.

Only one question remains — is it worth it?

For the price of a Mac Studio, I could have bought 1 billion uncached Opus tokens... or 4 years of the $200/month Max plan. And that’s not counting all the hours I spent chasing the whale through git diffs, network logs, and benchmark results.

Yet whenever I see the silver box humming away on my shelf, I feel genuinely proud. Via Claude Code, it aces my merge conflicts. Through my phone’s camera, it processes my bills. In my notes app, it proofreads my drafts.

By 2026 standards, these are mundane feats. But compared to yesterday’s frontier? The models running on my Mac Studio today are at least as capable as Sonnet 3.7... which itself was state-of-the-art just over one year ago.

Local AI definitely isn’t a drop-in replacement for cloud APIs. If you expect everything to work on the first try, you’re bound to be disappointed. But if reading this post intrigued rather than exhausted you — if part of you finds joy in the hunt for solutions — then I hope this helps justify your next hardware splurge. :)

Besides, at the current rate of new model releases, architecture innovations, software and hardware improvements... this expensive hobby might yet pay off.