Same Mac. Same model.
48% faster.

oMLX generates half as fast again.

oMLX is an inference server for Apple Silicon built on Apple's MLX framework: continuous batching, a tiered KV cache that spills to SSD, and OpenAI- and Anthropic-compatible endpoints, which is what lets tools like Claude Code and Cursor talk to a model on your own Mac. It went loud on X and Reddit this week, so we did what we always do: picked the same ready-provisioned M4 Pro Mac mini (24GB) that ran last week's Qwen3.8-27B benchmark, and measured.

One honesty note before the numbers: the engines cannot run literally identical weights. llama.cpp ran the unsloth UD-Q4_K_M GGUF (16.5GB); oMLX ran the mlx-community 4-bit conversion (15.0GB). Both are the standard 4-bit builds each engine actually uses, but the MLX file is about 9% smaller, and on bandwidth-bound hardware a smaller file is itself a speed advantage. Some of oMLX's win is quant size, the rest is kernel efficiency. We measured speed, not output quality.

The numbers, side by side.

All runs on the same machine, same 1,779-token prompt, temperature 0, flash attention on for llama.cpp, GPU wired limit raised to 20GB (see the setup section below). Prompt processing first, then sustained generation:

Prompt processing is a dead heat: it is compute-bound and both engines drive the GPU well. Generation is where they separate. 10.7 tok/s (llama-bench) to 15.8 tok/s does not sound dramatic until you sit in front of it: it is the difference between reading at the model's pace and waiting for it.

MTP made it slower. On a Mac, that's expected once you see why.

Qwen3.8 ships a multi-token prediction head: a small draft model (1.4GB as a GGUF) that guesses several tokens ahead so the big model can verify them in one pass. AMD Strix Halo owners report real gains with it. On our M4 Pro it ran at 7.9 tok/s against 10.7 without – 26% slower, despite a healthy 65% draft acceptance rate.

The reason is the shape of the hardware. Speculative decoding wins when verification is much cheaper than generation, which is true where compute is plentiful relative to memory bandwidth. Apple Silicon is the opposite: generation is bandwidth-bound, verification passes still read the full 15GB of weights, and the draft head's own overhead comes straight off the top. Our first attempt also ran out of GPU memory entirely – two models plus two KV caches do not fit under a 20GB wired limit with a Q4_K_M base – so the numbers above use the smaller IQ4_XS base, which is the only way it fits on 24GB at all.

If you are on a Mac, skip the MTP head for now. That advice may age: kernels improve, and a batched verify that reads weights once is theoretically possible. Today, it costs you a quarter of your speed.

The feature that actually changes daily use.

oMLX keeps processed prompts in a tiered cache: hot in RAM, cold on SSD. Send a prompt whose prefix it has seen and it skips straight to the new tokens. Our 1,779-token prompt took 18.4 seconds to process cold; sending it again took 4.0 seconds end to end, most of which was generating the reply.

Benchmarks undersell this. Real use of a local model is not one-shot prompts: it is a chat that regrows the same history every turn, or an agent like Claude Code re-sending a fat system prompt with every tool call. In those loops, prompt processing is most of the wall-clock time, and a prefix cache removes most of it. llama.cpp has an in-RAM prompt cache per slot; oMLX's is bigger, survives model swaps via SSD, and needs no configuration. This, more than the generation speed, is why the machine feels different to sit at.

Where 24GB runs out, whoever's serving.

oMLX's flagship feature is continuous batching: multiple users, one model, overlapping generation. On this machine, with this model, we could not make it shine, and the reason is worth understanding before you buy hardware.

Three simultaneous requests against llama.cpp simply queue: aggregate throughput was 10.0 tok/s, exactly one user's worth, shared. The same test against oMLX served one request at full speed and rejected the other two – its memory guard calculated, correctly, that batching three 1.8k-token contexts alongside a 15GB model would blow past what 24GB can hold. llama.cpp, given the same squeeze with the MTP config, crashed with a Metal out-of-memory error mid-request; oMLX declined politely up front. Same wall, better manners.

The honest conclusion: a 27B dense model on a 24GB Mac is a single-user machine, full stop. Continuous batching pays off when there is memory headroom for several contexts – which is exactly what 64GB machines are for. That is not a sales line; it is arithmetic.

The three settings that made it work.

Everything from last week's article still applies (wired memory limit, thinking mode off for interactive use). For oMLX on a 24GB machine, add its memory guard tier:

# allow the GPU to wire up to 20GB (resets on reboot)
sudo sysctl iogpu.wired_limit_mb=20480

# install and serve (models live in ~/.omlx/models)
brew tap jundot/omlx https://github.com/jundot/omlx
brew install jundot/omlx/omlx
omlx serve --model-dir ~/.omlx/models --memory-guard aggressive

The guard defaults to a conservative tier that refuses a 27B's prefill on 24GB. aggressive is safe here for single-user work: it still aborts rather than paging the machine to death. And that Anthropic-compatible endpoint means pointing Claude Code at your own Mac is one base-URL change.

Run oMLX. Skip the MTP head. Buy RAM before software.

• Single user on Apple Silicon: oMLX, clearly. Half again the generation speed, a prompt cache that transforms agent and chat loops, and cleaner failure modes when memory gets tight.
• llama.cpp remains the ecosystem king. Every quant under the sun, first support for new architectures, and if you need GGUF-only models it is still the tool.
• MTP on Macs: not yet. 26% slower in our tests. Watch this space; ignore the hype today.
• Concurrency is a memory question, not a software question. No server makes three users fit where one barely does. If your team wants to share a 27B, that is a 64GB machine.

How we ran these benchmarks.

One machine for every number: a ready-provisioned Max-tier Mac mini from our rack – Apple M4 Pro, 24GB unified memory, macOS 26.6.1, GPU wired limit at 20,480MB. Engines: llama.cpp b10470 (Homebrew, Metal, flash attention on) and oMLX 0.6.2 (Homebrew). Models: unsloth Qwen3.8-27B UD-Q4_K_M and UD-IQ4_XS GGUFs, the unsloth MTP Q4_0 draft head, and the mlx-community Qwen3.8-27B 4-bit conversion. Method: 1,779-token prompt, temperature 0, max 256 output tokens; single-stream numbers are server-reported timings for llama.cpp and two-point wall-clock measurements (1-token vs 256-token requests, unique prompts to defeat prefix caching) for oMLX. Concurrency: three simultaneous requests with distinct prompts. We measured speed, not output quality; the engines run different 4-bit quantisations of the same weights, as noted above.