< all articles
·5 min read

Breaking the Latency Wall: Optimizing MLX and Qwen for Local AI Agents on Apple Silicon

Why your local coding agent stalls on simple terminal commands, how Speculative Decoding breaks the KV cache in MLX, and how to configure your stack for subsecond responses.

  • local-ai
  • mlx
  • apple-silicon
  • agents
  • qwen

If you’re running local AI models on a high memory Mac, you already know Apple Silicon is a game changer. Recently, I started integrating local models into my development workflow, running Qwen 27B via MLX using autonomous agents in the CLI (like OpenCode and Aider).

But I quickly hit a wall. Every time the agent tried to execute a simple terminal command, I found myself staring at a 30 second delay.

After digging into the MLX architecture, I realized my inference configuration was silently crippling my productivity. Here is a technical breakdown of why your local agent is stalling, how Speculative Decoding is breaking your cache, and how to properly configure your stack.

1. The Agentic “Snowball” Effect

To understand the bottleneck, you have to understand how autonomous agents work in the CLI. They don’t just chat. They execute loops: reading files, running bash scripts, and dumping raw stdout errors back into their context window.

Because LLMs are stateless, every single action and file read must be appended to the prompt. A simple debugging session will quickly snowball a small prompt into a massive, ever growing context history in just a few turns.

2. The Trap: Using Draft Models (MTP) for Speed

When faced with an ever growing context window, token generation feels sluggish. The natural instinct for any developer optimizing MLX is to enable Speculative Decoding (Multi Token Prediction, or MTP).

You do this by passing a smaller draft model to the server alongside your main model.

How it’s supposed to work: The smaller draft model runs ahead, quickly guessing the next several tokens. The massive 27B model then steps in simply to verify those guesses. If they match, you get a massive boost to your Tokens per Second (TPS).

3. The Core Bug: Why Draft Models Kill the KV Cache

Here is the exact architectural conflict: this is a limitation of the MLX engine itself, not a quirk unique to Qwen.

When you enable Speculative Decoding, MLX has to maintain two separate KV caches simultaneously: one for the draft model and one for the main model. In a standard conversational chat, these two caches stay in lockstep. But in an autonomous CLI loop, your agent constantly injects unpredictable terminal output, stack traces, and diffs.

This abrupt influx of suffix data causes the draft model’s state tree to fall out of mathematical alignment with the main model’s KV tensors. To prevent text corruption or server crashes from desynchronized states, the MLX engine acts defensively: it dumps the entire KV cache.

While this dual cache breakdown affects any model paired with a draft model in MLX, Qwen amplifies the problem. Recent Qwen architectures use hybrid designs that blend standard Attention with state space (SSM/Mamba) layers. Because hybrid architectures already have tricky state tracking and prefix reuse quirks in current MLX builds, running Qwen with MTP turns an existing engine friction point into a guaranteed cache blowout.

The result? Your Time to First Token (TTFT) skyrockets. You wait over 30 seconds for the engine to recalculate a massive context prefill from scratch, just so the agent can output a simple, 10 token command like cat script.py.

4. The Fix: Prioritize TTFT over TPS

In agentic workflows, TTFT is king. A latency of 30 seconds on the prefill destroys the autonomous loop.

To fix this, you must drop the --draft-model and --draft-kind mtp flags completely. Run a standard mlx_lm.server and let basic Prefix Caching take over.

Without the draft model interfering, MLX successfully holds the cached context in memory. When the agent acts, the engine only processes the delta (the newest tokens). Your prefill latency drops from over 30 seconds to less than a second.

5. The Consequence: The 4-bit vs. 8-bit Dilemma

By disabling the draft model to save our cache, our agent is responsive again, but we have lost our Speculative Decoding speed boost. Our token generation speed drops significantly.

Because we are now relying solely on raw inference speed, quantization becomes your primary bottleneck. Apple Silicon inference is strictly bound by memory bandwidth, forcing you to make a choice based on your Mac’s RAM:

  • Option A: Optimize for Speed (4-bit)
    A 27B model in 4-bit reads half the data from memory, pushing your generation speed to over 30 TPS. However, autonomous coding logic is highly sensitive to quantization loss. 4-bit models are more likely to write syntax errors or get stuck in hallucinated debugging loops.

  • Option B: Optimize for Accuracy (8-bit)
    If you have enough Unified Memory (e.g., 128GB) to easily hold an 8-bit model and a large context window without hitting swap memory, this is the way to go. The 8-bit model is vastly superior at complex coding logic. The catch? Because you turned off the MTP draft model, and 8-bit pushes twice the data through the memory bus, your speed will hit a physical ceiling of around 16 to 17 TPS.

The Verdict

Right now, running local agents means optimizing for your specific hardware.

If you want an autonomous agent that actually works seamlessly: Kill the draft model, rely on prefix caching, and use an 8-bit model if you have the RAM for it. You will trade high TPS for logical stability and instant reaction times, which is exactly what an autonomous CLI agent needs to succeed.