Gemma 4 12B vs Qwen 3.5 9B: Which Local Model Wins for AI Agents?

Two small models. Both run on a laptop. Both support tool calling. Both Apache 2.0. The right choice depends on what your agent actually needs to do.

I was testing a customer support agent on my M2 MacBook Pro last Tuesday. No API calls. No cloud dependency. Just a local model classifying tickets, extracting key fields, and drafting responses.

The model was Qwen 3.5 9B. It was fast. Accurate on text tasks. The VRAM footprint was small enough that I could run it alongside my IDE without the fans screaming.

Then Gemma 4 12B dropped on June 3rd. I swapped it in. Same agent, same prompts, same test data.

Here's what changed: the agent could now read attached screenshots from customer emails. Without a separate vision pipeline. Without any code change. Gemma 4 12B processes images, audio, and video natively in the same model that handles text. No encoder. No extra VRAM for a vision module. It just... works.

But it uses 3 billion more parameters. And for agents that only need text, those extra parameters are cost you don't need.

That's the real decision. Not "which model is better" but which model fits what your agent does. Here's the head-to-head breakdown.

The specs that matter (everything else is noise)

Both models dropped in early 2026 and both target the same sweet spot: small enough to run locally, capable enough for real agent work. (For the wider field of what you can run on your own machine this year, see our roundup of local AI in 2026.)

Gemma 4 12B (Google DeepMind, June 3, 2026): 11.95 billion parameters, dense architecture, encoder-free unified multimodal (text + image + audio + video), 256K context window, Apache 2.0. Runs in ~6.6 GB VRAM at Q4 quantization. Native tool calling with optional step-by-step reasoning mode.

Qwen 3.5 9B (Alibaba Qwen, March 2, 2026): 9 billion parameters, Gated DeltaNet hybrid architecture (3:1 linear-to-full-softmax attention), unified vision-language (text + image + video), 262K context window, Apache 2.0. Thinking and non-thinking inference modes. Multi-token prediction.

Both are instruction-tuned. Both support structured tool calling. Both are commercially usable. Both fit on a laptop. If you have more VRAM to work with, our Gemma 4 31B vs Qwen 3.6 27B comparison covers the larger siblings.

The size lineup (and the 12B that doesn't exist)

One clarification first, because it catches people out constantly: there is no Qwen 3.5 12B. If that's what you came looking for, you're thinking of Gemma's lineup.

• Gemma 4 ships at E2B, E4B, 12B, 26B A4B (mixture-of-experts), and 31B. There is no Gemma 4 4B and no Gemma 4 27B; the "E" models are effective-parameter sizes, and 27B was Gemma 3's top size.
• Qwen 3.5 ships at 0.8B, 2B, 4B, 9B, and 27B dense, plus 35B-A3B, 122B-A10B, and 397B-A17B mixture-of-experts models.

The two families don't line up cleanly at any size. The 12B and the 9B are the closest matched pair they have, which is why this is the comparison worth running. Anything broader ("Gemma 4 vs Qwen 3.5") means choosing a pairing first: E4B against 4B at the small end, 12B against 9B in the middle, and 31B dense against Qwen's 27B dense at the top. Above that, Qwen's 122B and 397B have no Gemma answer at all.

The differences are in three dimensions that matter for agents: multimodal capability, memory footprint, and architecture efficiency.

Multimodal: Gemma 4 12B wins on audio

This isn't close. Gemma 4 12B is the first mid-sized model to natively process text, images, audio, and video without separate encoders. The architecture projects image patches and audio waveforms directly into the shared decoder. No bolted-on vision encoder eating extra VRAM. No separate audio pipeline.

For agents, this means your local model can read a screenshot, listen to a voice note, and watch a short video clip without any additional infrastructure.

Qwen 3.5 9B supports text, images, and video. No native audio. For text, image, or video agent workflows, this is fine. But if your agent handles support tickets with voice memos or needs to hear what's said in a recording, Gemma 4 12B is the only option in this weight class.

If your agent needs to process audio, Gemma 4 12B is the only model in this matchup that handles it natively, and the biggest Gemma 4 that takes audio. That's the clearest differentiator.

VRAM and speed: Qwen 3.5 9B is leaner

Nine billion parameters vs twelve billion. That 25% size difference matters on consumer hardware.

Qwen 3.5 9B runs comfortably on GPUs with 8 GB VRAM at Q4 quantization. Gemma 4 12B needs ~6.6 GB at Q4KM but runs more comfortably with 16 GB. On machines where every gigabyte counts (an RTX 3060 with 12 GB, or an older MacBook), the 3 billion parameter difference translates to real headroom for context and batch processing.

Speed is more nuanced. Community testing on RTX 3060 found Gemma 4 12B has "overwhelmingly fast" prefill (input processing) speed. On M2/M3 MacBook Pro, Gemma 4 12B hits 30-50 tokens per second at Q4. Qwen 3.5 9B's hybrid architecture (with its linear attention layers) is designed for efficient inference on long sequences, and the smaller parameter count gives it an edge on token-per-second throughput at equivalent quantization.

For agents that process long documents or maintain extended conversations (where prefill speed matters most), Gemma 4 12B's fast prefill is an advantage. For agents that need fast generation on constrained hardware, Qwen 3.5 9B's leaner profile wins.

Quantisation: QAT, Q4, Q5, Q8 compared

This is where the decision actually gets made. Nobody runs these models at full precision on a laptop, so the real question isn't "12B or 9B" — it's "which quantisation of which model fits my card."

There's one asymmetry worth understanding before the numbers.

Gemma 4 ships quantisation-aware trained (QAT) weights. Qwen 3.5 doesn't. QAT means Google simulated 4-bit quantisation during training, so the model learned to be robust to it. Standard post-training quantisation (PTQ), which is what you get for Qwen 3.5 and most other GGUFs, squeezes the weights after training is finished and accepts whatever quality falls out.

In practice: Gemma 4 12B at QAT Q4 sits much closer to its BF16 quality than a PTQ Q4 model does. You get the 4-bit footprint without the usual 4-bit tax. That's a genuine advantage, and it partly cancels out the VRAM gap that the raw parameter counts suggest.

Qwen 3.5 9B at Q4 is a normal PTQ quantisation. It's still good — the degradation at Q4_K_M is modest for most agent work — but it is a real trade, not a free one.

VRAM by quantisation level

Weights only. Add roughly 1-3 GB on top for KV cache at a working context of 8K-32K, and considerably more if you actually push toward the 256K window.

Gemma's QAT Q4 build has the same ~6.6 GB footprint as its Q4_K_M — QAT changes the quality you get at that size, not the size itself.

What each level costs you

Q4_K_M is the default for a reason. On Qwen 3.5 9B expect a small but measurable drop on multi-step reasoning and long tool-call chains; single-tool calls and classification hold up fine. On Gemma 4 12B, use the QAT build here and the drop is close to negligible.

Q5_K_M is the honest middle. It recovers most of what Q4 PTQ gives up, for about 1.2 GB more on either model. If you have 12 GB and were about to run Q4 out of habit, run Q5 instead.

Q8_0 is effectively lossless for agent work. It's also 12.5 GB for Gemma, which means a 16 GB card and not much else running. Worth it only if you've measured a Q5 failure on your actual workload.

BF16 is not a laptop proposition for either model. Skip it unless you're benchmarking.

Which to run on your hardware

• 8 GB VRAM: Qwen 3.5 9B at Q4_K_M (~5.1 GB), leaving ~3 GB for context. Gemma 4 12B technically fits at QAT Q4 but leaves you almost nothing for KV cache.
• 12 GB (RTX 3060): either model at Q4, or Qwen 3.5 9B at Q5_K_M with comfortable headroom. This is the sweet spot for Qwen.
• 16 GB: Gemma 4 12B at QAT Q4 or Q5_K_M with room for a long context. This is where Gemma stops being a squeeze.
• 24 GB+ (RTX 4090, M-series with 32 GB unified): Gemma 4 12B at Q8_0, or drop the size question entirely and look at Gemma 4's 26B A4B and 31B tier instead.

Pull tags

# Gemma 4 12B — QAT Q4 is the one you want on constrained hardware
ollama pull gemma4:12b
ollama pull gemma4:12b-it-qat
ollama pull gemma4:12b-it-q4_K_M
ollama pull gemma4:12b-it-q8_0

# Qwen 3.5 9B
ollama pull qwen3.5:9b
ollama pull qwen3.5:9b-q4_K_M
ollama pull qwen3.5:9b-q8_0

The bare gemma4:12b and qwen3.5:9b tags resolve to the Q4_K_M build, which is what you want in most cases. Note that Ollama's download sizes run higher than the weights-only estimates above: 7.6 GB for gemma4:12b, 7.2 GB for gemma4:12b-it-qat, and 6.6 GB for qwen3.5:9b at the time of writing, so budget from the download size. Check ollama.com/library for the current tag naming before you script anything against it — Ollama's suffix conventions shift between releases.

Tool calling and agentic performance: both are capable, differently

Both models support native tool calling. Both can parse function schemas, select the right tool, format arguments, and process results. But they approach it differently.

Gemma 4 12B includes a dedicated tool-calling mode and an optional step-by-step reasoning mode. When the reasoning mode is active, the model generates intermediate reasoning tokens before selecting and calling tools. This improves accuracy on multi-step tasks but increases token consumption per tool call.

Qwen 3.5 9B has thinking and non-thinking modes. In thinking mode, the model generates internal reasoning before responding. The Qwen 3.5 family was built with the same architecture as the 397B flagship, and the 9B variant matches or surpasses models 10-13x its size across several agentic benchmarks.

The honest assessment: for structured tool calling (single tool, clear schema, straightforward parameters), both are reliable. For complex multi-step agent workflows with chained tool calls, the larger Gemma 4 12B tends to hold up better in reasoning quality, while Qwen 3.5 9B is faster per step.

Benchmarks (with the usual caveats)

Benchmarks are reference points, not guarantees. Your agent's real-world performance depends on your prompts, your tool schemas, and your specific use case. But here's what the numbers show:

Gemma 4 12B benchmarks (Google-reported): MMLU Pro: 77.2%. GPQA Diamond: 78.8%. AIME 2026: 77.5%. Beats last year's Gemma 3 27B (67.6% MMLU Pro) at less than half the parameter count.

Qwen 3.5 9B benchmarks (Alibaba-reported): Matches or surpasses GPT-OSS-120B (a model 13x its size) across multiple language and vision benchmarks. Agentic index: 55.5 per independent evaluation. The 3.5 family's function calling capability (measured on the larger 122B variant) scored 72.2 on BFCL-V4, outperforming GPT-5 mini by 30%.

The benchmark numbers suggest Gemma 4 12B edges ahead on reasoning-heavy tasks (GPQA, AIME) while Qwen 3.5 9B punches above its weight on efficiency-per-parameter across general language and agent tasks.

Neither model will match a frontier API model (Claude Sonnet at $3/M tokens is significantly more capable than either for complex reasoning). The comparison that matters is these models against each other, on the workloads you'll actually run locally.

How good is Qwen 3.5 9B on its own?

Setting Gemma aside for a moment, because plenty of people arrive at this question without a second model in mind.

Qwen 3.5 9B is the best 9B-class model I've run for agent work, and it isn't especially close. The headline is efficiency per parameter: it matches or beats models roughly ten times its size on general language and vision benchmarks, and the family's function-calling scores are genuinely frontier-adjacent. For a model that fits in 5.1 GB, that's a strange sentence to be able to write.

What it's good at: structured extraction, classification, summarisation, single-tool and short-chain tool calling, and long-context work where the Gated DeltaNet architecture pays off. Fast enough that you stop thinking about latency. Stable enough that it doesn't drift out of JSON when you ask for JSON.

Where it runs out: deep multi-step reasoning chains, tasks needing genuine world knowledge, and anything where a wrong answer is expensive. It will confidently produce a plausible wrong answer in exactly the places a 9B model always has. Thinking mode helps and costs you tokens.

The honest summary: excellent as the workhorse in a routed setup, where it handles the 60-70% of tasks that don't need frontier capability and hands the rest to an API. Risky as the only model in a system that has to be right.

If you want the fuller picture on the newer Qwen releases, our Qwen 3.7 on Ollama honest review explains why 3.7 is API-only and goes deeper on where the open-weight Qwen line holds up and where it doesn't.

The recommendation (by use case)

Here's the opinionated take.

Choose Gemma 4 12B if:

• Your agent processes multimodal input (images, audio, video). Qwen 3.5 9B has no audio at all, and nothing else Gemma 4 ships above the 12B takes audio either.
• You have 16 GB VRAM or unified memory available and don't need the last 3 GB for other processes.
• Your agent runs reasoning-heavy workflows where quality matters more than generation speed.
• You want a single model for everything instead of a text model plus a separate vision model.

Choose Qwen 3.5 9B if:

• Your agent works with text, images, or video and doesn't need audio understanding.
• You're running on constrained hardware (8 GB VRAM, older GPUs) where every parameter counts.
• Your agent handles high-volume, lower-complexity tasks (classification, extraction, summarization) where speed matters more than reasoning depth.
• You want the Gated DeltaNet efficiency gains on long-context workloads.

Choose neither (use an API instead) if: Your agent handles high-stakes tasks where accuracy is critical. Complex multi-step reasoning. Legal or medical content. Financial decisions. For these, a frontier model via BYOK (Claude Sonnet at $3/M, GPT-5.5 at $5/M) is worth the API cost. One analysis found local models handle about 60-70% of typical developer automation tasks at comparable quality to paid APIs. The other 30-40% still needs frontier capability.

What about Qwen 3.6, 3.7, and 3.8?

Fair question, and increasingly the right one. Qwen ships fast, and 3.5 is no longer the front of the line. But the three newer releases are very different propositions for someone running locally.

Qwen 3.6 (April 2026) shipped open weights at two sizes: a 35B-A3B mixture-of-experts model that activates only about 3B parameters per token, and a dense 27B released on April 22. Both are on Ollama (qwen3.6:35b-a3b, qwen3.6:27b), but both want roughly 17-24 GB at Q4. If you're choosing today and have that much memory, either is a better bet than a dense 9B. Our Gemma 4 31B vs Qwen 3.6 27B comparison covers that tier.

Qwen 3.7 (Max and Plus, May and June 2026) is API-only. There are no open weights and no ollama pull qwen3.7. If a guide gives you one, it's wrong. Our Qwen 3.7 Ollama honest review covers what to run locally instead, and Qwen 3.7 vs Claude Sonnet 4.6 covers whether it's worth using over the API for your workload.

Qwen 3.8 is the current open-weight line. Alibaba released Qwen3.8-27B on August 14, 2026: dense, Apache 2.0, 262K native context, text, image, and video input, thinking on by default. It's on Ollama as qwen3.8:27b (about 18 GB). If you're setting up from scratch rather than maintaining something already running on 3.5, and you have the memory, start there. One caveat before you wire it into an agent: some agent integrations hang on Ollama's /v1 endpoint with it, which our Qwen3.8 27B tool-calling fix walks through.

So is Gemma 4 12B vs Qwen 3.5 9B still a live comparison? Yes, for two reasons. Neither Qwen 3.6 nor Qwen 3.8 ships an open model below 27B, so Qwen 3.5 9B is still the newest Qwen you can run on an 8-12 GB machine, and dense models are simpler to serve, quantise, and reason about than MoE. And if you're already running 3.5 in production, the upgrade question is genuinely open. 3.6 and 3.8 are better, but they need twice the memory, and they're not so much better that a working agent needs rebuilding around them this quarter.

If you're comparing Gemma 4 12B against the newer Qwen releases specifically, the shape of the answer doesn't change much: Gemma still owns multimodal (neither 3.6 nor 3.8 takes audio), Qwen still owns efficiency per parameter, and the gap on reasoning has narrowed with each Qwen release.

Other models worth comparing

The two-model framing is a simplification. If you want the whole picture, our Gemma 4 vs Qwen 3.5 size-by-size comparison covers every model in both families, from Qwen's 0.8B up to its 397B flagship, and explains which pairings are actually comparable. A few others land in the same weight class and are worth a look before you commit:

• Ornith 9B — the closest direct peer to Qwen 3.5 9B on size, and increasingly showing up in the same shortlists. Worth testing head-to-head if tool-calling reliability is your bottleneck. Our Ornith-1.5-35B-A3B guide puts the 9B's published benchmarks next to both models, and covers the 35B MoE step up for 24GB cards.
• Phi 4 — smaller, sharper on reasoning per parameter, weaker on breadth. See our Phi 4 vs Sonnet comparison for where it fits.
• Gemma 4 E4B and Qwen 3.5 4B — if 8 GB is a hard ceiling and Q4 still isn't fitting, drop a tier rather than over-quantising a bigger model. A well-fed 4B beats a starved 9B.

If you're looking for alternatives to Gemma 4 12B specifically, the honest ranking is: Qwen 3.5 9B if you don't need audio, Gemma 4 26B A4B or 31B if you have the VRAM and want to stay in the family (there is no Gemma 4 27B), and a routed API setup if what you actually need is reliability rather than locality.

BetterClaw supports 28+ model providers via BYOK, including both Gemma (through Google AI Studio) and Qwen (through Alibaba Cloud or OpenRouter). The smart approach: route simple tasks to a local model and reserve API calls for high-stakes work. Our model routing setup covers this in detail. Free plan with 1 agent and 100 credits a month. $49/month on Pro. Zero inference markup.

The real question: does local even make sense for your agent?

Here's the perspective shift most comparison articles skip.

Running a local model means managing hardware, quantization, inference servers, and updates yourself. That's engineering time. For a solopreneur or small team, the time spent configuring llama.cpp or vLLM is time not spent building agent workflows. (If you do go local, our Ollama vs LM Studio breakdown covers which runner fits your workflow.) (Our guide on running a local LLM agent on consumer hardware covers the realities in depth.)

For privacy-sensitive workloads (medical data, financial records, proprietary code) where data cannot leave your infrastructure, local is the right call and Gemma 4 12B or Qwen 3.5 9B are excellent choices.

For everything else, the cost math often favors an API. Claude Sonnet at $3/M tokens, with prompt caching bringing that to $0.30/M for repeated context, costs less than the electricity and GPU depreciation of running a local model 24/7. The API model is always up to date. The local model freezes at its training cutoff.

Gartner projects 40% of enterprise applications will embed AI agents by end of 2026. Most of those will use APIs, not local models. But the 10-20% that need data sovereignty, offline capability, or zero-latency inference will increasingly choose models exactly like Gemma 4 12B and Qwen 3.5 9B. These are genuinely production-capable models on consumer hardware. That's a real shift.

Pick the model that fits your agent's actual needs. Not the one with the higher benchmark score.

Give BetterClaw a look if you want to skip the local model configuration and get your agent running in 60 seconds. Free plan with 1 agent and 100 credits a month. $49/month on Pro. 28+ providers via BYOK including Google AI Studio (Gemma) and OpenRouter (Qwen). We handle the infrastructure. You handle the agent logic.

Frequently Asked Questions

What is the main difference between Gemma 4 12B and Qwen 3.5 9B for agents?

Gemma 4 12B (11.95B parameters, June 2026) is the first mid-sized model to natively process text, images, audio, and video without separate encoders. It requires ~6.6 GB VRAM at Q4 and excels on reasoning-heavy benchmarks. Qwen 3.5 9B (March 2026) is a leaner model that handles text, image, and video (but not audio), uses less VRAM, and is faster on generation throughput thanks to its Gated DeltaNet hybrid architecture. Both support native tool calling and are Apache 2.0 licensed.

Is there a Qwen 3.5 12B?

No. Qwen 3.5 ships at 0.8B, 2B, 4B, 9B, and 27B, plus 35B-A3B, 122B-A10B, and 397B-A17B mixture-of-experts models — there is no 12B in the family. The 12B is Gemma 4's mid-size variant, which is why "Qwen 12B" searches usually end up on Gemma comparisons. If you want the Qwen model closest in size to Gemma 4 12B, that's Qwen 3.5 9B, and that pairing is exactly what this article compares.

Which quantisation should I use for Gemma 4 12B and Qwen 3.5 9B?

For most agent workloads, Q4_K_M on either model. Gemma 4 12B has an advantage here because Google ships quantisation-aware trained (QAT) weights, so its Q4 build stays much closer to full-precision quality than a standard post-training quantisation does. Q4_K_M needs roughly 6.6 GB for Gemma 4 12B and 5.1 GB for Qwen 3.5 9B, weights only, before KV cache. If you have 12 GB or more, Q5_K_M recovers most of what Q4 gives up on Qwen for about 1.2 GB extra. Q8_0 (12.5 GB and 9.6 GB respectively) is effectively lossless but rarely worth the memory for agent tasks.

Which local model is better for AI agent tool calling?

Both Gemma 4 12B and Qwen 3.5 9B support native tool calling and both are reliable for structured single-tool calls. Gemma 4 12B edges ahead on multi-step reasoning chains (due to its step-by-step reasoning mode and 3B more parameters), while Qwen 3.5 9B is faster per step and more efficient on constrained hardware. The Qwen 3.5 family scored 72.2 on BFCL-V4 for function calling (measured on the 122B variant), outperforming GPT-5 mini by 30%.

Can I run Gemma 4 12B on 8 GB VRAM?

Technically yes, at aggressive quantization (Q4KM brings it to ~6.6 GB). However, 8 GB leaves minimal headroom for context processing. Google recommends 16 GB for comfortable operation. An RTX 3060 12 GB works at Q4 with some headroom. For 8 GB cards, Qwen 3.5 9B is the safer choice as it leaves more room for context and batch processing.

How much does it cost to run these models locally vs using an API?

Hardware cost is one-time (or depreciated): a Mac Mini M4 with 16 GB costs around $600. Electricity is minimal. The trade-off is setup and maintenance time. API comparison: Claude Sonnet costs $3/M input tokens, but with prompt caching drops to $0.30/M for repeated context. For agents processing under 1,000 requests per day, API costs are typically $5-30/month, which is comparable to the electricity and depreciation of local inference. Local makes financial sense at high volume (5,000+ daily requests) or when data sovereignty requires it.

Should I use a local model or a cloud API for my AI agent?

Use local models when data cannot leave your infrastructure (medical, financial, proprietary), when you need zero-latency inference, or when you're running high-volume workloads where API costs compound. Use cloud APIs when accuracy on complex reasoning matters most, when you want always-current models, or when your team's time is better spent on agent logic than infrastructure. The hybrid approach (route simple tasks locally, reserve API for complex reasoning) captures the best of both.

Should I use Qwen 3.6, 3.7, or 3.8 instead of running 3.5 9B?

If you're setting up from scratch and have roughly 18 GB or more, start with Qwen 3.8 27B (qwen3.8:27b on Ollama). It's the current open-weight line, released August 14, 2026. Qwen 3.7 is not an option locally: it's API-only, with no open weights and no Ollama tag. Qwen 3.6 ships at 27B and 35B-A3B. If you're on 8-12 GB, Qwen 3.5 9B is still the right Qwen, because neither 3.6 nor 3.8 has an open model below 27B. And if you already have an agent running on 3.5 9B, there's no urgency: the newer models are better, but not by enough to justify rebuilding a working system this quarter.

What are the alternatives to Gemma 4 12B?

Qwen 3.5 9B is the main one if your agent doesn't need audio, since it's leaner and faster for text, image, and video work. Gemma 4 26B A4B (mixture-of-experts, 3.8B active) and 31B (dense) are the in-family step ups if you have 24 GB or more; there is no Gemma 4 27B. Ornith 9B and Phi 4 are worth testing in the same weight class, particularly if tool-calling reliability is your bottleneck. And if the real requirement is accuracy rather than running locally, a frontier model via BYOK will beat every option on this list.