Two small models. Both run on a laptop. Both support tool calling. Both Apache 2.0. The right choice depends on what your agent actually needs to do.
I was testing a customer support agent on my M2 MacBook Pro last Tuesday. No API calls. No cloud dependency. Just a local model classifying tickets, extracting key fields, and drafting responses.
The model was Qwen 3.5 9B. It was fast. Accurate on text tasks. The VRAM footprint was small enough that I could run it alongside my IDE without the fans screaming.
Then Gemma 4 12B dropped on June 3rd. I swapped it in. Same agent, same prompts, same test data.
Here's what changed: the agent could now read attached screenshots from customer emails. Without a separate vision pipeline. Without any code change. Gemma 4 12B processes images, audio, and video natively in the same model that handles text. No encoder. No extra VRAM for a vision module. It just... works.
But it uses 3 billion more parameters. And for agents that only need text, those extra parameters are cost you don't need.
That's the real decision. Not "which model is better" but which model fits what your agent does. Here's the head-to-head breakdown.
The specs that matter (everything else is noise)
Both models dropped in early 2026 and both target the same sweet spot: small enough to run locally, capable enough for real agent work. (For the wider field of what you can run on your own machine this year, see our roundup of local AI in 2026.)
Gemma 4 12B (Google DeepMind, June 3, 2026): 11.95 billion parameters, dense architecture, encoder-free unified multimodal (text + image + audio + video), 256K context window, Apache 2.0. Runs in ~6.6 GB VRAM at Q4 quantization. Native tool calling with optional step-by-step reasoning mode.
Qwen 3.5 9B (Alibaba Qwen, March 2, 2026): 9 billion parameters, Gated DeltaNet hybrid architecture (3:1 linear-to-full-softmax attention), unified vision-language (text + image), 262K context window, Apache 2.0. Thinking and non-thinking inference modes. Multi-token prediction.
Both are instruction-tuned. Both support structured tool calling. Both are commercially usable. Both fit on a laptop.
The differences are in three dimensions that matter for agents: multimodal capability, memory footprint, and architecture efficiency.
Multimodal: Gemma 4 12B wins by a wide margin
This isn't close. Gemma 4 12B is the first mid-sized model to natively process text, images, audio, and video without separate encoders. The architecture projects image patches and audio waveforms directly into the shared decoder. No bolted-on vision encoder eating extra VRAM. No separate audio pipeline.
For agents, this means your local model can read a screenshot, listen to a voice note, and watch a short video clip without any additional infrastructure.
Qwen 3.5 9B supports text and image. No native audio. No video. For text-only or text-plus-image agent workflows, this is fine. But if your agent handles support tickets with screenshot attachments, voice memos, or video recordings, Gemma 4 12B is the only option in this weight class.
If your agent needs to process anything beyond text and static images, Gemma 4 12B is the only sub-15B model that handles it natively. That's the clearest differentiator.
VRAM and speed: Qwen 3.5 9B is leaner
Nine billion parameters vs twelve billion. That 25% size difference matters on consumer hardware.
Qwen 3.5 9B runs comfortably on GPUs with 8 GB VRAM at Q4 quantization. Gemma 4 12B needs ~6.6 GB at Q4KM but runs more comfortably with 16 GB. On machines where every gigabyte counts (an RTX 3060 with 12 GB, or an older MacBook), the 3 billion parameter difference translates to real headroom for context and batch processing.
Speed is more nuanced. Community testing on RTX 3060 found Gemma 4 12B has "overwhelmingly fast" prefill (input processing) speed. On M2/M3 MacBook Pro, Gemma 4 12B hits 30-50 tokens per second at Q4. Qwen 3.5 9B's hybrid architecture (with its linear attention layers) is designed for efficient inference on long sequences, and the smaller parameter count gives it an edge on token-per-second throughput at equivalent quantization.
For agents that process long documents or maintain extended conversations (where prefill speed matters most), Gemma 4 12B's fast prefill is an advantage. For agents that need fast generation on constrained hardware, Qwen 3.5 9B's leaner profile wins.
Tool calling and agentic performance: both are capable, differently
Both models support native tool calling. Both can parse function schemas, select the right tool, format arguments, and process results. But they approach it differently.
Gemma 4 12B includes a dedicated tool-calling mode and an optional step-by-step reasoning mode. When the reasoning mode is active, the model generates intermediate reasoning tokens before selecting and calling tools. This improves accuracy on multi-step tasks but increases token consumption per tool call.
Qwen 3.5 9B has thinking and non-thinking modes. In thinking mode, the model generates internal reasoning before responding. The Qwen 3.5 family was built with the same architecture as the 397B flagship, and the 9B variant matches or surpasses models 10-13x its size across several agentic benchmarks.
The honest assessment: for structured tool calling (single tool, clear schema, straightforward parameters), both are reliable. For complex multi-step agent workflows with chained tool calls, the larger Gemma 4 12B tends to hold up better in reasoning quality, while Qwen 3.5 9B is faster per step.
Benchmarks (with the usual caveats)
Benchmarks are reference points, not guarantees. Your agent's real-world performance depends on your prompts, your tool schemas, and your specific use case. But here's what the numbers show:
Gemma 4 12B benchmarks (Google-reported): MMLU Pro: 77.2%. GPQA Diamond: 78.8%. AIME 2026: 77.5%. Beats last year's Gemma 3 27B (67.6% MMLU Pro) at less than half the parameter count.
Qwen 3.5 9B benchmarks (Alibaba-reported): Matches or surpasses GPT-OSS-120B (a model 13x its size) across multiple language and vision benchmarks. Agentic index: 55.5 per independent evaluation. The 3.5 family's function calling capability (measured on the larger 122B variant) scored 72.2 on BFCL-V4, outperforming GPT-5 mini by 30%.
The benchmark numbers suggest Gemma 4 12B edges ahead on reasoning-heavy tasks (GPQA, AIME) while Qwen 3.5 9B punches above its weight on efficiency-per-parameter across general language and agent tasks.
Neither model will match a frontier API model (Claude Sonnet at $3/M tokens is significantly more capable than either for complex reasoning). The comparison that matters is these models against each other, on the workloads you'll actually run locally.
The recommendation (by use case)
Here's the opinionated take.
Choose Gemma 4 12B if:
- Your agent processes multimodal input (images, audio, video). This is the only sub-15B model that handles all four modalities natively. There's no close second.
- You have 16 GB VRAM or unified memory available and don't need the last 3 GB for other processes.
- Your agent runs reasoning-heavy workflows where quality matters more than generation speed.
- You want a single model for everything instead of a text model plus a separate vision model.
Choose Qwen 3.5 9B if:
- Your agent is text-only or text-plus-image and doesn't need audio/video understanding.
- You're running on constrained hardware (8 GB VRAM, older GPUs) where every parameter counts.
- Your agent handles high-volume, lower-complexity tasks (classification, extraction, summarization) where speed matters more than reasoning depth.
- You want the Gated DeltaNet efficiency gains on long-context workloads.
Choose neither (use an API instead) if: Your agent handles high-stakes tasks where accuracy is critical. Complex multi-step reasoning. Legal or medical content. Financial decisions. For these, a frontier model via BYOK (Claude Sonnet at $3/M, GPT-5.5 at $5/M) is worth the API cost. One analysis found local models handle about 60-70% of typical developer automation tasks at comparable quality to paid APIs. The other 30-40% still needs frontier capability.
BetterClaw supports 28+ model providers via BYOK, including both Gemma (through Google AI Studio) and Qwen (through Alibaba Cloud or OpenRouter). The smart approach: route simple tasks to a local model and reserve API calls for high-stakes work. Our model routing setup covers this in detail. Free plan with every feature. $19/month per agent on Pro. Zero inference markup.
The real question: does local even make sense for your agent?
Here's the perspective shift most comparison articles skip.
Running a local model means managing hardware, quantization, inference servers, and updates yourself. That's engineering time. For a solopreneur or small team, the time spent configuring llama.cpp or vLLM is time not spent building agent workflows. (Our guide on running a local LLM agent on consumer hardware covers the realities in depth.)
For privacy-sensitive workloads (medical data, financial records, proprietary code) where data cannot leave your infrastructure, local is the right call and Gemma 4 12B or Qwen 3.5 9B are excellent choices.
For everything else, the cost math often favors an API. Claude Sonnet at $3/M tokens, with prompt caching bringing that to $0.30/M for repeated context, costs less than the electricity and GPU depreciation of running a local model 24/7. The API model is always up to date. The local model freezes at its training cutoff.
Gartner projects 40% of enterprise applications will embed AI agents by end of 2026. Most of those will use APIs, not local models. But the 10-20% that need data sovereignty, offline capability, or zero-latency inference will increasingly choose models exactly like Gemma 4 12B and Qwen 3.5 9B. These are genuinely production-capable models on consumer hardware. That's a real shift.
Pick the model that fits your agent's actual needs. Not the one with the higher benchmark score.
Give BetterClaw a look if you want to skip the local model configuration and get your agent running in 60 seconds. Free plan with 1 agent and every feature. $19/month per agent on Pro. 28+ providers via BYOK including Google AI Studio (Gemma) and OpenRouter (Qwen). We handle the infrastructure. You handle the agent logic.
Frequently Asked Questions
What is the main difference between Gemma 4 12B and Qwen 3.5 9B for agents?
Gemma 4 12B (11.95B parameters, June 2026) is the first mid-sized model to natively process text, images, audio, and video without separate encoders. It requires ~6.6 GB VRAM at Q4 and excels on reasoning-heavy benchmarks. Qwen 3.5 9B (March 2026) is a leaner model that handles text and image, uses less VRAM, and is faster on generation throughput thanks to its Gated DeltaNet hybrid architecture. Both support native tool calling and are Apache 2.0 licensed.
Which local model is better for AI agent tool calling?
Both Gemma 4 12B and Qwen 3.5 9B support native tool calling and both are reliable for structured single-tool calls. Gemma 4 12B edges ahead on multi-step reasoning chains (due to its step-by-step reasoning mode and 3B more parameters), while Qwen 3.5 9B is faster per step and more efficient on constrained hardware. The Qwen 3.5 family scored 72.2 on BFCL-V4 for function calling (measured on the 122B variant), outperforming GPT-5 mini by 30%.
Can I run Gemma 4 12B on 8 GB VRAM?
Technically yes, at aggressive quantization (Q4KM brings it to ~6.6 GB). However, 8 GB leaves minimal headroom for context processing. Google recommends 16 GB for comfortable operation. An RTX 3060 12 GB works at Q4 with some headroom. For 8 GB cards, Qwen 3.5 9B is the safer choice as it leaves more room for context and batch processing.
How much does it cost to run these models locally vs using an API?
Hardware cost is one-time (or depreciated): a Mac Mini M4 with 16 GB costs around $600. Electricity is minimal. The trade-off is setup and maintenance time. API comparison: Claude Sonnet costs $3/M input tokens, but with prompt caching drops to $0.30/M for repeated context. For agents processing under 1,000 requests per day, API costs are typically $5-30/month, which is comparable to the electricity and depreciation of local inference. Local makes financial sense at high volume (5,000+ daily requests) or when data sovereignty requires it.
Should I use a local model or a cloud API for my AI agent?
Use local models when data cannot leave your infrastructure (medical, financial, proprietary), when you need zero-latency inference, or when you're running high-volume workloads where API costs compound. Use cloud APIs when accuracy on complex reasoning matters most, when you want always-current models, or when your team's time is better spent on agent logic than infrastructure. The hybrid approach (route simple tasks locally, reserve API for complex reasoning) captures the best of both.
