TL;DR: After extensively testing Qwen3.5-27B and 35B-A3B on vLLM (BF16) against Qwen3.5-122B on llama.cpp (Q6_K_XL), the quantized big model wins decisively for single-user workflows. But the journey revealed surprising constraints that matter more than raw benchmark numbers.

When Qwen3.5 dropped in late 2024, it brought an unusual family: models from 0.6B to 397B parameters sharing the same hybrid Gated DeltaNet plus MoE architecture. We had six RTX 3090s (144GB VRAM) sitting idle. The question wasn't just which model is fastest, but what actually works for daily coding with opencode.

This is the story of what we learned about inference backends, quantization tradeoffs, and why the obvious choice turned out to be wrong.

The Three Contenders

We tested three configurations that actually fit on consumer hardware:

• Qwen3.5-27B - BF16 on vLLM (approximately 56GB, 4 GPUs)
• Qwen3.5-35B-A3B - BF16 or FP8 on vLLM (approximately 70GB, 4 GPUs)
• Qwen3.5-122B-A10B - Q6_K_XL on llama.cpp (approximately 105GB, 6 GPUs)

At first glance, the full-precision models should win. Higher precision means better quality, right? And vLLM is the production backend with better concurrency. But the reality was more nuanced.

The vLLM Tensor Parallelism Trap

This single constraint changed everything. For Qwen3.5-27B, the hidden_size is 5120 and num_attention_heads is 24. Neither is evenly divisible by 6, but both work with 4.

vLLM's tensor parallelism implementation requires all key dimensions to be evenly divisible by the GPU count. Our 6-GPU setup couldn't use TP=6. TP=4 was the only option, leaving 2 GPUs completely idle.

llama.cpp has no such constraint. Its layer-split strategy works with ANY GPU count. The model distributes layers evenly across all 6 GPUs, regardless of hidden dimension divisibility. This is why llama.cpp uses 100% of our hardware while vLLM uses 67%.

The Setup Journey

vLLM with Qwen3.5-27B (BF16)

Getting vLLM working required some wrestling. vLLM 0.17.1 needs transformers version 5.3.0 for layer_type_validation. The 4.x series doesn't have it. Default 90% GPU memory utilization caused OOM during KV cache profiling. We settled on 80% to leave headroom. We kept GPU 0 for the llama.cpp server on port 8000, so vLLM used GPUs 1-4.

vLLM with Qwen3.5-35B-A3B (MoE)

The MoE model introduced new challenges. Fused MoE kernel config was missing for RTX 3090, causing fallback to untuned tile sizes. We had to run a tuning script. Initial launches had the --enforce-eager flag which disabled FlashInfer optimizations, dropping speed to about 15 tok/s. Removing it restored about 23 tok/s. Default 2048 sequences caused CUDA graph OOM. We set max-num-seqs to 32 to reduce recurrent state buffers.

Interestingly, the MoE model drew significantly less power (about 750W) than the 27B dense (about 1100W) at similar throughput. Fewer active FLOPs per token.

llama.cpp with Qwen3.5-122B-A10B (Q6_K_XL)

The llama.cpp setup was surprisingly simple. The build required GGML_CUDA_FA_ALL_QUANTS=ON to enable Flash Attention for quantized KV cache types. Without it, q8_0 KV would fall back to a slower path.

Key parameters include offloading all layers to GPU, full native 262K context, quantized KV cache to halve memory, derating GPU 0 for desktop workload, and native chat template support.

The Multimodal Adventure

Qwen3.5 supports vision input. The setup differed dramatically between backends.

llama.cpp: Manual mmproj

We initially got the error "image input is not supported" because we forgot the projector. The --mmproj flag is required. We downloaded it from Unsloth's HuggingFace repo. Three variants exist: F16 (867MB), BF16 (871MB), F32 (1.7GB). We used BF16.

vLLM: Automatic

vLLM automatically loads the vision encoder if it's bundled with the model directory. No separate flags needed. The logs showed "Encoder cache will be initialized" and "Using backend FLASH_ATTN for vit attention".

opencode Configuration

For multimodal support in opencode, we had to explicitly enable it with a modalities configuration. The modalities text and image config is critical. Without it, opencode won't send images even if the backend supports them.

What Actually Worked (and What Didn't)

SGLang: Dead End

We attempted SGLang 0.5.9, but it doesn't work with Qwen3.5's hybrid GatedDeltaNet architecture. The framework expects standard transformer patterns and crashes on the recurrent DeltaNet layers. Outputs would immediately EOS regardless of the prompt.

FP8 on 122B: Impossible

We considered FP8 for the 122B model on vLLM, but it's mathematically impossible. FP8 weights would be 122GB total. TP=4 split would be about 30.5 GB per GPU plus KV cache. Available VRAM is 24 GB per GPU. Result: Permanent OOM. Even if an FP8 122B model existed, it wouldn't fit on 4x3090 with valid TP configuration.

Expert Parallelism: Not Worth It

We tested Expert Parallelism (EP) for the 35B-A3B MoE model. EP only pays off at larger batch sizes (compute-bound regime). At batch=1, the model is latency-bound, and EP doesn't help. Plus, RTX 3090s have PCIe communication (not NVLink at 4-GPU scale). EP all-to-all adds latency that cancels any savings.

The Qualitative Tradeoffs

Beyond numbers, how do these setups feel in daily use?

For speed, the 27B BF16 on vLLM delivers about 23 tok/s, the 35B-A3B BF16 on vLLM also delivers about 23 tok/s, and the 122B Q6_K_XL on llama.cpp delivers about 48 tok/s.

For quality, the 27B offers good reasoning, the 35B-A3B offers good knowledge breadth, and the 122B offers excellent all-around performance.

For tool calling, all three are reliable, with the 122B being very reliable.

For setup complexity, the 27B is medium (transformers 5.x, TP constraints), the 35B-A3B is high (MoE tuning, --enforce-eager bug), and the 122B is low (single binary).

For GPU utilization, both vLLM setups use 67% (4 out of 6 GPUs), while llama.cpp uses 100% (all 6 GPUs).

For concurrency, vLLM excels with excellent batching, while llama.cpp is poor with single-request focus.

For multimodal, vLLM is automatic while llama.cpp requires manual mmproj configuration.

For power, the 27B draws about 1100W, the 35B-A3B draws about 750W, and the 122B draws about 920W.

The Surprising Winner

The quantized 122B model outperformed the full-precision smaller models in every way that mattered for our workflow.

Quality

Q6_K_XL quantization preserves most of the original model's quality. The 122B's higher parameter count shows in complex tasks: tool calling is more reliable, code generation is more accurate, and long-context retrieval is more precise.

Speed

Despite having 4.5x more parameters than the 27B, the 122B Q6_K_XL on llama.cpp delivers 2x the decode speed. This is because llama.cpp uses all 6 GPUs (144GB VRAM) while vLLM uses only 4 GPUs (96GB VRAM).

Reliability

The llama.cpp setup was rocksteady. No OOM crashes, no CUDA graph issues, no MoE tuning required. Just start the server and go.

When vLLM Wins

vLLM isn't wrong—it's just optimized for different use cases.

For solo coding with opencode, use llama.cpp plus 122B Q6_K_XL for fastest single-request and best quality.

For multi-user API, use vLLM plus 27B or 35B since it handles concurrency well.

For batch processing, use vLLM plus 27B or 35B since it's optimized for throughput.

For parallel sub-agents, use vLLM plus 27B or 35B since it's designed for this workload.

For power efficiency, use vLLM plus 35B-A3B since it has the lowest watts per token.

For our use case (solo coding with opencode), concurrency doesn't matter. We run one request at a time. llama.cpp's single-request speed wins.

The Final Decision

We keep the vLLM setup around for experimentation and multi-user scenarios, but llama.cpp plus 122B is our daily driver. The speed difference is noticeable, the quality is superior, and the setup is rocksteady reliable.

The lesson? Quantization choice depends on your priorities. If you need multi-user concurrency, vLLM with full-precision models makes sense. If you're a solo user prioritizing speed and quality, quantized big models on llama.cpp are the way to go.

And yes, this blog post was drafted with help from the Qwen3.5-122B Q6_K_XL model itself. It's good enough to write about itself.