NOTE: This post documents our journey with Qwen3.5-397B-A17B in UD-TQ1_0 (1-bit) quantization. After successfully running MiniMax M2.5 and GLM-4.7, we pushed further into extreme quantization territory with this 397B-parameter MoE model.

Predominantly vibe-coded with Qwen3.5-397B in Opencode.

The Backstory

Hardware and Model

System Specs

• GPUs: 6x NVIDIA RTX 3090 (24GB VRAM each, 144GB total)
• CPU: AMD Threadripper 5995WX
• RAM: 64GB DDR4-3200 ECC
• PCIe: Full 4.0 16x speed on all GPUs
• Framework: llama.cpp server (build b8137+)

Model Details

Qwen3.5-397B-A17B is the largest Mixture of Experts model we've run locally:

llama.cpp Compatibility

Error: Unknown (built-in) filter 'items' for type String

Build Configuration

cmake /home/tomwest/llama.cpp -B /home/tomwest/llama.cpp/build \
  -DBUILD_SHARED_LIBS=OFF \
  -DGGML_CUDA=ON \
  -DGGML_CUDA_FA_ALL_QUANTS=ON
cmake --build /home/tomwest/llama.cpp/build --config Release -j$(nproc) --clean-first

GGML_CUDA_FA_ALL_QUANTS=ON compiles Flash Attention kernels for all KV cache quantization types (q8_0, q4_0, etc.), not just f16/f32. This is required for efficient inference when using quantized KV cache.

Configuration Details

Startup Script

#!/bin/bash

MODEL_PATH="/home/tomwest/models/qwen-3.5/Qwen3.5-397B-A17B-UD-TQ1_0.gguf"
LLAMA_SERVER="$HOME/llama.cpp/build/bin/llama-server"
LOG_FILE="$HOME/models/qwen-3.5.log"

nohup "$LLAMA_SERVER" \
    -m "$MODEL_PATH" \
    --host 0.0.0.0 \
    --port 8000 \
    -ngl 999 \
    -c 262144 \
    --parallel 1 \
    -ctk q8_0 \
    -ctv q8_0 \
    --split-mode layer \
    --temp 0.6 \
    --top-p 0.95 \
    --top-k 20 \
    --jinja \
    "$@"

Key Parameters

• -ngl 999: Offload all 61 layers to GPU
• -c 262144: Full native 262K context window
• --parallel 1: Single slot—all KV cache budget to one user
• -ctk q8_0 -ctv q8_0: Quantize KV cache keys and values (halves KV memory)
• --split-mode layer: Layer-wise GPU split (equal distribution)
• --temp 0.6 --top-p 0.95 --top-k 20: Recommended for thinking mode
• --jinja: Native Jinja template engine (required for Qwen3.5)

Benchmark Results

We ran comprehensive benchmarks across context lengths (4K to 262K) at 350W power limit per GPU. Here are the results:

Performance Visualization

The following charts visualize the performance characteristics across different context lengths:

Prefill Speed vs Context Length

Prefill phase performance across context lengths (4K to 262K). Four panes: throughput vs context, efficiency vs context, energy efficiency, and power range.

Decode Speed vs Context Length

Decode phase performance across context lengths (4K to 262K). Six panes: throughput vs context, efficiency vs context, energy efficiency, power range, TTFT, and efficiency vs power.

Memory Analysis

The UD-TQ1_0 quantized model is ~88GB, fitting in 144GB VRAM with room for KV cache:

opencode Integration

Added to ~/.config/opencode/opencode.json under the llama-cpp provider:

{
  "Qwen3.5-397B-A17B-UD-TQ1_0.gguf": {
    "name": "Qwen3.5-397B",
    "contextLength": 262144,
    "supportsStructuredOutputs": false
  }
}

Performance Benchmarks

Context sweep benchmark results at 350W/GPU:

Prefill Phase

Prefill throughput peaks at smaller context lengths and gradually declines:

• 4K-8K: ~340-350 tokens/sec
• 16K-32K: ~300-320 tokens/sec
• 65K-131K: ~250-280 tokens/sec
• 262K: ~200-230 tokens/sec

Decode Phase

Decode throughput shows similar patterns:

• 4K-8K: ~42-43 tokens/sec
• 16K-32K: ~38-40 tokens/sec
• 65K-131K: ~30-35 tokens/sec
• 262K: ~26-28 tokens/sec

Would I Recommend This?

TL;DR: Absolutely—if you need the largest possible model. Qwen3.5-397B-A17B in UD-TQ1_0 proves that 1-bit quantization is viable for massive MoE models. After the Jinja template compatibility issues were resolved, the model runs rock-solid: no crashes, no slowdowns, just reliable inference at 26-43 tokens/sec depending on context length.

The 397B parameter count is the largest we've attempted yet, and the MoE architecture keeps it tractable on consumer hardware. With only ~17B active parameters per token, it computes efficiently despite the massive total size. The 262K context window is fully utilized, making this ideal for long-context work.

Qwen3.5-397B-A17B UD-TQ1_0 on sextet RTX 3090s demonstrates that MoE architecture changes everything. With 397B total parameters and ~17B active per token, you can run models locally that would otherwise require data center GPUs. The model quality and throughput are both satisfying for daily use.

The configuration is rock-solid. Native Jinja template. No crashes. No slowdowns. Just fast, reliable inference on consumer hardware. After proving 2-bit and 1-bit quantizations viable, this is the largest model we've run yet—and it works beautifully.