The Observation

Right now, I'm running on a 230-billion parameter model (Minimax M2.5) quantized to 2-bits, on a rig with six RTX 3090s that cost about $8,000 to build. Certainly not cheap, but this conversation—hundreds of thousands of tokens—costs $0 to generate. No API bills. No rate limits. No subscriptions.

Three years ago, this would have required a data center. Today it fits on a desk.

This isn't a demo. It's not a toy. It's the new reality of LLM inference, and it's happening because of a convergence of technologies that normies and Wall Street hasn't fully grasped yet.

The Thesis

LLM inference is undergoing massive cost deflation.

Not just "prices went down"—actual, fundamental deflation in what it takes to run a capable AI. The combination of:

1. MoE architectures (Mixture of Experts) that keep compute constant while scaling parameters
2. Aggressive quantization (2-6 bits, dynamic quant) that shrinks model weight by 8-10x
3. KV cache optimization that enables 256K-1M token contexts on consumer hardware
4. Maturing serving infrastructure (llama.cpp, vLLM, SGLang, LMCache)

These improvements compound as each technology addresses a different bottleneck. Together, they make "good enough" local inference viable for anyone with a bit of determination and some used hardware.

And here's what's novel: unlike typical tech cycles where new product releases make old hardware irrelevant, software improvements are extending the productive life of existing hardware. This is actually what the biggest AI companies have been saying, and have been publicly derided for, with claims that this is a bit of financial engineering to make their books look better. That could be true, but consider this: the RTX 3090, a consumer gaming GPU was $1,600 in 2020. It still costs $800-1,000 used—and it's still a formidable inference card in 2026. That's not typical.

The Evidence

1. The Economics: 10x Annual Decline

According to introl.com, LLM inference costs declined roughly 10x annually—faster than PC compute or dotcom bandwidth. GPT-4-equivalent performance now costs around $0.40 per million tokens.

That's a 75x decline in three years.

But here's the twist: cloud APIs still cost money per-token. Own your hardware once, and marginal cost drops to ~$0.

2. The Technology: KV Cache is the Bottleneck

We reviewed six arxiv papers, and they all arrive at the same conclusion: KV cache is the bottleneck.

The Key-Value cache stores context during inference. At 256K context, KV cache alone can reach 100GB—more than most GPUs have. Six recent papers address this:

QuantMoE-Bench (arXiv:2406.08155v2)

• Mixed-precision quantization for MoE models
• Achieves 65.35% vs 64.30% baseline—averaged across 6 benchmark tasks, showing layer-wise precision allocation beats uniform quantization
• Key insight: attention layers need higher precision than FFNNs; shared experts need more bits than token-conditioned experts

MoQAE (arXiv:2506.07533v1)

• Uses MoE routing to select optimal quantization bit-width
• Reduces KV cache memory by 0.79GB (~10-15% of typical 128K context cache), and decoding latency by 0.44ms per token—critical for long-context workloads where KV cache can reach 100GB+
• Chunk-by-chunk token processing

ExpertFlow (arXiv:2410.17954v1)

• Predictive routing for MoE expert caching—forecasts which experts will be needed and pre-caches them, instead of loading all 8+ experts into VRAM
• 93.72% GPU memory savings via intelligent expert offloading between CPU/GPU
• 2-10x inference speed improvement through dynamic token scheduling that groups tokens with similar routing paths

MiniKV (arXiv:2411.18077v3)

• 2-bit KV cache with adaptive policies
• 98.5% of full-precision model accuracy with 50% KV cache size—i.e., you can halve your KV memory footprint while losing less than 1.5% accuracy
• Layer-wise pyramid structure (different strategies for different layers)

LMCache (arXiv:2510.09665v2)

• First production KV caching layer
• 15x throughput improvement with vLLM
• Prefix reuse: System prompts and RAG context stay cached across queries—no recomputation when you send multiple prompts with the same context. Massive speedup for chat workflows.
• Prefill-decode disaggregation: Processing input (prefill) and generating tokens (decode) can now run on different GPUs in parallel—effectively doubling throughput on multi-GPU rigs.

Online Scheduling (arXiv:2502.07115v5)

• Theoretical framework for KV-aware scheduling—originally designed for multi-tenant cloud, but the algorithm applies when running parallel agents (multiple conversation threads with different contexts) on a single rig
• Polynomial-time algorithm with constant competitive ratio, proven optimal for KV cache eviction decisions

The winning formula: quantization (2-4 bit) + adaptive selection + layer-wise structure.

This is what enables 256K-1M context on consumer hardware—and it's what we're running right now.

3. The Models: The "Kill Zone"

The sweet spot for self-hosted inference is small-to-mid models (20-40B effective) running on consumer or workstation GPUs. Not "best benchmark"—but "good enough for 90% of what people actually do."

Qwen3.5-122B-A10B: 122B parameters, only 10B active (MoE), runs on 6x3090 at Q6_K_XL quantization. Note: The whole Qwen3.5 family (27B, 35B, 122B, 397B) is great for local use—MoE architecture keeps active params low while quality stays high.

Minimax-M2.5: 230B MoE, runs Q2 on our 6x3090 cluster—the model that powers this conversation

GLM-4.7-Flash: 30B/3B active, strongest in 30B class, single GPU viable

GLM-4.7: 358B parameters, MoE, 200k context, IQ1_M quantization fits

Practical Rules for Local Inference

It's often better to go heavily quantized + bigger model than smaller model + less quantization. With some limits...

1. Q1 is a bit too nerfed, and often starts to fail spectacularly at higher context lengths.
2. A balance should be struck that trades off quantization levels vs. model size. I'm surprised that a Q2 Minimax has become my go-to model. I wouldn't have figured this out without trying it. But a smaller parameter model (like Qwen3.5-122B-A10B) gives more headroom, allowing us to preserve the model's capabilities by selecting a higher quant.
3. Anywhere between Q2 and Q6 probably good enough for government work (city + county, anyway).
4. Dynamic quantization gives a big boost to heavily quantized models. It helps close the gap between the "true" representation of a model, and what you get when you try to cram it onto constrained hardware.
5. MoE is the way for local use. You get faster inference, save energy and time per forward pass, and it's such a UX improvement that when you find a good model, it doesn't really matter if it's "dense" or not.
6. This self-hosted mid-market that requires pro-sumer type hardware has been typically underserved, but just in recent weeks we've had new releases in Minimax, GLM, Qwen, Nemotron model families. I expect this market will grow, much to our delight.

4. The Hardware: Value Preserved

The RTX 3090 still runs 32B+ quantized models with 256K context. Its productive life has been extended by software, not just replaced by newer hardware. This is unusual. This is the inversion of typical tech cycles.

5. The Break-Even Analysis

Cloud pricing based on mid-tier models (e.g., Claude Sonnet, Gemini 1.5 Pro). Local cost assumes ~$1/day electricity + hardware amortization over 3 years. Based on current API pricing trends [introl.com].

Once you're doing 100K+ tokens/day, owning your stack is structurally cheaper. And the tools we're using (Hermes, local inference, open models) make that usage natural.

What Wall Street Misses

The market fixates on frontier model provider costs—the $10 billion data centers, the NVIDIA purchases, the training runs. They see "AI needs gigawatts of compute" and conclude "you need to be Google to do this."

But they're missing:

1. The "good enough" tier: A Sonnet 4.6 model is appropriate for most use cases. Not everyone needs Opus all the time.
2. Hardware extends its life: Software improvements (quantization, KV optimization) preserve the value of existing GPUs. Case in point: The 3090 isn't dead—it's been revitalized.
3. Jevons Paradox in action: When inference becomes cheap, usage explodes. People consume more tokens because the marginal cost is near-zero and the output is genuinely useful.

This creates new demand, not just cheaper supply. The market is underweighting this because it's hard to model "software extending hardware life by 2-3x."

What Wall Street might be getting right: the infinite capex on these gigawatt-scale facilities may indeed be cooked. If "good enough" merely requires a Macbook, or even a smartphone, then most inference could happen on-device, and the AI companies may be relegated to using compute for training, and corporate superusers that need greater capabilities and speed. But does Joe Shlub need Opus to schedule his appointments, tell him the missing ingredient in butter & sketti, or set a reminder for his colonoscopy? Certainly not, and his iPhone can probably already do that.

The Future: Dam Bursting

When a "superhuman for most tasks" model fits on less than $1-2k hardware, the adoption curve goes vertical.

Not because everyone becomes an ML researcher. Just because it's there, it's virtually free, and it works. That's what happened with MP3s, streaming, the iPhone app store. Infrastructure commoditization precedes explosion.

We're not there yet. But we're a lot closer than the market thinks.

Conclusion

We're living the thesis. This conversation is happening on local inference. Six RTX 3090s, a Threadripper, Minimax M2.5 at Q2, and Hermes running as the agent layer.

Even building this website, which I've managed to do for about 1-2 hours each weekend would have cost me 100's of personal man-hours without AI, or a small fortune using fiverr/upwork contractors. I wouldn't have even built this site if I didn't have the technological leverage of these models behind me. I wouldn't have had the time.

The deflation is real. The technology exists. The economics already work. And they're getting better.

My appeal to you: buy that GPU (or Mac), use a lower quant, and run the model locally. As the open model providers increasingly serve this homebrew inference market, things will get better. We're probably still the early adopters, and there will be a chasm to cross. But it might be less of a leap, and more of a hop.

Just like TVs went from fat "Bigscreen" CRTs, to LCD, to OLED, to 70" plasmas on every normies' wall. Or how cell phones went from bricks, to flip phones, to touchscreens, to the everything device in your pocket. One day we will wake up with digital companions that help us do our work, expand our knowledge, and improve our lives. What a fooking time to be alive!

Written March 2026. Running locally on 6x RTX 3090 + Minimax M2.5 (Q2). No cloud APIs were harmed in the making of this post.