AiGpu

Infrastructure·

Streaming Mixture‑of‑Experts Models from SSD: 180B Parameters on 32 GB RAM

A new expert‑streaming technique lets a 180‑billion‑parameter MoE model run on a single 32 GB GPU by keeping only active experts in VRAM and pulling the rest from fast storage, closing the performance gap with dedicated inference engines.

Diagram showing expert streaming from SSD to GPU VRAM for a large MoE model

Researchers have demonstrated that a 180‑billion‑parameter mixture‑of‑experts (MoE) model can be served on a single GPU with only 32 GB of memory by streaming expert weights directly from an SSD. The approach keeps the hot experts resident in VRAM while fetching cold experts on demand, achieving token‑generation speeds within 6 % of a specialized engine such as Strata.

Why it matters for GPU / AI infrastructure

For cloud providers and on‑premise clusters, this means larger MoE models no longer require multi‑GPU nodes or massive unified memory. Operators can pack more models per GPU, reduce hardware capital expenditure, and still deliver low‑latency inference for coding assistants, chat, and reasoning workloads.

  • Only routed experts move across the PCIe bus each step, cutting data movement.
  • SSD‑backed offload adds a third memory tier without page‑cache thrashing.
  • Compatible with existing Transformers and Accelerate pipelines, easing adoption.

The technique also works on Apple Silicon where CPU and GPU share memory, because the streaming layer bypasses the traditional offload‑to‑CPU path that frees no space on unified‑memory systems.

  • aigpu
  • ai gpu
  • ai gpu cloud
  • aigpu dubai
  • moe
  • inference
  • gpu cloud
  • expert streaming

By AiGpu Editorial · Editorial rewrite based on public reporting (Hugging Face Blog)

← All articles