Janas runs MoE models bigger than RAM by streaming experts from SSD

Janas is an experimental, GPL-3 C toolkit whose Janas-LLM engine runs mixture-of-experts models larger than system RAM by keeping weights on NVMe and streaming the experts each token needs into a memory cache. Author Maurizio Cammalleri reports that on a 32 GB Debian laptop (Intel Core Ultra 9 185H, integrated Arc GPU), Qwen3-Next-80B (48.4 GB file) wrote at 24 tok/s without drafts, versus 2.8 for llama.cpp on the same machine, and read a 1,024-token prompt at 62 tok/s versus 16. With drafts in chat he reports about 37 tok/s for that model. He says Janas-LLM was faster than llama.cpp on every model compared in Q4_K_M on that laptop across 13 models. Linux and Windows x86-64 binaries are on GitHub releases; the README stresses it is early, API-unstable, and hard on free memory. On Hacker News (205 points, ~77 comments), many welcome another open local-inference engine and note SSD expert streaming is becoming common. A camp says llama.cpp already covers much of this and specialised engines often reimplement it for speed. Others debate whether 60 tok/s is enough for agents (prefill vs decode) and whether SSD wear or High Bandwidth Flash is the next step. No broad consensus beyond "useful experiment, measure on your own machine". Sorted take: promising for laptop MoE if you accept experimental software. Run janas-bench on idle hardware before you trust the headline speeds.