Contents
Over the weekend one story climbed to the top of Hacker News and stayed there: somebody ran a 125 billion parameter AI model on a single RTX 4090 gaming card at more than 100 words a second. A year ago that size of model meant a rack of server GPUs. The phrase doing the rounds is "the VRAM barrier is dead", and I have now seen it in three separate GitHub projects, a viral post on X and a dozen comment threads.
I spent two days pulling the claim apart and trying it on my own desktop. The short version: the trick is real and clever, but the hardware it actually needs is not the hardware the headline implies. If you own a gaming PC, this post tells you whether you are in the club, and if not, what the one upgrade that gets you there would be.
What people are actually running
The model is Qwen3.8-Flash-Next, from Alibaba's Qwen team. It was released as open weights on 26 August 2026 as a research preview of the architecture Alibaba plans to use for Qwen 4. Anyone can download it from Hugging Face or ModelScope for free under the Qwen Community License, and it already has one of the busiest pages on Ollama. (If "open weights" and "Hugging Face" are new to you, my Hugging Face explainer covers both in ten minutes.)
The headline numbers from the model card:
- 125 billion total parameters, plus a separate 51 billion parameter "n-gram" lookup table.
- 6 billion active parameters per word. You will see this written as "125B-A6B".
- 512 experts, of which 10 plus one shared expert work on each word.
- A 262,144 token context window, which is roughly 400 pages of text in one conversation.
- Text, image and video understanding in one model.
The full-precision download is about 360GB, which is obviously not going on a gaming PC. What changed over the last two weeks is that a string of hobbyists published working recipes for squeezing it onto a single 24GB graphics card. One project measured 110 to 120 words a second on an RTX 4090. Another runs it on an older RTX 3090 at 43 to 51 words a second. A third got a quarter-million-token conversation going on a 4090 at a more modest 21 words a second. Those are the posts you have been seeing.
Mixture of experts, explained with a hospital
To understand why a 125B model can run at the speed of a small one, you need one idea: mixture of experts, usually shortened to MoE.
Picture a hospital with 512 specialists on the payroll. When a patient walks in, a triage nurse reads the chart and sends them to the ten specialists who are relevant, plus one general physician who sees everyone. The other 501 specialists are still employed, still in the building, but they do no work on that patient.
That is exactly how Qwen3.8-Flash-Next handles each word. A small routing network (the triage nurse) looks at the word, picks 10 of the 512 experts plus the shared one, and only those 6 billion parameters do any maths. Total staff: 125 billion. Staff actually working on any given word: 6 billion.
The consequence is a model that thinks with the speed of a 6B model but knows what a 125B model knows. That is the whole appeal.
Compare that with a dense model such as Qwen3.8 27B, the one I tested in my quantization guide. Dense means every single one of the 27 billion parameters works on every single word. It is more work per word than Flash-Next does, even though the file is far smaller.
So where is the catch? The hospital still needs a building big enough for all 512 specialists. In computer terms: every expert has to be loaded into memory somewhere, even though most of them sit idle. The maths is cheap. The memory is not.
The "VRAM barrier" trick: hot and cold experts
The old rule of local AI was simple: the model file has to fit inside your graphics card's memory (VRAM), with a little room to spare. A 24GB card comfortably runs a 17GB file. It does not run an 83GB file, and that is roughly what Flash-Next weighs at the 3-bit quantization the 4090 project used. Even the aggressive 1-bit version is 75GB. By the old rule, a 24GB card cannot run this model at all.
The new trick exploits the fact that experts are not used equally. Some get picked constantly. Many get picked rarely. So the software does this:
- Hot experts go on the GPU. The RTX 3090 project keeps 32 of the most-used experts per layer in VRAM, along with the parts of the model that run on every word.
- Cold experts go in system RAM. Ordinary DDR4 or DDR5 memory holds the thousands of experts that are only needed now and then, and the CPU does the maths for those.
- The rarest experts stay on the SSD and are streamed in when a conversation really needs them.
On top of that sits a second trick called multi-token prediction (MTP). Instead of producing one word at a time, the model drafts several words ahead and then checks them in a single pass, which the Unsloth team measured at 1.3 to 1.7 times faster output with no change to what the model would have said.
Here is the part the headline leaves out. The graphics card is not doing this alone. The machine that hit 110 words a second had 192GB of DDR5 system RAM. The RTX 3090 recipe lists 64GB of RAM as the minimum and 96GB as the amount that keeps every expert in memory. The barrier did not die. It moved from your graphics card to your motherboard.
What happened when I tried it on my own PC
My test machine is a desktop with 32GB of RAM and no dedicated graphics card, the same box I used for the quantization tests. I knew it was going to be a stretch. I wanted to see how it fails.
I downloaded the smallest file available, the 75GB 1-bit build, and pointed LM Studio at it. LM Studio flagged the file as too large for my memory before I even clicked load, and when I overrode the warning, it spent eleven minutes paging the model in from the SSD and then produced text at well under one word a second. A single paragraph took longer than making a coffee. That is not a usable experience, and it is not LM Studio's fault. There is no amount of software cleverness that fits 75GB of experts into 32GB of RAM.
Then I did what I should have done first and reloaded the 4-bit Qwen3.8 27B dense model. It ran at just under five words a second on the processor, exactly as it did in September. On a machine like mine, the 27B is the real option and Flash-Next is a spectator sport.
That experience shaped the table below.
Which hardware can run what
Tier 1: a normal laptop, 8GB to 16GB of RAM, no graphics card. Flash-Next is out of the question. Stick with the 4B to 12B models from my local AI starter guide, or run a small model on your phone. They are good, and they are fast.
Tier 2: a 32GB desktop, no graphics card or a small one. This is me. The 27B dense model at 4-bit works, at about five words a second on the CPU. Flash-Next does not fit.
Tier 3: a 16GB graphics card, such as an RTX 5060 Ti. The 27B at 3-bit fits in VRAM and runs at around 46 words a second according to one detailed write-up. That is a very nice local assistant. Flash-Next still does not fit.
Tier 4: a 24GB graphics card (RTX 3090, 4090 or similar) plus 64GB of RAM. Welcome to the club, barely. The 3090 recipe runs Flash-Next at 43 to 51 words a second with 128K of context. Be aware this is a weekend project: Linux, Docker, a 130GB download and a first start that compiles code. 96GB of RAM makes it comfortable.
Tier 5: a 24GB card plus 128GB to 192GB of RAM, or a Mac with 128GB or more of unified memory. This is where the 110-plus words a second numbers come from. On Apple silicon the Ollama MLX build is a 105GB file, so 128GB is the realistic floor, and Mac owners at that level report anywhere from 50 to well over 100 words a second depending on the chip.
Notice what separates Tier 3 from Tier 4. It is not a better graphics card. It is RAM. When I wrote how much RAM you actually need, I said almost nobody had a reason to go past 32GB. Local MoE models are the first mainstream reason for a home user to buy 64GB or more.
Should you bother?
Three honest considerations.
Quality. Alibaba's own benchmark table puts Flash-Next ahead of Claude Opus 4.6 Max on SWE-bench Pro (62.5 versus 53.4) and CoWorkBench (73.9 versus 68.2). Those are Alibaba's numbers on Alibaba's chosen tests, and independent evaluation of the research preview is still thin, so treat them as a claim rather than a fact. More telling for home use: the 4090 project ran its own checks on the squeezed 3-bit version and got 28 of 30 coding functions correct and 115 of 140 general knowledge questions right. Squeezing a model does cost something.
Privacy. Nothing leaves your machine. If you have followed my guide on stopping ChatGPT, Claude and Gemini from training on your chats, you already know how much of a hassle that is with cloud services. A local model skips the problem entirely.
Cost. This is where the romance fades. A 24GB card plus 64GB of RAM is a four-figure build. Alibaba's hosted version of the same model is priced at $0.16 per million input tokens and $0.47 per million output tokens. You could send it the entire text of this blog every day for a year and not spend what the RAM costs. If your motive is money, cloud wins by a mile. Local wins on privacy, offline use and the sheer fun of it.
One gotcha for anyone who does take the plunge: the model defaults to its highest "reasoning effort" setting, and on home hardware that can turn a one-line question into twenty minutes of thinking. Set it to low or medium before your first prompt.
The verdict
The "VRAM barrier is dead" headline is half right. Mixture of experts really does let a 125B model think at 6B speed, and the hot-and-cold expert trick really does let a 24GB gaming card host it. But the barrier did not disappear. It moved to system RAM, and the 110 words a second machines are carrying 192GB of it.
My recommendations:
- Own a 24GB card already? Do not buy a new GPU. Buy RAM. 64GB gets you in the door, 96GB makes it pleasant.
- Own a 16GB card or a 32GB desktop like mine? Run Qwen3.8 27B at 4-bit and be happy. It is the best local model most people will ever need, and it just works.
- Everyone else? Nothing has changed for you this month, and that is fine. Use the small models locally and the cloud for the heavy lifting.
The exciting part is the direction. Twelve months ago a 125B model was a data centre product. Today it is a weekend project for a well-stocked gaming PC. The trend line points at ordinary desktops, and I will be testing every step of the way.
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.