The Most Expensive Chip Ever Built Spends Its Time Waiting

All that silicon, fed through one thin line.

Imagine the fastest kitchen ever built. Every cook works at superhuman speed. The equipment cost millions. There's one problem: every ingredient arrives through a single mail slot.

That kitchen is an AI accelerator, and this is the least intuitive fact in the industry. The chip that everyone competes to make faster spends a great deal of its time waiting.

What has to be remembered before a single word appears

Before an AI model writes the next word, it needs three things in hand.

The weights — everything it learned during training, stored as billions of numbers. A 70-billion parameter model at 16 bits needs roughly 140 gigabytes just for those.

The activations — temporary scratch work created as information passes through its layers.

The conversation — a working notebook, technically the KV cache, holding what's already been said so the model doesn't have to reread the whole exchange for every new word. Every token adds an entry. Longer conversations mean thicker notebooks. More users mean more notebooks, all piling up around the same processor.

None of those are types of chip. They're information, and information needs somewhere physical to live.

Two jobs, two very different appetites

When your prompt arrives, the model reads all of it at once. Engineers call this prefill, and it's a burst of raw arithmetic — hundreds of cooks prepping different parts of a large order in parallel. That pause before the first word appears is prefill happening.

Then it starts writing, one token at a time. This is decode, and it's a completely different shape of work. Each token depends on the one before it, so it can't be parallelised away. For every single word, the machine streams the model's weights and the growing notebook through the processor again.

Lots of fetching, comparatively little arithmetic. The chip isn't switched off while it waits — it has issued a request and cannot continue until the data arrives.

Which is why quoting a processor's peak calculations per second tells you half the story at most. It's like advertising the world's fastest chefs without mentioning that the loading dock is one metre wide.

The ladder, and why every rung exists

The industry's answer is a hierarchy, and each level exists only because the one above it is too small or too expensive and the one below it is too slow.

Hands, countertop, fridge, pantry, warehouse. Each step away holds more and answers slower.

Registers and on-chip cache are the cook's hands. Retrieval in billionths of a second. They can hold what's needed this instant and nothing more — this memory is fast because it's built on the processor itself, and chip area is brutally limited.

High bandwidth memory is the countertop. Memory stacked vertically into towers and placed beside the processor in the same package, connected through an exceptionally wide interface. Its advantage isn't that each cell is magic; it's that thousands of lanes deliver at once. Latency is how long you wait for the first truck. Bandwidth is how many trucks can unload side by side. HBM turns the loading dock into a wall of doors.

Same cooks. One mail slot leaves them idle; a wall of doors keeps all of them busy.

Server DRAM is the refrigerator — much larger pools, much cheaper per bit, but further away through a narrower connection.

Flash storage is the pantry. It remembers without power, which none of the above do. Model files rest here before deployment; training checkpoints protect weeks of work.

Hard drives are the warehouse. Milliseconds to respond, which is an eternity to a processor, and unbeatable for enormous piles of data you rarely touch. The technology everyone declared dead is selling out, because AI doesn't only consume data — it produces it endlessly, and somebody has to store every conversation and checkpoint forever.

No rung is better than another. A countertop isn't better than a warehouse. Capacity is not speed, and persistence is not proximity.

Why the countertop is so hard to build

Instead of spreading memory outward across a board, HBM builds upward: DRAM dies thinned, stacked eight, twelve, up to sixteen high, and pierced by thousands of microscopic vertical connections running between the floors like elevator shafts. At the bottom sits a base die managing the interface. The whole tower goes on an interposer beside the processor.

Then comes the manufacturing bill. The dies have to be produced at extremely high yield. They must be thinned without breaking. Thousands of vertical connections must line up. The stack gets bonded, filled, cooled, tested, then packaged beside the accelerator.

If one floor in the tower fails, the value of the whole stack is damaged. If the package fails after the processor and several towers are assembled, an enormous amount of expensive silicon dies at once.

One cracked floor, and the eleven good ones go in the bin with it.

And there's a hidden cost that explains a lot about consumer prices: producing a given capacity of HBM consumes roughly three times the wafer volume of ordinary server memory. When a manufacturer shifts capacity toward HBM, it isn't simply making a premium product — it's using more of the factory to produce fewer total bits. Conventional memory gets tighter for everyone else at the same time.

Why this stopped being a commodity business

Ordinary memory is interchangeable. If one qualified supplier stumbles, a customer substitutes another. That's why memory spent decades breaking investors' hearts: every supplier sees the same shortage, every supplier expands, the new capacity all arrives together, prices collapse.

HBM is different in a specific way. Each product has to be co-designed and qualified with a particular accelerator and packaging system. Thermal behaviour matters. Signal integrity matters. Yield matters. Delivery timing matters.

The consequence is that a product announcement is not a qualification, and a qualification is not mass-production revenue. The winner isn't whoever publishes the best specification sheet — it's whoever delivers millions of known-good stacks on time, at acceptable yield, integrated with the customer's package.

Three companies make essentially all of it. One of them bet early, built the relationship with the dominant accelerator designer, and entered the boom already qualified. The largest memory manufacturer in the world was late, and its qualification delays became a public embarrassment. The smallest of the three went from afterthought to selling out capacity in advance.

Scale did not guarantee the win. Execution did.

The question that decides the next few years

Everything above explains why memory margins went somewhere they have never been. It does not explain whether they stay there.

The structural arguments are real: HBM isn't interchangeable, it sells through longer agreements, packaging capacity limits supply as much as wafers do, and serving millions of users happens every day, not once per model.

The counter-argument is the oldest one in the industry. All three suppliers see the same extraordinary incentive. All three are expanding. Yields improve, packaging lines get built, stacks get taller. Customers redesign models to use memory more efficiently.

Scarcity sends an invitation to the very supply that ends it.

Memory stopped being a supporting actor the moment feeding the processor became the limiting job. Whether it stays the main character depends on something much more boring than technology: how fast everyone builds.