DeepSeek: How did DeepSeek train a frontier model so cheaply?

This is why a lab locked out of the best chips could still ship a model that rivalled OpenAI’s and Google’s, and why it rattled the assumption that more capital always wins — by refusing to treat memory as somebody else’s problem. Its latent attention is the most aggressive attack yet on how much state a single token has to carry.

The room it came out of

A Chinese lab under export controls in 2024, unable to buy its way out of a memory constraint at any price. So it attacked the constraint instead, and found a compression of attention state that had been sitting in plain sight for seven years while thousands of better-funded researchers read the same code. Nobody looks hard at a wall they can afford to pay somebody to move.

How it works

DeepSeek is worth a page here for a reason that has nothing to do with benchmark scores. It is the clearest case in the last few years of a lab that could not obtain compute on the terms its competitors could, and therefore had to treat a constraint as a research problem rather than a purchasing one.

Its most cited contribution is multi-head latent attention. Where grouped-query attention reduces how many key and value heads you store, latent attention changes what you store: the keys and values are compressed into a much smaller shared latent representation and reconstructed on read. It is the same idea as the rest of this thread — spend arithmetic to save state — pushed considerably further than anyone else had been willing to push it.

That move is in this atlas as an intervention, not as an anecdote. Switch it on in the model and state per token falls by about four times against grouped-query attention alone, which at long context is the difference between a machine that serves a hundred conversations and one that serves four hundred.

The strategic read is the interesting part, and it generalises past this one lab. Constraints do not slow research down evenly — they redirect it. A lab with abundant memory optimises what it is already doing; a lab without it goes looking for a different architecture. Several of the most-copied efficiency ideas of the last decade came from whoever had the least of the resource in question, and that is a pattern worth carrying into whatever the next scarce thing turns out to be.

The caution, stated plainly: reported training costs are not audited, the comparison figures circulated in the press mixed several different things together, and this atlas has no way to check any of them. What it can check is the architecture, which is published, and the architecture is genuinely different.

What it traded

gave up: architectural simplicity, and a great deal of engineering effort
got: state per token roughly an order of magnitude below the frontier standard

What exists now that didn’t before

A public demonstration that a hardware constraint can be attacked rather than paid around, which cost the industry its assumption that frontier capability follows capital in a straight line.

What it left undone

Six hundred billion parameters still have to sit somewhere, which moves the bill from bandwidth to capacity and to the fabric between machines.

Asked, and answered

How did deepseek train a frontier model so cheaply?

They found a way to compress how much state each token carries in memory — a technique that had been sitting in public, published research for seven years, just never pushed this hard, because nobody with easier options had needed to.