Updated: August 2026.
If the hottest question about AI in 2023–2024 was “which model is better,” then starting in early 2026, the most costly question has shifted to: how much does it cost to run one token, and who runs it more cheaply than others. This is no longer a hardware question—anyone can buy a GPU. What separates the winners from the losers now is a very specific algorithmic layer between the model and the chip: how memory is organized, how tasks are scheduled, how to predict, and how to streamline operations.
What’s strange about 2026 is that API prices continue to fall while memory and GPUs are at record highs. Much of that discrepancy stems from the algorithms described below.
Root: a two-phase requirement with two opposing knots
To understand why a whole range of seemingly unrelated techniques emerged at the same time, we must recognize a fundamental physical truth: text generation consists of two phases with completely different characteristics.
Prefill phase — the model reads the entire prompt. Since all tokens are loaded at once, the GPU can perform large-scale matrix multiplication and run at nearly full capacity: this is the computational bottleneck, with the cost scaling linearly with length and the attention cost scaling quadratically with length.
Decoding phase — a token-by-token generation model where each token depends on the previous one; there is no way to parallelize it. Each step must re-read all active weights from HBM just to generate a single token: for a model with tens of billions of parameters, that amounts to tens of gigabytes of data read per token, while the actual computations required are minuscule. The result: on small batches, the GPU uses only a few percent of its computational capacity; the rest of the time is spent waiting for memory.
This entire article is essentially just a consequence of one statement: during the decoding phase, there is an excess of computations but a shortage of bandwidth. Every inference acceleration algorithm is a way to reduce the excess in order to make up for the shortage.
KV-cache: When a model needs a small operating system
To avoid recalculating attention for the entire history at every step, the model retains the key and value vectors of every token that has passed through—this is the KV-cache, which transforms long-term context into short-term context: each new token adds a piece to it, and that cache must reside within the HBM, competing for space with the weights themselves.
The naive approach is to allocate a contiguous block of memory large enough to accommodate the maximum length for each request—resulting in terrible fragmentation, with most of the memory reserved but unused. PagedAttention solves this problem exactly as an operating system solves the virtual memory problem: it divides the KV-cache into fixed-size blocks, maintains a mapping table from logical addresses to physical blocks, and allocates memory incrementally as needed. No contiguity is required, fragmentation is virtually nonexistent, and more importantly—if two requests share a common beginning, they share the same block, just like copy-on-write.
From that idea of chunking emerged the most cost-effective solution in practice: prefix caching. System prompts, tool descriptions, attached documents, conversation history—in real-world applications, repetitive content makes up the majority of the input. Hashing each block of content and looking it up in a table is enough to eliminate most of the prefill phase for subsequent calls. This is why major providers list separate prices for cached inputs, which are typically about one-tenth the cost. In other words: a data structure has become a line item on the price list.
By 2026, this cache will be tiered to server RAM and then to SSDs, and the router must know which machine holds the cache in order to route requests to that specific machine—even load balancing has become the wrong choice, because sending a request to an idle machine without the cache is more expensive than queuing it on a machine that does have the cache.
Scheduling: Continuous batch merging and prefill splitting
Combining multiple requests into a single run helps share the cost of reading weights—read once, use for the entire batch. But the old-style batching (waiting for a full batch, running, then waiting for the entire batch to finish) is wasteful: short requests have to wait for long ones. Continuous batching schedules at the token-generation step level: once a request is done, it leaves the batch immediately, and the empty slot is filled by a new request in the next step. Simply changing the scheduling granularity has increased throughput many times over on real-world workloads, without touching the model at all.
The remaining issue is two phases that overlap: a long prompt being prefilled takes up the GPU for hundreds of milliseconds, causing all requests currently being decoded to stall. Chunked prefill breaks the prefill process into smaller chunks interspersed with decoding steps, trading a bit of wait time for the first character for smoother performance across the board—blurring the line between algorithm and business policy.
Guess and then check: get rid of what you have too much of to buy what you’re missing
If decoding involves extra computations but lacks bandwidth, checking five tokens in a single pass is almost no more expensive than checking one token—because both require reading the same number of weights. That is the entire intuition behind speculative decoding.
Mechanism: An inexpensive draft model suggests the next few tokens; the large model runs a single pass to check the entire sequence; if a token matches, it’s accepted; if the first token is off, it’s discarded and regenerated. The most subtle aspect is the verification step, which uses weighted sampling so that the output distribution is identical to that of a normal large-scale model run. This is not an approximation that sacrifices quality—which is why it’s enabled by default without requiring anyone’s permission.
The trend for 2026 is to use a separate draft model: instead of training a separate small model, researchers attach a multi-token prediction head directly to the large model itself, or use a lightweight draft model to read the model’s internal state directly. The EAGLE-3 family and its multi-token prediction variants were merged into the main branches of vLLM, SGLang, and TensorRT-LLM early this year; the average number of tokens processed per turn ranges from two to three.
But let’s be clear about the limitations: the benefits diminish as the batch size increases. The larger the batch, the closer the GPU gets to full computational load, leaving no room for over-prediction; the acceptance rate also drops when the context is very long. Therefore, this is a tool for low latency and off-peak hours, not an unconditional solution for increasing throughput.
The Real Breakout of 2026: Pay Attention—Check Out Every Single Token
All of the above are operational optimizations: the model remains unchanged. The biggest change starting in early 2026 runs deeper—the architecture itself is being modified to prevent the KV-cache from continuing to grow. Three approaches are being pursued in parallel:
- Compress KV pairs into a latent state. Instead of storing the full keys and values for each attention head, project them onto a much smaller latent vector and then unpack them during computation—the multi-head latent attention approach reduces cache usage by a factor of several times with virtually no loss in quality.
- Sparse attention is learnable. Each new token does not need to examine the entire history; a low-cost index selects a small set of relevant tokens, and attention is computed only on that set. Unlike the old-school “sliding window” approach, the selection here is trained alongside the model rather than being determined by hard-coded rules set by the user.
- The most surprising innovation is the hybrid with linear-directional attention. Most layers are replaced by a form of gated regression, where each layer maintains a fixed-size state instead of an infinitely long cache, and a full attention layer is inserted every few layers to preserve accurate memory. The ratio of three linear layers to one full layer has appeared in many model families released in the first half of 2026.
The economic implications of the third approach are significant. With traditional attention, where the context is doubled, each decoding step must read twice as much—the cost per token increases with the length of the conversation. With most of the layer consisting of fixed-state regression, the cost per token remains nearly constant regardless of length. Just as models and agents begin to generate sessions hundreds of thousands of tokens long, that is the difference between having a business model and not having one.
One point that may not necessarily need mentioning: when removing tokens to examine them, there is always a risk of overlooking the very tokens that matter most. Accurate recall tests in long contexts remain the area where these architectures are scrutinized most closely, and current measurement methods are not yet standardized.
Separate the machine: prefill one cluster, decode one cluster
If two phases require two conflicting types of resources, confining them to the same machine forces both to compromise. By 2026, separating prefill from decode had become the default configuration at scale: one group of machines runs only prefill, generates the KV-cache, and then transfers it over a high-speed network to a group of machines running only decode. Each side selects its own parallel configuration and batch size, scaling independently based on the ratio of long-to-short prompts in the actual workload.
This is combined with large-scale expert parallelization for the MoE model. MoE activates only a small fraction of the parameters for each token, but if all experts are distributed across a few GPUs, there is still a significant amount of data to read. By distributing the experts across dozens of GPUs connected via ultra-high-bandwidth links—such as the GB200/GB300 NVL72 rack—each GPU holds only a few experts, significantly reducing the amount of weight data read per token and increasing throughput per GPU many times over compared to running on a single machine.
Economics Corner: The Four Numbers Behind the Price of Each Token
The cost of a token, put simply, is the hourly GPU rental fee divided by the number of tokens that GPU produces per hour. The numerator is determined by the hardware market and is already under strain; the entire algorithmic layer above does nothing but push the denominator up—but must maintain the latency commitment, because throughput gained by making users wait is unsellable. The industry calls this metric “goodput.” Four levers, ranked by profitability:
- Prefix cache hit rate — the cheapest and most overlooked. A stable system prompt that places the invariant part at the beginning can reduce a significant portion of the input cost without changing a single line of the model.
- Effective batch size. Larger batches evenly distribute the cost of reading weights, but require more KV-cache, which is limited in HBM. You should either lower the KV precision to FP8 or compress KVs using architectural techniques—both are indirect ways to buy more batch size. That’s why KV quantization belongs to the cost-reduction algorithm category rather than the data compression category.
- Number of tokens generated per run. Speculation suggests this number could be increased from one to two or three.
- The number of weights that must be read for each token. MoE combined with parallel wide-area experts directly targets this—the lever with the greatest impact.
There is one consequence that few people notice: the inference model generates a large number of output tokens relative to the input, which shifts the cost focus from prefill to decode. This means that everything optimized for decode—bandwidth, small KV pairs, speculation, MoE—has suddenly become many times more valuable than it was two years ago, while prefill optimization has become less important. The infrastructure of 2026 is being reimagined around this shift.
Predictions
- Attention will once again become the default. By the end of 2026, most new open models will use a hybrid of linear/sparse layers and full attention layers; full attention across the entire model will become the exception, reserved only for small models or short contexts.
- Draft models will disappear from deployment profiles. Multi-token predictors will be trained and released with their weights as part of the model, rather than something operators have to assemble themselves.
- Performance metrics shift from GPUs to racks. The comparable metric will be tokens per second per rack at a guaranteed latency level—since phase separation and wide-area parallelism only make sense when calculated across an entire cluster.
- Cache becomes a storage tier with its own pricing. A tiered prefix cache (HBM → RAM → SSD), shared across multiple machines, includes cache-aware routing; the API pricing table continues to be broken down by cache state rather than a single input price.
- A minor reality check regarding metrics. The more systems reduce token usage to save resources, the more likely there will be significant debate over whether long-term performance metrics mask actual performance losses in precise retrieval tasks.
Key takeaway: Over the past two years, most of the cost reduction in AI has not come from new chips but from fewer memory reads per token. All the algorithms listed above—paging, prefix caching, continuous batching, predict-and-check, sparse and linear attention, phase separation, and expert spreading—are simply different ways of expressing that same principle. Those who understand that the bottleneck lies in bandwidth rather than computation will optimize effectively; those who don’t will just buy more GPUs and still pay a high price.
Thảo luận