Updated: August 2026.
An operating system is designed to do one thing: act as a go-between for expensive hardware and a bunch of resource-hungry programs, then allocate resources so that no single program hogs them all. For forty years, the Linux kernel has done this exceptionally well with CPUs, memory, disks, and networks. But in 2026, the most expensive component in a server is no longer the CPU—it’s the eight GPUs tucked under the hood, which account for the lion’s share of the bill. And as is typical for the most expensive component, the kernel has almost no control over it.
There are four abstract pillars—the GPU is not one of them
List the factors that make a resource something a user truly “owns.” For the CPU: there’s the concept of processes, a fair-share scheduler, cgroups that track each cycle, and most importantly, the ability to preempt—the clock interrupts, and low-priority processes are immediately pushed off the core without needing permission. For memory: there are memory pages, virtual memory that allows requesting more than the system has, and swap space for a soft landing when memory runs out. For disk and network: there are inodes, sockets, queues, and policies.
With a GPU, there’s practically nothing on that list. From the kernel’s perspective, a GPU is just a device file: a process opens /dev/nvidia*, calls a few vendor-specific ioctl commands, and then everything happens inside the driver and firmware. The kernel has no idea how many computing “kernels” are queued, which ones are about to finish, or which ones are hogging all the bandwidth. There’s no `gpu.weight` to say, “This job gets two parts, that one gets one part.” There’s no swap for VRAM: once GPU memory runs out, the job dies—end of story. And there’s no preemption: a three-day training job will lock up eight GPUs for three days, even if more urgent tasks are waiting in line behind it.
The architectural implications are significant: since the node has no place to store policies, all decisions are pushed up the chain—to Kubernetes, to the cluster scheduler. That layer sees pods and labels, but cannot look inside the GPU. It simply counts: “This machine has 8 cards, 8 cards have been allocated—that’s it.”
Economic Corner: How Much Is Each Percentage Point of Occupancy Worth?
When resources are allocated only in fixed blocks, the leftover capacity goes unused. A Kubernetes optimization report from early 2026, which surveyed more than twenty thousand clusters, revealed a staggering figure: the average GPU utilization rate was around 5%. Other FinOps surveys are more conservative, typically citing 20–40% for well-maintained clusters. No matter which figure you use, the conclusion is the same: most of the time, the most expensive silicon on the planet is just sitting idle—waiting for data, waiting to synchronize, or simply because a pod has claimed it and left it unused.
The price for renting a data-center-class GPU on an on-demand basis on major cloud platforms is around ten dollars per hour; long-term contracts are much cheaper, but you have to pay even when you’re not using them. Either way, an idle GPU burns through dollars per hour, while an idle CPU burns through only a few cents. That two-order-of-magnitude difference completely reverses the technical priorities. If utilization is only 20%, the cost per hour of actual usable compute time is five times the listed price; increasing utilization from 20% to 40% isn’t “twice as fast”—it’s halving the cost for the same amount of work—without needing to buy additional GPUs, wait for factory production, or request extra power.
While HBM and packaging capacity are the industry-wide bottlenecks, the cheapest way to gain additional computing power isn’t to buy more hardware but to reclaim what’s currently being wasted—so dry topics like cgroups, schedulers, and checkpoints suddenly become the most financially significant components in an AI cluster.
Starting to make up for it: cgroup for VRAM, write scheduling using BPF
The first building block is in place: starting with Linux 6.14, the kernel includes the dmem cgroup—a device memory controller—that allows video memory to be limited via the cgroup tree, initially integrated with the Intel Xe graphics driver. It may sound modest, but this marks the first time that the memory of an acceleration device has been accounted for using the exact same kernel mechanism used for RAM. The idea of “cgroups for DRM” had been rejected numerous times over nearly a decade; it’s finally been implemented now because someone has paid the price for its absence.
On the scheduling front, the breakthrough came from an unexpected direction. sched_ext — a framework that allows CPU schedulers to be written in BPF and hot-loaded into the kernel — has gone from controversial to commonplace: many leading distributions have it enabled by default, and SteamOS even uses a BPF scheduler as the default when gaming. Notably for AI servers, the roadmap shows the development team has placed “GPU awareness” on their to-do list. The reason is very practical—which CPU thread feeds data to which GPU is critical information; misassigning a thread to a core in a different NUMA region leaves the GPU starved for data.
The orchestration layer has reached a milestone: Kubernetes’ Dynamic Resource Allocation (DRA) has been released as a stable feature in v1.35, completely replacing the old device plugin mechanism, which could only count integers. DRA allows workloads to specify their requirements—device types, memory capacity, NVLink connection types, and MIG slices—and then lets the scheduler handle the matching. At KubeCon Europe 2026, NVIDIA contributed its DRA driver to the CNCF, meaning this resource model is no longer proprietary to a single vendor. While the core abstraction is still under development, the ecosystem is building upon it and standardizing it.
A New Kind of Hijacking: Snapping a Photo of the GPU and Then Moving On
Of the four remaining challenges, preemption is the most difficult—and has seen the most interesting progress. To suspend a job to make room for another, you must save its entire state: CPU memory, open files, and both VRAM contents and the CUDA context—threads, events, and memory mappings. That piece of the puzzle is now in place: NVIDIA provides `cuda-checkpoint`, a tool that locks CUDA calls, pulls device memory into user space, and then releases the GPU; combined with CRIU—a tool for capturing process snapshots in the Linux user space—we get a unified snapshot of both the CPU and GPU, which can be restored to the exact same state, even on a different machine.
This is more important than it seems, because it transforms the GPU from a resource that’s “lent out until it dies” into a recoverable resource. Only with checkpoints can we run low-priority jobs to fill gaps, since we can push them out when needed without losing progress; only then can we migrate jobs to consolidate scattered resources; and only then can we sell idle capacity on a “spot” basis. The entire economic calculation described above depends on this capability.
The memory layer breaks into four parts
At the same time, the second foundational assumption of the human mind is also being called into question: that memory is a flat, uniform space where access is the same no matter where you go. In today’s AI servers, memory is a four-tiered ladder with latency differences of thousands of times: HBM integrated into the GPU, the CPU’s local DRAM, memory expanded via CXL, and finally NVMe.
The new approach is that the kernel has begun to manage that ladder itself instead of leaving it up to the application. Starting on 6.9, the weighted interleave policy allows memory to be distributed among nodes based on weights proportional to bandwidth, rather than a naive even split—with CXL, an even split is counterproductive, as the slowest tier drags down the entire system. Next, DAMON—the system that monitors memory access patterns directly within the kernel—has been expanded to not only observe but also take action: moving hot pages to faster tiers, moving cold pages to slower tiers, and, in the 2026 updates, dynamically distributing them across multiple destination nodes with their own weights. This marks the transition of virtual memory to a multi-tiered era: it’s still the same old idea—“the kernel knows which pages belong where”—but now it must choose among four types of memory that differ by a factor of ten or more in cost per GB.
When an agent runs code on its own: the sandbox becomes the new frontier
A few years ago, there was a type of workload that didn’t exist on a significant scale: code generated by models, running automatically, with no one reviewing it beforehand. Each session was a short process—it came and went in a matter of seconds—and was completely unreliable.
Containers aren’t sufficient for this, for a very basic reason: every container on a machine shares a single kernel, so the attack surface is the entire system call interface of that kernel. By 2026, industry consensus had firmly shifted toward microVMs: each agent instance runs in its own virtual machine with its own Linux kernel, under KVM. Firecracker boots in about one-tenth of a second at a cost of a few MiB of memory per virtual machine; Kata Containers packages that idea into a “runtime class” that integrates directly into Kubernetes. The other approach is gVisor—a Linux kernel rewritten in user space that intercepts system calls before they reach the actual kernel. The balance is shifting now as two forces collide: the startup time gap between containers and microVMs has narrowed to the point where it is no longer a valid excuse, while the cost of a single sandbox escape has skyrocketed—the machine now holds API keys, customer data, and the ability to run other agents.
Shared machines: The user must prove they are not eavesdropping
The final piece is a consequence of shared hosting. When models and data run on someone else’s machine, the question is no longer “Can my neighbor read it?” but rather “Can the host read it?” Confidential computing addresses this by wrapping the entire virtual machine within a CPU-encrypted zone—AMD SEV-SNP or Intel TDX—so that the provider’s virtualization layer cannot read the guest’s memory. The new development for 2025–2026 is bringing the GPU into that trusted zone: Confidential Computing mode on data center GPUs starting with the H100 generation encrypts both device memory and the PCIe and NVLink buses, then issues an attestation so that the tenant can verify that the CPU, GPU, and the bus between them are all sealed. The performance overhead for inference is now reported to be in the range of a few percent—much lower than initially predicted—so it is shifting from a “bank-level option” to the default for sensitive workloads.
Prediction
- GPU cgroups will follow in the footsteps of CPU cgroups. Once device memory is locked down, the next challenge will be some form of “weighting” for compute time. It will be a slow and contentious process because each vendor hides its scheduler in the firmware—but the money is on the side of those who have it.
- GPU checkpoints are becoming standard infrastructure, no longer just a workaround. Within the next 12–18 months, pausing, migrating, and resuming a GPU job will be a default feature of cluster platforms, paving the way for a true “spot” market.
- Utilization becomes a publicly competitive metric. Providers will advertise “cost per useful hour” rather than the listed price per card.
- Kernel-controlled memory tiering will become the default. Weighted interleave and DAMON-based tiering will move out of the “expert manual tuning” realm and into the preconfigured settings of server distributions as CXL becomes more widespread.
- Per-session MicroVMs will become the default for model-generated code. Containers will remain robust for trusted internal workloads, but “unfamiliar code sharing the kernel” will be treated as a configuration error—much like how running services as root was once reevaluated.
- The reverse risk: if the pace of AI investment slows and GPUs suddenly become abundant, this optimization pressure will fade quickly—historically, infrastructure has only been properly optimized when resources were scarce.
Overall, this isn’t “Linux adding AI features.” It’s the same old story playing out again: a type of hardware so expensive that it can’t be wasted, so the operating system is forced to develop additional layers of abstraction to allocate it fairly and make full use of it. Virtual memory, time-sharing scheduling, and cgroups all emerged following that exact logic. This time, the protagonist is the GPU, and we’re in the middle of the story.
Thảo luận