Updated: August 2026.
An operating system was created to do one thing: act as an intermediary between expensive hardware and a host of resource-hungry programs, allocating resources so that no single program monopolizes them. For forty years, the Linux kernel has done this exceptionally well with CPUs, memory, disks, and networks. But in 2026, the most expensive component in a server is no longer the CPU—it’s the eight GPUs tucked under the hood, which account for the lion’s share of the bill. And as is typical for the most expensive component, the kernel has almost no control over it.
There are four abstract pillars—the GPU is not one of them
List the factors that make a resource truly “owned” by a process. For the CPU: there is the concept of processes, a fair-scheduling mechanism, cgroups that track cycles, and most importantly, the ability to preempt—triggering a clock interrupt causes low-priority processes to be immediately pushed off the core without needing permission. For memory: there are memory pages, virtual memory that allows requesting more than the system has, and swap to provide a soft landing when memory runs out. For disk and network: there are inodes, sockets, queues, and policies.
With a GPU, there’s practically nothing on that list. From the kernel’s perspective, a GPU is just a device file: a process opens /dev/nvidia*, calls a few vendor-specific ioctl commands, and then everything happens inside the driver and firmware. The kernel has no idea how many computing “kernels” are queued up, which ones are about to finish, or which ones are hogging all the bandwidth. There’s no `gpu.weight` to say, “This job gets two shares, that one gets one.” There’s no swap space for VRAM: once GPU memory runs out, the job dies—end of story. And there’s no preemption: a three-day training job will lock up eight GPUs for three days, even if more urgent tasks are waiting in line behind it.
The architectural implications are significant: since the node has no place to store policies, all decisions are pushed upward—to Kubernetes, to the cluster scheduler. That layer sees pods and labels, but cannot see inside the GPU. It simply counts: “This machine has 8 cards, 8 cards have been allocated—that’s it.”
Economic Corner: How Much Is Each Percentage Point of Occupancy Worth?
When resources are allocated only in bulk, the leftover capacity goes unused. A Kubernetes optimization report from early 2026, which surveyed more than twenty thousand clusters, revealed a staggering figure: average GPU utilization hovered around 5%. Other FinOps surveys are more conservative, typically citing 20–40% for well-maintained clusters. No matter which figure you use, the conclusion is the same: most of the time, the most expensive silicon on the planet is sitting idle—waiting for data, waiting to synchronize, or simply because a pod has claimed it and then abandoned it.
The price for renting a data-center-class GPU on an on-demand basis from major cloud providers is around ten dollars an hour; long-term contracts are much cheaper, but you have to pay even when you’re not using them. Either way, an idle GPU burns through dollars per hour, while an idle CPU burns through only a few cents. That two-order-of-magnitude difference completely reverses the technical priorities. If utilization is only 20%, the cost per hour of actual usable compute time is five times the listed price; increasing utilization from 20% to 40% isn’t “twice as fast”—it’s halving the cost for the same amount of work—without needing to buy any additional GPUs, wait for factory production, or request more power.
While HBM and packaging capacity are major bottlenecks for the entire industry, the cheapest way to gain additional computing power isn’t to buy more hardware but to reclaim what’s currently being wasted—so dry topics like cgroups, schedulers, and checkpoints suddenly become the most financially significant components in an AI cluster.
Starting to catch up: cgroup for VRAM, write scheduling using BPF
The first building block is in place: starting with Linux 6.14, the kernel includes the dmem cgroup—a device memory controller—that allows video memory to be limited via the cgroup tree, initially integrated with the Intel Xe graphics driver. It may sound modest, but this marks the first time that the memory of an acceleration device has been accounted for using the same kernel mechanism used for RAM. The idea of “cgroups for DRM” had been rejected many times over nearly a decade; it’s finally coming to fruition now because someone has paid a real price for its absence.
On the scheduling front, the breakthrough came from an unexpected direction. sched_ext—a framework that allows writing CPU schedulers using BPF and hot-loading them into the kernel—has gone from being controversial to commonplace: many leading distributions have it enabled by default, and SteamOS even uses a BPF scheduler as the default when gaming. Notably for AI servers, the roadmap indicates that the development team has added “GPU awareness” to their to-do list. The reason is very practical—which CPU thread feeds data to which GPU is critical information; assigning a thread to a core in a different NUMA region leaves the GPU starved for data.
The orchestration layer has reached a milestone: Kubernetes’ Dynamic Resource Allocation (DRA) has been released as a stable feature in v1.35, completely replacing the old device plugin mechanism that could only count integers. DRA allows workloads to specify their requirements—device types, memory capacity, NVLink connection types, and MIG slices—and then lets the scheduler handle the pairing. At KubeCon Europe 2026, NVIDIA donated its DRA driver to the CNCF, meaning this resource model is no longer proprietary to a single vendor. While the kernel abstraction layer is still missing, the ecosystem is building upon the upper layers and standardizing it.
A New Kind of Hijack: Snapping a Photo of the GPU and Then Moving On
Of the four remaining challenges, preemption is the most difficult—and also the one with the most interesting progress. To suspend a job to make room for another, you must save its entire state: CPU memory, open files, and both the contents of VRAM and the CUDA context—threads, events, and memory mappings. That piece of the puzzle is now in place: NVIDIA provides `cuda-checkpoint`, a tool that locks CUDA calls, pulls device memory back to the host, and then releases the GPU; combined with CRIU—a tool for taking process snapshots in the Linux user space—we get a unified snapshot of both the CPU and GPU, which can be restored to the exact original state, even on a different machine.
This is more important than it seems, because it transforms the GPU from a resource that’s “lent out until it dies” into a recoverable resource. Only with checkpoints can we run low-priority jobs to fill gaps, since we can push them out when needed without losing progress; only then can we migrate jobs to consolidate leftover capacity; and only then can we sell idle capacity on a “spot” basis. The entire economic calculation in the section above depends on this capability.
The memory layer breaks into four parts
At the same time, the second foundational assumption of the human mind is also being called into question: that memory is a flat, uniform space where access is the same everywhere. In today’s AI servers, memory is a four-tiered ladder with latency differences of thousands of times: HBM integrated into the GPU, the CPU’s local DRAM, memory expanded via CXL, and finally NVMe.
The new approach is that the kernel has begun to manage that ladder itself instead of leaving it entirely up to the application. Starting on 6.9, the weighted interleave policy allows memory to be distributed among nodes based on weights proportional to bandwidth, rather than a naive equal distribution—with CXL, equal distribution is counterproductive, as the slowest tier drags down the entire system. Next, DAMON—the system that monitors memory access patterns directly within the kernel—has been expanded to not only observe but also act: moving hot pages to faster tiers, moving cold pages to slower tiers, and, in the 2026 updates, dynamically distributing them across multiple destination nodes with their own weights. This marks the transition of virtual memory to a multi-level era: it’s still the same old idea of “the kernel knowing which pages belong where,” but now it must choose among four types of memory that differ by a factor of ten in cost per GB.
When an agent runs code on its own: The sandbox becomes the new frontier
A few years ago, there was a type of workload that didn’t exist on a significant scale: code generated by models, running automatically, with no one reviewing it beforehand. Each session was a short process that came and went in a matter of seconds and was completely unreliable.
Containers aren’t sufficient for this, for a very fundamental reason: every container on a machine shares the same kernel, so the attack surface is the entire system call interface of that kernel. By 2026, industry consensus had firmly shifted toward microVMs: each agent instance runs in its own virtual machine with its own Linux kernel, under KVM. Firecracker boots in about one-tenth of a second at a cost of a few MiB of memory per virtual machine; Kata Containers packages that idea into a “runtime class” that plugs directly into Kubernetes. The other approach is gVisor—a Linux kernel rewritten in user space, which intercepts system calls before they reach the actual kernel. The balance is shifting right now as two forces collide: the boot time gap between containers and microVMs has shrunk to the point where it’s no longer a valid excuse, while the cost of a single sandbox escape has skyrocketed—the machine now holds API keys, customer data, and the ability to launch other agents.
Shared machines: The user must prove they are not eavesdropping
The final piece is a consequence of shared hosting. When models and data run on someone else’s machine, the question is no longer “Can my neighbors read it?” but “Can the host read it?” Confidential computing addresses this by wrapping the entire virtual machine within a CPU-encrypted trusted region—AMD SEV-SNP or Intel TDX—so that the provider’s virtualization layer cannot read the guest’s memory. The new development for 2025–2026 is bringing the GPU into that trusted region: Confidential Mode on data center GPUs starting with the H100 generation encrypts both device memory and the PCIe and NVLink links, then issues an attestation so that the tenant can verify that the CPU, GPU, and the bus between them are all sealed. The performance overhead for inference is currently reported to be in the low single-digit percentage range—much lower than initially predicted—so it is shifting from a “bank-only option” to the default for sensitive workloads.
Prediction
- GPU cgroups will follow in the footsteps of CPU cgroups. Once device memory is locked down, the next challenge will be a form of “weighting” for compute time. This will be a slow and contentious process because each vendor hides its scheduler in the firmware—but the money is on the side of those who implement it.
- GPU checkpoints are becoming standard infrastructure, no longer just a workaround. Within the next 12–18 months, the ability to pause, migrate, and resume a GPU job will be a default feature of cluster platforms, paving the way for a true “spot” market.
- Utilization becomes a publicly competitive metric. Providers will advertise “price per useful hour” rather than the listed price per card.
- Kernel-controlled memory tiering will become the default. Weighted interleave and DAMON-based tiering will move out of the “expert-only manual tuning” realm and into the pre-configured settings of server distributions as CXL becomes more widespread.
- Per-session MicroVMs will become the default for model-generated code. Containers will remain robust for trusted internal workloads, but “unfamiliar code sharing the kernel” will be treated as a configuration error—much like how running services as root was once reevaluated.
- The reverse risk: if the pace of AI investment slows and GPUs suddenly become abundant, this optimization pressure will fade quickly—historically, infrastructure has only been properly optimized when resources were scarce.
Overall, this isn’t simply “Linux with AI features.” It’s the same old story playing out again: a type of hardware so expensive that it can’t be wasted, so the operating system is forced to develop additional layers of abstraction to allocate it fairly and make full use of it. Virtual memory, time-sharing scheduling, and cgroups all emerged following that exact logic. This time, the protagonist is the GPU, and we’re in the middle of the story.
Thảo luận