Updated: August 2026.
An operating system was created to do one thing: act as a mediator between expensive hardware and a host of resource-hungry programs, allocating resources so that no single program monopolizes them. For forty years, the Linux kernel has done this exceptionally well with CPUs, memory, disk, and networking. But in 2026, the most expensive component in a server is no longer the CPU—it’s the eight GPUs tucked under the hood, which account for the lion’s share of the bill. And as is typical for the most expensive component, the kernel barely manages it at all.
There are four abstract pillars—the GPU is not one of them
List the factors that make a resource truly “owned” by a process. For the CPU: there’s the concept of processes, a fair-sharing scheduler, cgroups that track each cycle, and most importantly, the ability to preempt—the clock is interrupted, and low-priority processes are immediately pushed off the core without needing permission. For memory: there are memory pages; virtual memory allows requesting more than the system has; and swap provides a soft landing when memory runs out. For disk and network: there are inodes, sockets, queues, and policies.
With a GPU, there’s almost nothing on that list. From the kernel’s perspective, a GPU is just a device file: a process opens /dev/nvidia*, calls a few vendor-specific ioctl commands, and then everything happens inside the driver and firmware. The kernel has no idea how many compute “kernels” are queued up, which ones are about to finish, or which ones are hogging all the bandwidth. There’s no gpu.weight to say, “This job gets two parts, that one gets one part.” There’s no swap for VRAM: once GPU memory runs out, the job dies—end of story. And there’s no preemption: a three-day training job will lock up eight GPUs for three days, even if more urgent tasks are waiting in line behind it.
The architectural implications are significant: since the node has no place to store policies, all decisions are pushed up the chain—to Kubernetes, to the cluster scheduler. That layer sees pods and labels, but cannot look inside the GPU. It simply counts: “This machine has 8 cards, 8 cards have been allocated—that’s it.”
Economics Corner: How Much Is Each Percentage Point of Occupancy Worth?
When resources are only allocated in bulk, the leftover capacity goes unused. A Kubernetes optimization report from early 2026, which surveyed more than twenty thousand clusters, revealed a staggering figure: average GPU utilization hovered around 5%. Other FinOps surveys are more conservative, typically citing 20–40% for well-maintained clusters. No matter which figure you use, the conclusion is the same: most of the time, the most expensive silicon on the planet is sitting idle—waiting for data, waiting to synchronize, or simply because a pod has claimed it and then abandoned it.
The price for renting a data-center-class GPU on an on-demand basis on major cloud platforms is around ten dollars per hour; long-term contracts are much cheaper, but you have to pay even when you’re not using them. Either way, an idle GPU burns through dollars per hour, while an idle CPU burns through only a few cents. That two-order-of-magnitude difference completely reverses the technical priorities. If utilization is only 20%, the cost per hour of actual usable compute time is five times the listed price; increasing utilization from 20% to 40% isn’t “twice as fast” but rather cuts the cost in half for the same amount of work—no need to buy additional GPUs, no need to wait for factory production, and no need to request more power.
While HBM and packaging capacity are major bottlenecks for the entire industry, the cheapest way to gain additional computing power isn’t to buy more hardware but to reclaim what’s currently being wasted—so dry topics like cgroups, schedulers, and checkpoints suddenly become the most financially significant components in an AI cluster.
Starting to make up for it: a cgroup for VRAM, write scheduling using BPF
The first building block is in place: starting with Linux 6.14, the kernel includes the dmem cgroup—a device memory controller—that allows video memory to be limited via the cgroup tree, initially integrated with the Intel Xe graphics driver. It may sound modest, but this marks the first time that the memory of an acceleration device has been accounted for using the same kernel mechanism used for RAM. The idea of “cgroups for DRM” had been rejected many times over nearly a decade; it’s finally coming to fruition now because someone has paid a real price for its absence.
On the scheduling front, the breakthrough came from an unexpected direction. sched_ext—a framework that allows writing CPU schedulers using BPF and hot-loading them into the kernel—has gone from being controversial to commonplace: many leading distributions have it enabled by default, and SteamOS even uses a BPF scheduler as the default when gaming. Notably for AI servers, the roadmap shows the development team has added “GPU awareness” to their to-do list. The reason is very practical—which CPU thread feeds data to which GPU is critical information; assigning a thread to a core in a different NUMA region leaves the GPU starved for data.
The orchestration layer has reached a milestone: Kubernetes’ Dynamic Resource Allocation (DRA) has been released as a stable feature in v1.35, completely replacing the old device plugin mechanism, which could only count integers. DRA allows workloads to specify their requirements—device types, memory capacity, NVLink connection types, and MIG slices—and then lets the scheduler handle the matching. At KubeCon Europe 2026, NVIDIA contributed its DRA driver to the CNCF, meaning this resource model is no longer proprietary to a single vendor. The kernel abstraction layer is still missing, but the ecosystem is being built at the upper layers and standardized.
A New Kind of Hijacking: Snapping a Photo of the GPU and Then Continuing
Of the four remaining challenges, preemption is the most difficult—and also the one with the most interesting progress. To suspend a job to make room for another, you must save its entire state: CPU memory, open files, and both VRAM content and the CUDA context—threads, events, and memory mappings. That piece of the puzzle is now in place: NVIDIA provides `cuda-checkpoint`, a tool that locks CUDA calls, pulls device memory back to the host, and then releases the GPU; combined with CRIU—a tool for taking process snapshots in the Linux user space—we get a unified snapshot of both the CPU and GPU, which can be restored to the exact original state, even on a different machine.
This is more important than it seems, because it transforms the GPU from a resource that’s “lent out until it dies” into a recoverable resource. Only with checkpoints can we run low-priority jobs to fill gaps, since we can push them out when needed without losing progress; only then can we migrate jobs to consolidate fragmented resources; and only then can we sell idle capacity on a “spot” basis. The entire economic calculation in the section above depends on this capability.
The memory layer breaks into four parts
At the same time, the second foundational assumption of the human mind is also being called into question: that memory is a flat, uniform space where access is the same everywhere. In today’s AI servers, memory is a four-tiered ladder with latency differences of a thousand-fold: HBM integrated into the GPU, the CPU’s local DRAM, memory expanded via CXL, and finally NVMe.
The new approach is that the kernel has begun to manage that ladder itself rather than leaving it up to the application. Starting on 6.9, the weighted interleave policy allows memory to be distributed among nodes based on weights proportional to bandwidth, rather than a naive equal distribution—with CXL, equal distribution is shooting oneself in the foot, as the slowest tier drags down the entire system. Next, DAMON—the system that monitors memory access patterns directly within the kernel—has been expanded to not only observe but also take action: moving hot pages to faster tiers, moving cold pages to slower tiers, and, in the 2026 updates, dynamically distributing them across multiple destination nodes with their own weights. This marks the transition of virtual memory to a multi-tiered era: it’s still the same old idea—“the kernel knows which pages belong where”—but now it must choose among four memory types that differ by a factor of ten in cost per GB.
When an agent runs code on its own: The sandbox becomes the new frontier
A few years ago, there was a type of workload that didn’t exist on a significant scale: code generated by models, running automatically, with no one reviewing it beforehand. Each session was a short process that came and went in a matter of seconds and was completely unreliable.
Containers aren’t sufficient for this, for a very basic reason: every container on a machine shares a single kernel, so the attack surface is the entire system call interface of that kernel. By 2026, industry consensus had firmly shifted toward microVMs: each agent instance runs in its own virtual machine with its own Linux kernel, under KVM. Firecracker boots in about one-tenth of a second at a cost of a few MiB of memory per virtual machine; Kata Containers packages this idea into a “runtime class” that plugs directly into Kubernetes. The other approach is gVisor—a Linux kernel rewritten in user space that intercepts system calls before they reach the actual kernel. The balance is shifting now as two forces collide: the boot time gap between containers and microVMs has narrowed to the point where it’s no longer an excuse, while the cost of a single sandbox escape has skyrocketed—the machine now contains API keys, customer data, and the ability to run other agents.
Shared machines: The user must prove they are not eavesdropping
The final piece is a consequence of shared hosting. When models and data run on someone else’s machine, the question is no longer “Can my neighbors read it?” but “Can the host read it?” Confidential computing addresses this by wrapping the entire virtual machine within a CPU-encrypted region—AMD SEV-SNP or Intel TDX—so that the provider’s virtualization layer cannot read the guest’s memory. The new development for 2025–2026 is bringing the GPU into that trusted region: Confidential Computing mode on data center GPUs starting with the H100 generation encrypts both device memory and the PCIe and NVLink buses, then issues an attestation so that the tenant can independently verify that the CPU, GPU, and the bus between them are all sealed. The performance overhead for inference is currently reported to be in the range of a few percent—much lower than initially predicted—so it is shifting from a “bank-only option” to the default for sensitive workloads.
Prediction
- GPU cgroups will follow in the footsteps of CPU cgroups. Once device memory is locked down, the next challenge will be some form of “weighting” for compute time. It will be a slow and contentious process because each vendor keeps its scheduler hidden in the firmware—but the money is on the side of those who have it.
- GPU checkpoints are becoming standard infrastructure, no longer just a workaround. Within the next 12–18 months, stopping, migrating, and resuming a GPU job will be a default feature of cluster platforms, paving the way for a true “spot” market.
- Utilization becomes a publicly competitive metric. Providers will advertise “price per useful hour” rather than the listed price per card.
- Kernel-controlled memory tiering will become the default. Weighted interleave and DAMON-based tiering will move out of the “expert-only manual tuning” realm and into the preconfigured settings of server distributions as CXL becomes more widespread.
- Per-session MicroVMs will become the default for model-generated code. Containers will remain robust for trusted internal workloads, but “shared-kernel code from unknown sources” will be treated as a configuration error—much like how running services as root was once reevaluated.
- The reverse risk: if the pace of AI investment slows and GPUs suddenly become abundant, this pressure to optimize will fade quickly—historically, infrastructure has only been properly optimized when resources were scarce.
Overall, this isn’t “Linux adding AI features.” It’s the same old story repeating itself: a type of hardware so expensive that it can’t be wasted, so the operating system is forced to develop additional layers of abstraction to allocate it fairly and make full use of it. Virtual memory, time-sharing scheduling, and cgroups all emerged following that same logic. This time, the main character is the GPU, and we’re in the middle of the story.
Thảo luận