AWS engineers have published a field report from running Kubernetes at Amazon EKS scale with GPU workloads, and the recurring failure they call out is one every GPU operator eventually meets: the GPU disappears from the machine while the pod is still scheduled on it. The New Stack ran the piece on July 19, and the specific symptom named is a GPU falling off the PCIe bus.
The failure the default node model was never taught to see
If you have only ever run CPU workloads on Kubernetes, node health reads as a solved problem. The kubelet reports back to the control plane, the node condition transitions to NotReady when something is wrong, the controller manager evicts pods after a grace period, the scheduler places them elsewhere. That loop assumes hardware is either present or absent. A machine that is up, reachable, and running a kubelet is a machine you can send work to.
GPU nodes break the assumption. The host stays up. The kubelet keeps sending heartbeats. The CPU cores are fine. What has changed is that one of the GPUs the pod actually needs is no longer visible on the PCIe bus, or the driver has hung, or a memory error has taken a region out from under the workload. From above, the node still shows Ready. From inside the pod, the job just failed a CUDA call and will keep failing every retry until someone drains and recycles the host.
