GPUs keep falling off the PCIe bus, and standard node health does not notice

AWS engineers write up what running Kubernetes at Amazon EKS scale taught them about GPU node failure, and why the default node health model has nothing to say when the GPU just vanishes from the bus.

Raccontata dadev.to