Today, we are announcing new Ray capabilities on Amazon SageMaker HyperPod that integrate Ray with the HyperPod purpose-built infrastructure for foundation model training and serving.

Ray is an open-source framework that data scientists use to scale distributed Python workloads across clusters of GPUs, from distributed training with Ray Train to model serving with Ray Serve. On Kubernetes, Ray clusters are managed by KubeRay, an open-source operator that handles cluster lifecycle through custom resources (RayCluster, RayJob, RayService). SageMaker HyperPod provides purpose-built infrastructure for large-scale machine learning (ML) on Amazon Elastic Kubernetes Service (Amazon EKS) with built-in node health monitoring and automatic recovery. Until now, running Ray on Kubernetes required data scientists to write YAML manifests, manage Docker image rebuilds for every dependency change, set up kubectl port-forward to access their Ray Dashboard, and configure Prometheus and Grafana manually for observability.

With this launch, data scientists can create Ray clusters, open the Ray Dashboard and Amazon Managed Grafana observability dashboards, connect a JupyterLab or Code Editor workspace to their cluster, submit distributed jobs, and configure hung job detection, all from SageMaker Studio. At the application level, Ray training jobs gain automatic fault tolerance through HyperPod node health monitoring and recovery, plus tiered checkpointing for faster resume through HyperPod distributed tiered storage. SageMaker JumpStart integration loads model weights directly into Ray Serve endpoints, with KV cache offloading to tiered storage for serving long-context requests. These capabilities work with open-source KubeRay and standard Ray APIs, so existing scripts and workflows run without modification.