Software Quality Engineer
It's Friday evening. Priya, a machine language (ML) engineer at a financial services firm, queues up a fraud-detection fine-tuning job on an GPU cluster costing $55 an hour. The run should take about 40 hours—roughly $2,200 in compute. She double-checks the hyperparameters, submits the job, and heads home for the weekend.Monday morning, she opens her laptop. The model had stopped learning sometime Friday night, but the job kept running—burning through 2 full days of GPU time on a training run that was going nowhere. That's over $1,500 in wasted compute, and she has to start over.If this sounds familiar, you're not alone. Every practitioner who has run a multiday training job knows the uncomfortable questions:Is the model actually learning anything?Should I stop early and try different hyperparameters?Will it finish before the stakeholder review on Tuesday?The cost of not knowing is real. On demand GPU clusters can cost upwards of $50 per hour. A single multiday training run can easily reach thousands of dollars, and a misconfigured run can waste an entire weekend of compute before anyone notices. In shared clusters, the impact multiplies: a stalled job doesn't just waste one team's budget—it blocks other teams from accessing GPUs they need. One team's blind run becomes another team's delay. The problem isn't that teams are careless. It's that they're flying blind. Every GPU-hour matters—and without real-time visibility into training progress, too many of them are wasted.The observability gapHere's the uncomfortable truth—from the platform's perspective, a training job that's converging beautifully and one that's completely stalled look identical. Both are pods in running state, consuming the same resources, showing the same green status indicator. Kubernetes has no concept of whether your model is learning.Today, monitoring training progress is possible—but it's harder than it should be.Logs are scattered and hard to use: In a distributed training job, logs are spread across multiple pods. Finding the relevant output means knowing which pod to look at, how to access it, and how to interpret framework-specific log formats. That requires Kubernetes expertise, an extra skill set that most data scientists and AI engineers shouldn't need to acquire just to monitor their training runs. Bad and unstructured logs aren't easily machine-readable, so they can't drive automation.External tools solve observability but add management overhead: Experiment tracking platforms like MLflow and Weights & Biases are excellent for what they do. But they require additional infrastructure to deploy and maintain, API keys and access controls to manage, and they operate above the platform layer. They're also disconnected from the Kubernetes API, meaning it’s harder to integrate them with custom Kubernetes-native workflows and custom operators. They tell you what happened inside the training loop, but they can't tell the platform what's happening.Every framework does it differently: PyTorch DDP, FSDP, DeepSpeed, JAX—each has its own logging conventions, its own metrics format, its own way of reporting progress. There's no consistency across frameworks, which means there's no unified view for platform teams managing a heterogeneous training environment.There's no standard API: Until now, training jobs have had no standardized way to report progress back to the platform. The result is that platform administrators managing shared GPU clusters are left guessing which jobs are progressing, which are stalled, and which are about to finish. Without that visibility, capacity planning and troubleshooting are guesswork.Closing the gap: Progress tracking in Red Hat OpenShift AIRed Hat OpenShift AI now includes production-ready progress tracking for distributed training jobs, built into Kubeflow Trainer v2 and generally available beginning with Red Hat OpenShift AI 3.4. You can see what's happening while it's still happening—and act before it's too late. This feature directly addresses the observability gap by giving data scientists, ML engineers, and platform administrators real-time visibility into how training jobs are performing.This visibility rests on 3 pillars:Dashboard visualization: Real-time progress metrics are visible directly in the Red Hat OpenShift AI dashboard. At a glance, you can see progress percentage, current step and total steps, current epoch, estimated time remaining, training loss, and evaluation metrics for your TrainJobs. No separate tool, no additional setup.SDK integration: The same progress data is available programmatically through the Kubeflow SDK. This enables teams to build automated pipelines that react to training progress—triggering early stopping, sending notifications, or reallocating resources based on real-time metrics.Multi-framework consistency: Progress tracking works the same way whether you're using PyTorch DDP, FSDP, DeepSpeed, or JAX. It also supports custom training code through the CustomTrainer API. This provides a single unified experience across frameworks.Seeing your training in motionThe best way to understand progress tracking is to walk through what using it actually feels like. There are 3 steps that matter.Start your training job: You submit a training job from a notebook using the Kubeflow SDK, exactly as you would today. If you're using HuggingFace Transformers, progress tracking is automatic—no code changes needed. The integration detects the Kubeflow environment and begins reporting metrics from your training loop.Watch its progress: As training runs, real-time metrics appear in the Red Hat OpenShift AI dashboard: progress percentage, estimated time remaining, completed steps, epochs, gradient norm, training loss, learning rate. No log tailing, no SSH-ing into pods, no external tools to set up.
