The 3 AM PagerDuty Dilemma: Who moved my cheese… I mean training job
by Michael Zhang, SWE and Eddie Villalba, PM - GKE Observability
Meet Alex, a Lead AI Platform Engineer. It is 3:00 AM on a Tuesday, and the glow of her monitors is the only light in the room. She is staring at a Grafana dashboard that looks more like abstract art than a reliable system state. Her team is currently training a massive model on a Google Kubernetes Engine (GKE) cluster powered by a fleet of reliable v5e Cloud TPUs. But something is amiss. Some training jobs take an eternity to initialize, while others seemingly crash at random.
It is the classic machine learning engineering dilemma: is there a subtle, diverging bug in the PyTorch code, or did a highly sophisticated, liquid-cooled physical accelerator just bite the dust?
Training massive Large Language Models (LLMs) or running high-throughput inference serving represents a significant investment in specialized AI hardware, such as Cloud TPUs and GPUs. On GKE, running these massive workloads involves complex orchestration using JobSets—an open-source Kubernetes API designed to manage groups of replicated pods as a single logical workload. For platform operators and machine learning engineers like Alex, ensuring the reliability and efficiency of these large clusters is paramount. Unplanned interruptions, slow node provisioning, or long workload startup times directly impact time-to-market and compute budgets.
When things break at this scale, it feels like herding exceptionally expensive, computationally dense cats. You need more than just standard CPU metrics; you need deep, specialized telemetry.
Stabilizing the Present: Mastering Workload Lifecycle Metrics
To stabilize her current v5e workloads, Alex turns to the foundational Workload Lifecycle (WLI) metrics in GKE. These metrics focus on three core aspects of cluster operations: Startup Latency, Reliability & Resilience, and Workload Scheduling Mapping.
These metrics provide native, out-of-the-box visibility into cluster startup speeds, reliability KPIs, and workload-to-node scheduling mappings at both the Node Pool (infrastructure) level and the JobSet (workload) level. Let’s break down the exact telemetry Alex uses to tame her clusters:
1. Startup Latency Metrics
How long does it take from the moment resource creation is requested to the moment resources are ready for execution? Monitoring this helps isolate scheduling bottlenecks, slow image pulling, or VM provisioning delays.
-
kubernetes.io/node_pool/accelerator/startup_duration: Measures the duration in seconds from a node pool’s creation to its initial availability.
-
kubernetes.io/jobset/startup_duration: Measures the duration from a JobSet’s creation to its initial availability.
2. Reliability & Resilience (TBI and TTR)
AI workloads are prone to interruptions—whether due to spot VM preemption, hardware faults, or application-level exceptions. GKE now calculates two key industry metrics automatically: Time Between Interruptions (TBI), which is the duration of stable uptime periods between consecutive failures, and Time to Recover (TTR), the duration of downtime periods from the start of an interruption until the system or workload is fully recovered.
-
kubernetes.io/node_pool/accelerator/times_between_interruptions and kubernetes.io/jobset/times_between_interruptions: Distribution of stable uptime intervals between consecutive failures.
-
kubernetes.io/node_pool/accelerator/times_to_recover and kubernetes.io/jobset/times_to_recover: Distribution of durations taken for the node pool or JobSet to recover from a downtime event.
-
kubernetes.io/jobset/uptime: The total cumulative time that the JobSet has been available.
3. Workload Scheduling Mapping
To debug performance differences or isolate faulty zones, platform operators need to know exactly which workloads are scheduled on which physical nodes.
-
kubernetes.io/node_pool/assigned_jobsets: Emits True if at least one JobSet pod is scheduled in the node pool; False is written when the node pool becomes empty.
-
kubernetes.io/jobset/assigned_node_pools: Maps a JobSet to the GKE node pools where its pods are scheduled.
-
kubernetes.io/jobset/assigned_nodes: Maps a JobSet to the specific node names where its pods are scheduled.
The Scenario: Queue Preemption
Alex, our AI Platform Engineer was paged by an ML Engineer who was running a long-term, low-priority LLM training job. At 9:57 AM, a high-priority interactive job was submitted. The queue manager (Kueue) decides to preempt the job to free up TPU quota.
Instead of deleting the job, Kueue suspends the JobSet.
The Scale-Down
Since the TPUs are no longer in use, GKE Autopilot automatically scales the idle TPU node pool down to 0 nodes to save costs. The system remains in this state for 5 minutes.
The Resume and Scale-Up
At 10:02 AM, the high-priority job finishes, and Kueue resumes the training job (suspend: false). GKE Autopilot triggers a scale-up to provision new TPU VMs, which takes 3 minutes. At 10:05 AM, the nodes are ready, the pods are scheduled, and training resumes.
Visualizing the Timeline (Overview Dashboard)
The Overview dashboard captures this sequence of events:
-
At 9:57 AM: The Suspended state (green line) spikes to 1, and Active replicas drop to 0. JobSet readiness drops to 0.
-
At 10:02 AM: The Suspended state drops back to 0.
-
From 10:02 AM to 10:05 AM: Readiness remains at 0 (the scale-up delay).
-
At 10:05 AM: Readiness returns to 1, and uptime begins to climb.
The Cost of Preemption (Analytics Dashboard)
The Analytics dashboard calculates the impact of this preemption:
- Node Pool MTTR: ~3 minutes. This represents the infrastructure recovery time (the scale-up duration from 10:02 to 10:05).
- JobSet MTTR: ~8 minutes. This represents the total downtime for the training application (5 minutes suspension + 3 minutes scale-up).
-
JobSet MTBI: ~1 minute. The time between the job becoming stable/active (~9:55 AM) and the interruption/suspension (9:56 AM) was exactly 1 minute (60 seconds).
-
Re-deployment (9:53 AM): We recreated the JobSet at 9:53 AM. The pods were created but remained Pending while GKE Autopilot scaled up the TPU nodes.
-
Going Active (~9:55 AM): The scale-up finished, and the pods became Running and Ready at approximately 9:55 AM. (You can see this as the first small green peak on the JobSet replica readiness chart in your first image).
-
Suspension (9:56 AM): We ran the suspend command at 9:56 AM, terminating the pods and transitioning the availability to false.
-
Conclusion (Part 1): Key Takeaways & Next Steps
Workload Lifecycle (WLI) metrics turn opaque cluster behaviors into actionable operational telemetry. Tracking startup duration, MTTR, and MTBI equips platform teams to isolate infrastructure bottlenecks from application exceptions instantly. Relying solely on standard CPU/memory metrics risks misdiagnosing node preemptions as code failures, leading to wasted engineering hours and unnecessary compute expenditure. Implementing these lifecycle metrics establishes baseline stability before scaling up infrastructure.
Enable system metrics forwarding in Cloud Monitoring today to gain visibility into your GKE TPU clusters. Audit your job startup latency and MTTR baselines to identify hidden queueing or provisioning overheads before your next big model run.
Preview: Part 2 — Navigating TPU v7 and Dynamic Slicing
While WLI metrics stabilize fixed-shape cluster deployments, next-generation TPU architectures require a different approach to telemetry. In Part 2, we shift focus to TPU v7 workloads and GKE Dynamic Slicing—examining how to trace jobs down to physical reservation subblocks and manage hardware-level dynamic topology changes.