Stop Training Blind: Scaling AI with the New OpenTelemetry-Based TPU AI Telemetry Collector Agent

[Sohamn_Chatterjee]

Author: Sohamn Chatterjee Ehsan Nasiri

When training large-scale foundation models, compute efficiency directly dictates your competitive velocity. Every minute your multi-node, multi-slice training run sits idle or suffers from a silent hang, budgets burn and milestones slip.

In high-performance computing, you cannot manage what you do not measure. Yet, historically, monitoring TPU hardware has often meant dealing with siloed, proprietary, or hidden monitoring tools.

To break down these barriers, Google Cloud has launched the new AI Telemetry Collector agent. Based on the architecture outlined in our unified design frameworks, Towards a Unified OpenTelemetry-Based Observability Agent for the TPU Ecosystem and TPU Observability, this agent standardizes TPU monitoring on open-source standards. For the first time, you have the flexibility to route high-fidelity TPU hardware telemetry to Google Cloud Monitoring, Google Managed Prometheus, or your own self-hosted Grafana stack—simultaneously and with near-zero overhead.

Why This Matters: From Telemetry to Training ROI

If you are running enterprise-scale ML workloads, the new GCE TPU Monitoring Agent delivers direct operational and financial value:

1. Eliminate “Silent” Failures: Multi-slice workloads can bottleneck due to single-node “stragglers” or network latency variations. By collecting real-time device-to-host transfer latencies and collective end-to-end network latencies, you can identify and isolate unhealthy nodes before they drag down your entire training epoch.

2. Optimize Cluster Sizing with OpenTelemetry: Built as a specialized OpenTelemetry (OTEL) Collector, the agent utilizes a unified codebase across GCE. It reuses highly optimized receivers (such as the tpuruntime and tpuhost recievers) to capture granular device utilization without dragging down host CPU cycles.

3. Zero-Cost Operational Metrics: All standard metrics generated under the compute.googleapis.com namespace are classified as system metrics. You can monitor your core hardware performance footprint 24/7 without incurring custom metric ingestion costs.

4. Consolidated GCE Experience: Compute telemetry (like host CPU and RAM usage) is now handled natively by GCE’s primary monitoring agent. This ensures that whether you are profiling a CPU VM, a GPU node, or a TPU VM, your platform telemetry experience remains entirely uniform.

Architecture: How Telemetry Flows

The agent functions as a high-performance guest-space daemon that splits, processes, and securely routes your metrics.

Collection: The tpuruntime receiver scrapes active workload metrics (duty cycle, HBM footprint) via a local gRPC metrics server inside the guest space, while the tpuhost receiver accesses the PCIe bus mapping directly to fetch TensorCore and memory bandwidth utilization regardless of workload activity.
Processing: The Resource Detection Processor stamps incoming metrics with instance metadata (project_id, zone, instance_id). The Transform Processor then normalizes the metric paths (e.g., mapping internal structures to standard GCM names).
Exporting: The metrics are securely dispatched to Monarch (Google’s global monitoring database) or exposed as standard PromQL scraping targets.
Deployment: Getting the Agent on Your Nodes

1. Google-Optimized Ubuntu Images (Pre-Installed)
   If your VMs run on Google’s official, TPU-optimized Ubuntu LTS images, no installation steps are required. The agent is pre-installed as a system snap package and boots automatically.
   Opt-out: If you ever need to manually stop the collector, simply run:
   sudo snap stop gce-tpu-monitoring-agent

2. Custom Operating Systems (Bring Your Own OS)
   For teams maintaining custom OS environments, the agent is distributed as a lightweight Docker container. Because the agent must interface with host hardware buses to decode metric values, it requires container host privileges:

docker run --privileged --net=host \
  -v /sys:/sys \
  -v /path/to/config.yaml:/etc/gce-tpu-monitoring-agent/config.yaml \
  us-central1-docker.pkg.dev/cloud-tpu-images/monitoring/gce-tpu-monitoring-agent:latest

Two Scenarios: Viewing Your TPU Metrics

The agent is designed to support two distinct observability patterns, matching your existing enterprise logging and monitoring configurations.

Scenario A: Google Cloud Monitoring (Native GCM)

Your metrics are routed directly to GCP out-of-the-box.

• How to view: Navigate to the Google Cloud Console and open Monitoring > Metrics Explorer. Select PromQL or MQL as your query engine.

• Pre-built dashboards: Google Cloud automatically provisions curated dashboards for TPU system health, allowing you to instantly visualize cluster health state shifts (HEALTHY, DEGRADED, UNHEALTHY) without manual setup.

Scenario B: Custom/Self-Hosted Prometheus & Grafana

If you rely on a self-hosted Prometheus scraper and visualize your telemetry inside a custom Grafana setup, you can expose a pull-based endpoint on the TPU VM.

Step 1: Expose the Port

Provide a configuration snippet that mounts a Prometheus scrape endpoint to the standard OTel port 9464.

prometheus-local.yaml:
exporters:
  prometheus:
    endpoint: "0.0.0.0:9464"

service:
  pipelines:
    metrics:
      receivers: [tpuruntime, tpuhost]
      exporters: [googlecloud, prometheus]

Apply the configuration and restart the agent.

Step 2: Open VPC Firewall Ports

By default, GCP blocks external ingress traffic to port 9464. To allow your centralized Prometheus server to scrape your TPU nodes safely:

1. Assign a Network Tag: Ensure your TPU VM instances are provisioned with a network tag, such as tpu-monitoring-node.

2. Create Ingress Firewall Rule:

gcloud compute firewall-rules create allow-prometheus-scrape \
  --direction=INGRESS \
  --priority=1000 \
  --network=YOUR_VPC_NETWORK \
  --action=ALLOW \
  --rules=tcp:9464 \
  --target-tags=tpu-monitoring-node \
  --source-ranges=YOUR_PROMETHEUS_SERVER_IP/32

1. (Note: Restricting the source range directly to your Prometheus server IP prevents exposed scrapers from leaking hardware stats to the public internet).

Actionable PromQL for Grafana Dashboards

Once your scrapers are gathering metrics from your private Prometheus target, use these queries to build rich Grafana dashboards:

1. Identify Idle Workloads (TensorCore Saturation)

If your TensorCore utilization falls below expected thresholds, your training pipeline is likely bottlenecked by data loading, host-to-device bottlenecks, or checkpointing.

• Query: avg(compute_googleapis_com_instance_tpu_accelerator_tensorcore_utilization) by (instance_id)

• Visual Type: Time-Series graph or Stat Gauge. Threshold alerts set below 70% during active training run times.

2. Monitor High Bandwidth Memory (HBM) Saturation

Avoid Out-of-Memory crashes before they happen by keeping track of real-time memory pressure.

• Query: sum(compute_googleapis_com_instance_tpu_accelerator_memory_used) by (instance_id) / sum(compute_googleapis_com_instance_tpu_accelerator_memory_total) by (instance_id) * 100

• Visual Type: Bar gauge. Set alert thresholds at 95% to warn of impending OOM crashes.

3. Track Straggler Slices via Collective Latency

In multi-slice setups, one slow link can bottleneck thousands of other chips. Identify networking anomalies instantly.

• Query: histogram_quantile(0.99, sum(rate(compute_googleapis_com_tpu_multislice_network_collective_end_to_end_latencies_bucket[5m])) by (le, instance_id))

• Visual Type: Heatmap. Look for outliers climbing above the cluster’s baseline latency.

Next Steps

To plan your migration path, review the official metrics schemas and deployment tutorials over on our Cloud TPU Observability Documentation. Leverage the new agent to stop training blind and start driving peak performance out of your hardware clusters today!