Fractional G4 (g4-standard-12/24, vGPU) hitting cudaErrorECCUncorrectable and CUDA_ERROR_LAUNCH_FAILED since late August; full-GPU g4-standard-48 unaffected

Google Cloud Compute Infrastructure
, ,
1

Since roughly Aug 28, 2026 we’ve been seeing a sharp rise in fatal GPU errors on fractional G4 VMs (g4-standard-12 and g4-standard-24, NVIDIA RTX PRO 6000 vGPU) in us-central1-b. Failures ramped up from Sep 7 and now hit dozens of tasks per day. Wondering if anyone else on fractional G4 is seeing the same.

Symptoms

  • GKE (COS, GKE 1.36): PyTorch training pods die with CUDA error: uncorrectable ECC error encountered (cudaErrorECCUncorrectable). About 90 seconds later kubelet reports NodeShutdown and GKE auto-repair recreates the node.
  • Cloud Batch (Ubuntu 24.04 custom image): CuPy tasks die with CUDA_ERROR_LAUNCH_FAILED: unspecified launch failure, typically on a plain cudaHostAlloc, so the context is already dead by the time the error surfaces.
  • Guest kernel logs contain no NVRM: Xid lines at all, so we can’t see the underlying fault. They do contain constant NVRM: updateHostVgpuFbUsage: NVRM_RPC: Get FB usage info failed : 0x40 on every fractional G4 VM.

What doesn’t fail

  • Same container images and workloads on full-GPU g4-standard-48 nodes in the same cluster and zone: zero ECC or launch-failure errors over the same period.
  • Same workloads on L4 (g2-standard-12) and T4: zero.

Versions (unchanged over the whole period)

  • Guest driver on fractional G4: 580.126.09-grid-gcp (GKE: auto-selected by cos-gpu-installer; Batch: installed per the GCP GRID driver doc)
  • Host driver reported via metadata nvidia-host-driver-version: 580.126.24
  • Full-GPU nodes: 580.159.04
  • GKE node image cos-129-19506-224-49, no upgrades in the window

We’re looking into opening a support ticket, but would like to know: is anyone else running fractional G4 seeing ECC or launch failures recently? Did anything change on your side around that time?

2

Hello! Have you tried other driver versions? There’s a newer one available, 580.159.03-grid-gcp.

You can install it as so:

gcloud storage cp gs://compute-gpu-installation-us/drivers/NVIDIA-Linux-x86_64-580.159.03-grid-gcp.run NVIDIA-Linux-x86_64-580.159.03-grid-gcp.run && sudo sh NVIDIA-Linux-x86_64-580.159.03-grid-gcp.run -s