Sudden "cannot execute binary file" error in Google Cloud Batch (Since March 1st)

Google Cloud Compute Infrastructure
1

Hi,

I’m reaching out to see if anyone else has been experiencing issues with Google Cloud Batch starting around March 1st.

I have a stable job that has been running without any issues, but suddenly it started failing with the following error:
textPayload: “/bin/bash: /bin/bash: cannot execute binary file”

This is happening even though I haven’t changed my job configuration or my Docker images. Here is our setup:

  • Machine Type: n1-standard-8

  • GPU: NVIDIA Tesla T4 (1 count)

  • Region: us-central1

  • Image: Custom image based on Standard Ubuntu

What I have tried so far:

  1. Architecture Check: Verified that both the instance (n1) and the Docker image are using amd64. I confirmed this by running docker inspect on the image.

  2. Entrypoint Variations: I tried changing the entrypoint to /bin/bash, /bin/sh, and directly to python3.11, but all of them result in the same “cannot execute binary file” error.

  3. Base Image Change: I tried switching from a vanilla Ubuntu image to other types of images, but the problem persists.

  4. Clean Build: I rebuilt the image from scratch using --no-cache and pushed it to the registry again, but it didn’t help.

It seems like the system is suddenly failing to recognize standard binaries as executable within the container runtime. Since this started on March 1st without any changes on my side, we suspect there might have been a platform-level update that affected how entrypoints are handled.

Is anyone else seeing this behavior? Any suggestions on how to resolve this would be greatly appreciated.

Thanks!

4

The issue has been resolved without any changes to the execution environment.