Issues Pulling Artifact images to an autopilot Kubernetes deployment

Google Cloud Serverless Applications
1

I created a Cluster as always and an artifact repository, but when I try for Kubernetes to pull the images I keep getting the error ImagePullBackOff and ErrImagePull.

Failed to pull image “us-central1-docker.pkg.dev/…”: failed to pull and unpack image “us-central1-docker.pkg.dev/…”: failed to resolve reference “us-central1-docker.pkg.dev/…”: failed to authorize: failed to fetch oauth token: unexpected status from GET request to… 403 Forbidden

I have configured this kind of setup several times in the past and I never encountered this problem.

1 Like
2

According to the documentation, the Compute Engine default service account is used by the nodes to interact with other services like Artifact Registry if a specific service account is not set when creating the GKE cluster.
To troubleshoot the problem, ensure that the default service account has the Artifact Registry Reader role (roles/artifactregistry.reader).

4 Likes
3

@albetancourt 's answer is the correct one. Just to add to this, since it wasn’t mentioned above:

I was hitting this issue as well because I was trying to pull images between separate GCP projects. It turns out I just needed to set everything up against the default compute service account (or just create a new cluster from scratch, specifying a different service account). This is because:

  1. The service account in GKE Autopilot clusters cannot be changed once the cluster is created and
  2. The service account used for image pulling cannot be overridden

I believe this is because it’s fully managed and therefore locked down for security reasons. Unfortunately, I couldn’t see any details in the error logs explaining why it was not working; merely that it was “Forbidden”, even though logging into the pods themselves revealed I had the access I needed. It was very confusing until I found this. Thanks @albetancourt !

1 Like
4

Hi @borjagomez

Could you please review the answers provided? If they are helpful in resolving the issue, feel free to select the correct answer to mark the case as solved.

Additionally, you can use Workload Identity to address this issue. It’s a recommended approach for securely managing identity and access when GKE clusters interact with Google Cloud services, such as pulling images from Artifact Registry. https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity

5

The ImagePullBackOff and ErrImagePull errors with a 403 Forbidden message indicate that your Kubernetes cluster does not have the necessary permissions to pull images from your Google Artifact Registry. Ensure that the service account used by your Kubernetes nodes has the required permissions to access the Artifact Registry repository.

  • Grant the roles/artifactregistry.reader role to your node service account:

    bash
    gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \ –member=“serviceAccount:YOUR_NODE_SERVICE_ACCOUNT” \ –role=“roles/artifactregistry.reader”

  • If you’re unsure of your node service account, find it using:

    bash
    gcloud container clusters describe YOUR_CLUSTER_NAME --region YOUR_REGION \ –format=“value(nodeConfig.serviceAccount)”

1 Like