helm pull from artifact registry fails after successfull login (in 90% of cases)

Google Cloud Compute Infrastructure
1

Hi,

I am trying to pull a package from Google’s Artifact Registry following this documentation.
After the successful login helm pull fails with Error: failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden but sometimes it succeeds. I wasn’t able to figure out how and why. Service account, which key I am using, has all required permissions.
Here is the full log (successful):

cat key.json | helm registry --debug login -u _json_key_base64 --password-stdin https://europe-west3-docker.pkg.dev/engineering-368717/helm-registry && helm pull --debug oci://europe-west3-docker.pkg.dev/engineering-368717/helm-registry/staging/email-service --version 0.1.0
Login Succeeded
DEBU[0000] resolving                                     host=europe-west3-docker.pkg.dev
DEBU[0000] do request                                    host=europe-west3-docker.pkg.dev request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=Helm/3.10.3 request.method=HEAD url="https://europe-west3-docker.pkg.dev/v2/engineering-368717/helm-registry/staging/email-service/manifests/0.1.0"
DEBU[0000] fetch response received                       host=europe-west3-docker.pkg.dev response.header.alt-svc="h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"" response.header.content-length=102 response.header.content-type=application/json response.header.date="Fri, 20 Jan 2023 15:26:39 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.www-authenticate="Bearer realm=\"https://europe-west3-docker.pkg.dev/v2/token\",service=\"europe-west3-docker.pkg.dev\",scope=\"repository:engineering-368717/helm-registry/staging/email-service:pull\"" response.header.x-content-type-options=nosniff response.header.x-frame-options=SAMEORIGIN response.header.x-xss-protection=0 response.status="401 Unauthorized" url="https://europe-west3-docker.pkg.dev/v2/engineering-368717/helm-registry/staging/email-service/manifests/0.1.0"
DEBU[0000] Unauthorized                                  header="Bearer realm=\"https://europe-west3-docker.pkg.dev/v2/token\",service=\"europe-west3-docker.pkg.dev\",scope=\"repository:engineering-368717/helm-registry/staging/email-service:pull\"" host=europe-west3-docker.pkg.dev
DEBU[0000] do request                                    host=europe-west3-docker.pkg.dev request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=Helm/3.10.3 request.method=HEAD url="https://europe-west3-docker.pkg.dev/v2/engineering-368717/helm-registry/staging/email-service/manifests/0.1.0"
DEBU[0000] fetch response received                       host=europe-west3-docker.pkg.dev response.header.alt-svc="h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"" response.header.content-length=353 response.header.content-type=application/vnd.oci.image.manifest.v1+json response.header.date="Fri, 20 Jan 2023 15:26:39 GMT" response.header.docker-content-digest="sha256:28b201f65198b6f5ecc401c6c58ccd5767bb89c619b288aa9b606ade4500d4dc" response.header.docker-distribution-api-version=registry/2.0 response.status="200 OK" url="https://europe-west3-docker.pkg.dev/v2/engineering-368717/helm-registry/staging/email-service/manifests/0.1.0"
DEBU[0000] resolved                                      desc.digest="sha256:28b201f65198b6f5ecc401c6c58ccd5767bb89c619b288aa9b606ade4500d4dc" host=europe-west3-docker.pkg.dev
DEBU[0000] do request                                    digest="sha256:28b201f65198b6f5ecc401c6c58ccd5767bb89c619b288aa9b606ade4500d4dc" request.header.accept="application/vnd.oci.image.manifest.v1+json, */*" request.header.user-agent=Helm/3.10.3 request.method=GET url="https://europe-west3-docker.pkg.dev/v2/engineering-368717/helm-registry/staging/email-service/manifests/sha256:28b201f65198b6f5ecc401c6c58ccd5767bb89c619b288aa9b606ade4500d4dc"
DEBU[0000] fetch response received                       digest="sha256:28b201f65198b6f5ecc401c6c58ccd5767bb89c619b288aa9b606ade4500d4dc" response.header.alt-svc="h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"" response.header.content-length=353 response.header.content-type=application/vnd.oci.image.manifest.v1+json response.header.date="Fri, 20 Jan 2023 15:26:39 GMT" response.header.docker-content-digest="sha256:28b201f65198b6f5ecc401c6c58ccd5767bb89c619b288aa9b606ade4500d4dc" response.header.docker-distribution-api-version=registry/2.0 response.header.x-content-type-options=nosniff response.header.x-frame-options=SAMEORIGIN response.header.x-xss-protection=0 response.status="200 OK" url="https://europe-west3-docker.pkg.dev/v2/engineering-368717/helm-registry/staging/email-service/manifests/sha256:28b201f65198b6f5ecc401c6c58ccd5767bb89c619b288aa9b606ade4500d4dc"
DEBU[0000] do request                                    digest="sha256:7404113340c566b9200059ba22aa8f8074d299794b2178c7efc309fa8c34222c" request.header.accept="application/vnd.cncf.helm.chart.content.v1.tar+gzip, */*" request.header.user-agent=Helm/3.10.3 request.method=GET url="https://europe-west3-docker.pkg.dev/v2/engineering-368717/helm-registry/staging/email-service/blobs/sha256:7404113340c566b9200059ba22aa8f8074d299794b2178c7efc309fa8c34222c"
DEBU[0000] do request                                    digest="sha256:2083e87173b09ed1d66ac5d35b08dfe581f7c2f938cc2f4f045eb8b98b410abe" request.header.accept="application/vnd.cncf.helm.config.v1+json, */*" request.header.user-agent=Helm/3.10.3 request.method=GET url="https://europe-west3-docker.pkg.dev/v2/engineering-368717/helm-registry/staging/email-service/blobs/sha256:2083e87173b09ed1d66ac5d35b08dfe581f7c2f938cc2f4f045eb8b98b410abe"
DEBU[0000] fetch response received                       digest="sha256:2083e87173b09ed1d66ac5d35b08dfe581f7c2f938cc2f4f045eb8b98b410abe" response.header.accept-ranges=bytes response.header.alt-svc="h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"" response.header.content-length=137 response.header.content-type=application/octet-stream response.header.date="Fri, 20 Jan 2023 15:26:40 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.x-content-type-options=nosniff response.header.x-frame-options=SAMEORIGIN response.header.x-xss-protection=0 response.status="200 OK" url="https://europe-west3-docker.pkg.dev/v2/engineering-368717/helm-registry/staging/email-service/blobs/sha256:2083e87173b09ed1d66ac5d35b08dfe581f7c2f938cc2f4f045eb8b98b410abe"
DEBU[0000] encountered unknown type application/vnd.cncf.helm.config.v1+json; children may not be fetched 
DEBU[0000] fetch response received                       digest="sha256:7404113340c566b9200059ba22aa8f8074d299794b2178c7efc309fa8c34222c" response.header.accept-ranges=bytes response.header.alt-svc="h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"" response.header.cache-control="private, max-age=0" response.header.content-length=3952 response.header.content-type=application/octet-stream response.header.date="Fri, 20 Jan 2023 15:26:40 GMT" response.header.expires="Fri, 20 Jan 2023 15:26:40 GMT" response.header.server=UploadServer response.header.x-goog-hash="crc32c=DYoQ4w==" response.header.x-guploader-uploadid=ADPycdu3XwUND8_8d27U5Rr9aO6e0RnD3yT6M_lbQXZvdK5Yf2bwEbsUVl9h7PZ6r26MBH0qBYV9uakQT2EqfRVBOYQuej5StMOT response.status="200 OK" url="https://europe-west3-docker.pkg.dev/v2/engineering-368717/helm-registry/staging/email-service/blobs/sha256:7404113340c566b9200059ba22aa8f8074d299794b2178c7efc309fa8c34222c"
DEBU[0000] encountered unknown type application/vnd.cncf.helm.chart.content.v1.tar+gzip; children may not be fetched 
Pulled: europe-west3-docker.pkg.dev/engineering-368717/helm-registry/staging/email-service:0.1.0
Digest: sha256:28b201f65198b6f5ecc401c6c58ccd5767bb89c619b288aa9b606ade4500d4dc

And unsuccessful:

cat key.json | helm registry --debug login -u _json_key_base64 --password-stdin https://europe-west3-docker.pkg.dev/engineering-368717/helm-registry && helm pull --debug oci://europe-west3-docker.pkg.dev/engineering-368717/helm-registry/staging/email-service --version 0.1.0
Login Succeeded
DEBU[0000] resolving                                     host=europe-west3-docker.pkg.dev
DEBU[0000] do request                                    host=europe-west3-docker.pkg.dev request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=Helm/3.10.3 request.method=HEAD url="https://europe-west3-docker.pkg.dev/v2/engineering-368717/helm-registry/staging/email-service/manifests/0.1.0"
DEBU[0000] fetch response received                       host=europe-west3-docker.pkg.dev response.header.alt-svc="h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"" response.header.content-length=102 response.header.content-type=application/json response.header.date="Fri, 20 Jan 2023 15:26:43 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.www-authenticate="Bearer realm=\"https://europe-west3-docker.pkg.dev/v2/token\",service=\"europe-west3-docker.pkg.dev\",scope=\"repository:engineering-368717/helm-registry/staging/email-service:pull\"" response.header.x-content-type-options=nosniff response.header.x-frame-options=SAMEORIGIN response.header.x-xss-protection=0 response.status="401 Unauthorized" url="https://europe-west3-docker.pkg.dev/v2/engineering-368717/helm-registry/staging/email-service/manifests/0.1.0"
DEBU[0000] Unauthorized                                  header="Bearer realm=\"https://europe-west3-docker.pkg.dev/v2/token\",service=\"europe-west3-docker.pkg.dev\",scope=\"repository:engineering-368717/helm-registry/staging/email-service:pull\"" host=europe-west3-docker.pkg.dev
DEBU[0000] do request                                    host=europe-west3-docker.pkg.dev request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=Helm/3.10.3 request.method=HEAD url="https://europe-west3-docker.pkg.dev/v2/engineering-368717/helm-registry/staging/email-service/manifests/0.1.0"
INFO[0000] trying next host                              error="failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden" host=europe-west3-docker.pkg.dev
Error: failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden
helm.go:84: [debug] failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden

Output of helm version:

version.BuildInfo{Version:"v3.10.3", GitCommit:"835b7334cfe2e5e27870ab3ed4135f136eecc704", GitTreeState:"clean", GoVersion:"go1.18.9"}

Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.8", GitCommit:"fdc77503e954d1ee641c0e350481f7528e8d068b", GitTreeState:"clean", BuildDate:"2022-11-09T13:38:19Z", GoVersion:"go1.18.8", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.7-gke.900", GitCommit:"e35c4457f66187eff006dda6d2c0fe12144ef2ec", GitTreeState:"clean", BuildDate:"2022-10-26T09:25:34Z", GoVersion:"go1.18.7b7", Compiler:"gc", Platform:"linux/amd64"}
1 Like
2

Hi @Valerii ,

Welcome to Google Cloud Community!

It seems that the issue is related to the authentication of the service account you are using. The error message Error: failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden suggests that the service account does not have the correct permissions to pull the package from the registry.

One possible reason for this issue is that the service account’s JSON key is being cached by the system and not being refreshed. To fix this, you can try clearing the cache and logging in again.

Another possibility is that the service account’s roles or permissions have been modified since the last successful pull, which would prevent the service account from being able to access the package. In this case, you should check that the service account has the necessary roles and permissions to access the package, and if necessary, update the roles or permissions.

It’s also worth checking if there is any rate limiting or request quota enforced on the registry, which might cause the 403 error when requesting too many times in a short period.

The official documentation for pulling packages from Google’s Artifact Registry using Helm can be found here: https://cloud.google.com/artifact-registry/docs/helm
This page provides an overview of the process, including information on how to authenticate with the registry, configure Helm to use the registry, and pull packages from the registry.

Additionally, you may find this page helpful: https://cloud.google.com/artifact-registry/docs/helm/authentication
It explains how to authenticate using a JSON key, which is the method you are using in your example. This page also includes information on how to troubleshoot authentication issues.

You may also find it helpful to check the Helm’s official documentation for troubleshooting: https://helm.sh/docs/intro/troubleshooting/.
It contains information about common issues and solutions for working with Helm, including information about pulling packages from registries.

Thanks

3

Thank you for the answer.

This service account has permissions:

Artifact Registry Writer
Artifact Registry Reader
Artifact Registry Repository Administrator
Artifact Registry Administrator

Also, if it was a permission issue, it wouldn’t work successfully in 10% of cases.

It looks like the cache issue, but as far as I understood - the service account’s json keys are permanent. I do the authentication as described here: https://cloud.google.com/artifact-registry/docs/helm/authentication#json-key

How can I check rate limiting and request quota for the registry?

2 Likes
4

Hi,

I encountered the same problem, but with greater unsuccesful rate (1 call ok on a total of 5-6).

I follow all authentication method and all perform bad. Is there some hidden quota for encrypted registry?

1 Like
5

Same issue here. Trying to use a Service Account JSON token, using the same authentication steps in the docs mentioned above. It works maybe once in a blue moon, but usually fails with “401 Unauthorized”. My Service Account token has the “Artifact Registry Administrator” permission.

See https://github.com/helm/helm/issues/11415 for reference

6

We are having the same issue. It was ok last Friday. It behaves exactly like this. Only a small number of requests went through.

7

@s8w I have found success with the workaround in this comment from the issue I posted above: https://github.com/helm/helm/issues/11415#issuecomment-1289477275 - which is helpful, but still obnoxious that this is the only way to get helm commands to work with GAR (and from what I’ve read GCS as well).

8

Thank you. We did a little further investigation. It turns out that when I destroyed the terraform state of the GKE cluster, I also destroyed the GCP service accounts. Even if I recreated the GCP service account with the same name but the resource id was new. It appears that the permission got lost due to that. Today the permission was added again and it solved the problem.

9

I see - that is an interesting issue. When you say, “the permission got lost,” what do you mean by that? Was the permission(s) that were attached to your Google SA standard ones, or was it a custom role?

10

It is the IAM permissions granted to this GCP service account on Artifact Registry. Pretty standard reader roles.

11

I found an issue a few days after posting the answer. Unfortunately time passed and I forgot what I did. Sorry, guys.

The only thing I can say - that’s not a bug (at least in my case). I was doing something wrong.

12

Thanks @Valerii - it seems like your original issue is actually a bit different from the one I’m experiencing / mentioned in https://github.com/helm/helm/issues/11415 (401 unauthorized).

13

I encountered the same error on my end and when I finally deleted my helm configuration folder (~/.config/helm), access worked fine. There were various entries in the config.json of the registry directory whose paths were only marginally different, but which apparently caused this error.