Getting large amounts of "Unauthenticated request." errors when using GCP Batch.

[vedantroy-genmo]

Task Name: rw-id3779-task4000-par4000-stage0-download.

Task UID: rw-id3779-task4000-637b650e-07c4-429d0.

1942 succeeded, 2056 tasks failed, 2 running. (Total 4K tasks).

I’m getting a lot of errors of the form:

docker: Error response from daemon: Head "[https://us-central1-docker.pkg.dev/v2/${my_project}/gcpbatch-us-central1/${my_repo}/manifests/latest](https://us-central1-docker.pkg.dev/v2/${my_project}/gcpbatch-us-central1/${my_repo}/manifests/latest)": denied: Unauthenticated request. Unauthenticated requests do not have permission "artifactregistry.repositories.downloadArtifacts" on resource "projects/${my_project}/locations/us-central1/repositories/gcpbatch-us-central1" (or it may not exist).

[vedantroy-genmo]

It certainly looks like many (if not most?) of the task failures are due to this error:

[vedantroy-genmo]

Update: Reducing docker image size cut down on failure rate dramatically, so that was likely the issue.

[ket0825]

I guess you are using COS for run the container. And the issue occurs when you try to pull image from artifact registry.

I think your docker config does not have any permission to pull image from artifact registry.

You should modify /root/.docker/config.json to get permissions.

You can use docker-credential-gcr.

Simply, just enter like this.

/usr/bin/docker-credential-gcr configure-docker --registries [REGION]-docker.pkg.dev

But In COS, you cannot modify the /root/.docker/config.json due to the security. You can Read-only at root directory.

So, I used the cloud-init.yaml file to bypass the root config and make new user.

And give access to myuser about docker.

you can reference my cloud-init.yaml file.

#cloud-config

users:

• name: myuser
  uid: 2000

write_files:

• path: /etc/systemd/system/docker-credential-gcr.service
  permissions: 0644
  owner: root
  content: |
  [Unit]
  Description=Configure Docker to use GCR credentials
  Wants=docker.service
  After=docker.service

[Service]
User=myuser
Type=oneshot
ExecStart=/usr/bin/docker-credential-gcr configure-docker --registries [REGION]-docker.pkg.dev
StandardOutput=journal+console
StandardError=journal+console
Restart=on-failure
RestartSec=10s

[Install]
WantedBy=multi-user.target

• path: /etc/systemd/system/group-myuser-docker.service
  permissions: 0644
  owner: root
  content: |
  [Unit]
  Description=Group myuser with docker
  Wants=docker.service docker-credential-gcr.service
  After=docker.service docker-credential-gcr.service

[Service]
User=root
Type=oneshot
ExecStart=usermod -aG docker myuser

[Install]
WantedBy=multi-user.target

• path: /etc/systemd/system/my-app.service
  permissions: 0644
  owner: root
  content: |
  [Unit]
  Description=Run a application container
  Requires=docker-credential-gcr.service group-myuser-docker.service
  After=docker-credential-gcr.service group-myuser-docker.service

[Service]
User=myuser
Type=simple
ExecStart=/bin/bash -c ‘docker run --rm
-u 2000
–name=my-app
[IMAGE_URL]’
ExecStop=/usr/bin/docker stop my-app
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

runcmd:

• systemctl daemon-reload
• systemctl enable docker-credential-gcr.service
• systemctl start docker-credential-gcr.service
• systemctl enable group-myuser-docker.service
• systemctl start group-myuser-docker.service
• systemctl enable my-app.service
• systemctl start my-app.service

you can get some more details below links.

ref: https://cloud.google.com/container-optimized-os/docs/concepts/security

https://cloud.google.com/container-optimized-os/docs/how-to/run-container-instance

https://cloudinit.readthedocs.io/en/latest/

https://medium.com/@benmorel/creating-a-linux-service-with-systemd-611b5c8b91d6