Access Denied (anonymous caller) error when accessing GCS bucket from GKE using Workload Identity

Google Cloud Serverless Applications
1

Hello everyone,

Background

I have a simple a service in Go that receives a request and makes a read request to the GCS bucket. This service is deployed in GKE in the same project as the GCS bucket. GKE is using Workload Identity to make requests to the storage bucket. This workload identity user is bound to one of the Service Accounts which has the storage admin permissions on the bucket that the service wants to read from.

To summarize the setup:

  • Created a Google service account and K8s service account
  • Gave the Google service account permissions to fetch data from the bucket. Though redundant, I also added **roles/storage.**admin role.
gcloud storage buckets add-iam-policy-binding gs://mybucket --member "serviceAccount:google-service-account@$PROJECT.iam.gserviceaccount.com" --role "roles/storage.objectViewer"
  • Added Service account impersonation using this binding for GCP
gcloud iam service-accounts add-iam-policy-binding google-service-account@$PROJECT.iam.gserviceaccount.com \
    --role roles/iam.workloadIdentityUser \
    --member "serviceAccount:$PROJECT.svc.id.goog[namespace/k8sserviceaccount]"
  • Added Service account impersonation using this binding for Kubernetes
kubectl annotate serviceaccount k8sserviceaccount iam.gke.io/gcp-service-account=google-service-account@$PROJECT.iam.gserviceaccount.com -n namespace
Problem

When the service makes a request to the storage bucket, I see this error

"error":"googleapi: got HTTP response code 403 with body: <?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message><Details>Anonymous caller does not have storage.objects.get access to the Google Cloud Storage object. Permission 'storage.objects.get' denied on resource (or it may not exist).</Details></Error>"

I verified that the Workload Identity is working as expected by running the following

kubectl exec -i -t -n namespace gcloud -- /bin/sh
# curl -H "Metadata-Flavor: Google" http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/email
google-service-account@myproject.iam.gserviceaccount.com

I also verified that the Workload Identity User is able to make a read request using this Python script. In fact, read and write both are working correctly using this Workload Identity.

Questions
  • It seems all permissions are working. Is there any additional setup that is needed to make the workload Identity be able to connect to the storage account?
  • Since GKE is connecting to the storage account, does the storage client have to be initialized with any specific option(s) to make it work with workload identity?

Any thoughts/ideas that can help me proceed here are appreciated. Thank you!

1 Like
2

Hey folks ā€“ Iā€™d really appreciate if someone can help me out with this. Thanks!

3

Hi @invisible ,

Welcome to the Google Cloud Community!

I found a StackOverflow thread and this YouTube tutorial that has a similar issue with yours.

Let me know if it worked. Thanks.

4

To close on this thread, it turned out to be an issue in the code. As mentioned in my original post, the problem occurred when making a request to the storage bucket. This was happening because I had initialized the storage client with incorrect options. I had initialized it with [option.WithoutAuthentication()](https://pkg.go.dev/google.golang.org/api/option#WithoutAuthentication) option, which is a valid option for one of the cases in my code. ClientOption is an optional argument when initializing a new storage client. To fix this problem, I just had to provide no explicit client options for this particular case.

4 Likes
5

Thank you, @lawrencenelson ! I appreciate your help. :blush:

4 Likes