Configuring HPA using cross project pub/sub metrics

Google Cloud Serverless Applications
1

Hello Team,

I am trying to configure HPA for GKE workload in one of our projects (lets say ProjectA) using the pubsub metrics from another project of ours (lets say ProjectB). I receive the following permission denied error:

Warning FailedGetExternalMetric 9s (x12 over 2m56s) horizontal-pod-autoscaler unable to get external metric default/pubsub.googleapis.com|subscription|num_undelivered_messages/&LabelSelector{MatchLabels:map[string]string{resource.labels.project_id: ProjectB,resource.labels.subscription_id: ProjectB.Subscription,},MatchExpressions:LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: the server could not find the descriptor for metric pubsub.googleapis.com/subscription/num_undelivered_messages: googleapi: Error 403: Permission monitoring.metricDescriptors.get denied (or the resource may not exist)., forbidden

I have Mpnitoring.admin role assigned on both the projects and have pub/sub APIs enabled on both the projects.

Also, I am able to configure HPA for GKE workload in ProjectA using the pubsub metrics from with the ProjectA. I want to not extend the same configuration using metrics from ProjectB.

Can anybody assist me on the issue please?

2

Assuming you are using the Stackdriver custom metrics adapter, you can do this by adding the resource.labels.project_id to the metricSelector:

metrics:
  - external:
      metricName: pubsub.googleapis.com|subscription|num_undelivered_messages
      metricSelector:
        matchLabels:
          resource.labels.subscription_id: super-awesome-sub
          resource.labels.project_id: other-project

You can check out https://github.com/GoogleCloudPlatform/k8s-stackdriver/pull/212 for an example.

3

Hello Garisingh,

Thank you for your reply. I had already added the metricName, and metricSelectors resource.labels.subscription_id and resource.labels.project_id as mentioned but sill get the error. The stackdriver custom metrics adapter as well as the pubsub API are also enabled. I had followed the following KB:
https://cloud.google.com/kubernetes-engine/docs/tutorials/autoscaling-metrics?_ga=2.189193512.-2038413145.1679461982#pubsub_7

What can we check further?

4

Any update on this one? or what was the issue
facing similar error

5

You need to configure the cluster service account or the service account for the custom metrics adapter to have the monitoring metrics viewer role/permissions, monitoring.metricDescriptors.get specifically in your case (the service account will need a few other metrics reading permissions) for the the project that the metrics are in.