GCP Cloud Observability: Monitoring (Managed Prometheus): How to use user labels

Google Cloud Cloud Foundations & Onboarding
,
1

I am trying to use metrics collected automatically by GCP Kubernetes (GKE) using the Managed Prometheus and query them via PromQL in GCP Cloud Observability: Monitoring.

I have certain node pools for different use cases and I would like to create different alerts for node CPU and RAM usage depending on the node pool to which the node belongs.

I have tried the following PromQL query, but it yields no results:


avg_over_time(kubernetes_io:node_cpu_allocatable_utilization{monitored_resource="k8s_node",project_id="hirundo-mvp-dev",location="us-central1",cluster_name="gke-mvp-dev-cluster", metadata_user_cloud_google_com_gke_nodepool="mvp-dev-pool"}[${__interval}])

The issue seems to be with the metadata_user_cloud_google_com_gke_nodepool label. I do not understand how else I can transform the cloud.google.com/gke-nodepool label value shown in the Builder UI as one of the “User Metadata Labels”.

Furthermore, the example from the documentation (https://cloud.google.com/stackdriver/docs/managed-prometheus/promql#metadata-labels)

compute_googleapis_com:instance_cpu_utilization{monitored_resource="gce_instance",metadata_user_env="prod"}

Does not seem to work for me whatsoever.

What am I missing?

2

Hi@ @blewis

To assist us in conducting thorough investigation, we kindly request your cooperation in providing the following information regarding the reported issue:

  1. Has this scenario ever worked as expected in the past?
  2. It would be greatly helpful if you could attach screenshots of the output related to this issue.

Thank you

3

Hi kamblem,

No, it has not worked in the past. I don’t know exactly what help a
screenshot would be. The results table and graph are empty :see_no_evil_monkey: .

4

@kamblem Do you have any ideas why this does not work?