Alert policies: ${metric.label.value} null for promQL policies

Google Cloud Cloud Foundations & Onboarding
1

I have several promql alert policies.

Sample request:

round(histogram_quantile(0.99, increase(run_googleapis_com:container_cpu_utilizations_bucket{monitored_resource="cloud_run_revision"}[1m])) * 100 , 0.01) > 90

The result returned is done as expected ( a list of fields + “Value” variable well defined)

On the alert policy, I have a documentation text field containing:

The cloud run CPU utilization for [${resource.label.revision_name}] has exceeded 90% for over 5 minutes. Current CPU usage is [${metric.label.value}%]

When the alert trigger is reach, the alert is created (incident creation OK).

The variable ${resource.label.revision_name} is well defined with the field value for the could run revision name as expected but the field concerning ${metric.label.value} is replaced by “Null” rather thant the promql value that trigger the alert…

The cloud run CPU utilization for [my_ressource_name] has exceeded 90% for over 5 minutes. Current CPU usage is [(Null)%]

Several months ago, I designed about 20 promql policies like that (cpu / ram / network parameters etc…) , the variable ${metric.label.value} was always well defined with the promql value.

Since several days or weeks (i don’t know exactly), the returned value for ${metric.label.value} is always “Null” even if the data is well returned by the promql request.

Can you tell me if Google has updated the way they manage the Value field for promql queries ? The GCP documentation about variables confirms that ${metric.label.value} stays the solution to put the promql “Value” field in the documentation.

Many thx for your help, I’m stuck in this since several days ! :slightly_smiling_face:

Chris

PromQL request:

Policy configuration 1.jpg
Policy configuration 1.jpg691×392 41.2 KB

Returned data:

capture returned value.jpg
capture returned value.jpg1357×688 56.8 KB

Documentation field defined:

Documentation text field.jpg
Documentation text field.jpg798×525 40.8 KB

1 Like
2

Hi @Christophe31fr ,

Welcome to the Google Cloud Community!

Thanks for outlining the issue in detail. Given that ${metric.label.value} is correctly configured but now returns Null, this may be due to missing time series in your PromQL expression output, rather than a problem with variable substitution itself.

As explained in this PromLabs blog post, Prometheus can return an empty result if the label combination referenced hasn’t been emitted yet (e.g., if the Cloud Run revision hasn’t produced any metric samples in the alerting window). This commonly happens when a metric with custom labels hasn’t been initialized or when the associated target has not emitted data for that label combination yet.

To confirm that this is the cause, you can follow the Managed Prometheus troubleshooting steps to check if data is missing or if the query returns no series. If the metric is indeed missing, one workaround is to add a PromQL expression that provides a fallback value such as or on() vector(0), which converts null as zero.

If this seems like a regression or an undocumented behavior change, I’d recommend filing a report either via the GCP issue tracker or by opening a GitHub issue on the Prometheus Engine repo. Include steps to replicate, configuration settings, screenshots, and other details of the issue to help us investigate. While there isn’t a specific timeframe for resolution, once we’ve fixed an issue in production we’ll indicate this and then update and close the bug.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.

1 Like
3

Checking my alerts emails (notification by Email), I notice that the problem occurs after the 17th of april.

The latest alerts with the value displayed in the notification was 17th of april at 15h09 French time, the next alert (17th of april at 16h09 French time) had the null value for the metric.label.value field. After the 17th of april, all the usage of the field value is always set to null for all of my alerts (and I change nothing in it… and this is the case for a lot of common alerts … cpu … memory… redis parameters…etc…)

Someone can check that a prometheus on GCP has not be put in place on this date ?

1 Like
4

For your information, I also created an issue on the Google side

https://issuetracker.google.com/issues/415748355

5

Prometheus checking done. No “null” value on the prom result side. All is OK. I asked to Google to investigate this issue (ticket created on GCP Issue tracker).

1 Like