Google Cloud Monitoring: Set an alert on a grouping

Google Cloud Cloud Foundations & Onboarding
,
1

I am trying to find a workaround for this issue, which says that PromQL for Google Cloud Monitoring metrics doesn’t support label regexs for integer. @lyanco FYI

In short, supposing status_code is an int label, it is not possible to do something like this:

rate(
  response_count{status_code=~"4.."}[${__interval}]
)

Also, PromQL does not have inequality, so this also does not work:

rate(
  response_count{status_code>="400"}[${__interval}]
)

And there is no way to cast an int label to a string, and THEN perform a regex comparison. Without these capabilities, it’s not possible to express “status_code in the 400 range” in a filter.

But by using label_replace , we can produce a string label from a group of int labels. It looks like this:

sum(
  label_replace(
    rate(
      response_count{label1="value-to-filter-on"}[${__interval}]
    ),
    "status_group",
    "${1}xx",
    "status_code",
    "^(\\d).+"
  )
) by (status_group)

When the initial series has “status_code” labels with int values like 200, 401, 403, 400, 429, etc, this PromQL query gets me a time series with the status_group label (a string) taking values like 2xx and 4xx etc. I can also convert that into a call percentage, with something like this:

sum(
  label_replace(
    rate(
      response_count{label1="value-to-filter-on"}[${__interval}]
    ),
    "status_group",
    "${1}xx",
    "status_code",
    "^(\\d).+"
  )
) by (status_group) /
scalar(
  sum (
    rate(
      response_count{label1="value-to-filter-on"}[${__interval}]
    )
  )
) * 100

And I can also convert it to a square-wave that indicates whether the percentage has exceeded a threshold (let’s say 7%) like this:

sum(
  label_replace(
    rate(
      response_count{label1="value-to-filter-on"}[${__interval}]
    ),
    "status_group",
    "${1}xx",
    "status_code",
    "^(\\d).+"
  )
) by (status_group) /
scalar(
  sum (
    rate(
      response_count{label1="value-to-filter-on"}[${__interval}]
    )
  )
) * 100 > bool 7

But in Google Cloud Monitoring, I cannot find a way to filter the result of THAT. Or to set an alert if and only if the 4xx group exceeds a specific threshold.

Can anyone suggest?

2

Just a heads up that using label_replace() inside an aggregation function works, but it will move the query execution out of the very efficient Monarch query engine and into the much less efficient PromQL query engine. So while your proposed solution might be fine for small amounts of data, you could see timeouts or errors for large amounts of data.

We’re working on replatforming PromQL and the integer issue will be fixed when that’s done. ETA sometime in H1 2025.

3

Thanks for that caution , @lyanco . Duly noted.

4

based on a response to the GitHub issue, I Was able to use the following to get the alert I wanted, in Apigee.

sum(
    rate(apigee_googleapis_com:proxy_response_count{monitored_resource="apigee.googleapis.com/Proxy",response_code="400"}[5m]) or
    rate(apigee_googleapis_com:proxy_response_count{monitored_resource="apigee.googleapis.com/Proxy",response_code="401"}[5m]) or
    rate(apigee_googleapis_com:proxy_response_count{monitored_resource="apigee.googleapis.com/Proxy",response_code="403"}[5m]) or
    rate(apigee_googleapis_com:proxy_response_count{monitored_resource="apigee.googleapis.com/Proxy",response_code="429"}[5m])
    )
/ sum(
    rate(
      apigee_googleapis_com:proxy_response_count{monitored_resource="apigee.googleapis.com/Proxy"}[5m]
    )
)
 * 100 > bool 14