Missing count of a histogram counter in Metrics Explorer for prometheus_target

Google Cloud Cloud Foundations & Onboarding
1

It seems the *_count and *_sum metrics that gets tracked for every histogram in prometheus is not visible in Stackdriver explorer under prometheus_target.

For example:-

Let’s say I have an histogram with name save_time_in_seconds. I see these metrics created

save_time_in_seconds_bucket{le="0.3",} 106.0
save_time_in_seconds_bucket{le="1.0",} 106.0
save_time_in_seconds_bucket{le="2.0",} 106.0
save_time_in_seconds_bucket{le="4.0",} 106.0
save_time_in_seconds_bucket{le="9.0",} 106.0
save_time_in_seconds_bucket{le="20.0",} 106.0
save_time_in_seconds_bucket{le="+Inf",} 106.0
save_time_in_seconds_count 106.0
save_time_in_seconds_sum 1.5158215649999998

I can query save_time_in_seconds as histogram using both PromQL and Metrics Explorer/MQL. But I cannot get the save_time_in_seconds_count showing up in Metrics Explorer. It is only available through PromQL. Anyway I can get this to show up in metrics explorer too?

2

Hey Pras,

Histogram data in Cloud Monitoring is stored under a single Distribution point instead of separate time series as in Prometheus. See https://cloud.google.com/monitoring/charts/charting-distribution-metrics for details.

tl;dr you can easily get sum and count series by running a sum or count aggregator on the save_time_in_seconds distribution point, which should show in in Metrics Explorer as prometheus.googleapis.com/save_time_in_seconds/histogram.

2 Likes
3

Thanks for responding @lyanco , appreciate it. Would I be able to do rate on it though? Aligners seems to be restricted in case of using count agg on histogram (tried from UI and MQL on GCP and Grafana Cloud)

Basically an equivalent for

sum(rate(save_time_in_seconds_count[1m]))
4

Yeah - upon further review I think you’re right, you can only do sums and counts on distribution metrics by using the _sum and _count streams in PromQL.