Prometheus managed GKE stopped collecting metrics after Micrometer dependency update

Google Cloud Cloud Foundations & Onboarding
1

Hi,

I use Prometheus to monitor Java microservices. After updating the Micrometer dependency, the collections stopped working. I updated the following versions:

implementation 'io.micrometer:micrometer-registry-prometheus:1.12.3'
implementation 'io.micrometer:micrometer-tracing-bridge-otel:1.2.3'

Previously, I used:

implementation 'io.micrometer:micrometer-registry-prometheus:1.11.5'
implementation 'io.micrometer:micrometer-tracing-bridge-otel:1.1.6'

I tested a few things and the path /actuator/prometheus returns data in both the new and old versions of Micrometer. I’m a bit lost, does anyone have any ideas?

Additional information:

  • I am using Spring Boot 3.2.3.
  • I am using the Micrometer Console to monitor the metrics.

I have already tried:

  • Checking the Prometheus logs for errors.
  • Checking the application logs for errors.

I am still not sure what the problem is.

Any help would be appreciated.

1 Like
2

Hello @julio-costa ,

Welcome to Google Cloud Community!

You might want to double-check the application logs for any specific errors or warnings related to Micrometer or Prometheus after the update. Look for keywords like micrometer, prometheus, metrics, registration or error.

Make sure your Micrometer configuration hasn’t changed unintentionally during the update process. Verify that metrics are still being registered with the micrometer-registry-prometheus registry.

If you find anything that would likely be useful, don’t hesitate to post back here with any questions. We are delighted to try and help.

Thanks!

3

Hi,
I’m facing the same issue. It appears that spring boot 3.2 changed to use exemplars with metrics. This functionality was introduced in prometheus 2.43 - unfortunately Managed Prometheus collectors seem to be at 2.41 (our images in the collectors are at prometheus:v2.41.0-gmp.10-gke.1). If you check your (Cluster)PodMonitorings you’ll see the endpointStatuses say things like:
lastError: metric name X does not support exemplars

This has the splendid effect of dropping all metrics for that endpoint :disappointed_face:
If anyone googly has ideas on how to make the collectors use a later version of prometheus, that would be great