Alert is not generated though it is shown in the graph. Java -> Micrometer -> statsd-exporter -> Prometheus-to-sd -> GCP

Google Cloud Cloud Foundations & Onboarding
1

I am sending the metric to the GCP from my java service to statsd-exporter to prometheus-to-sd to GCP.

I have enabled the springboot to generate the metrics for all the RestAPI call. I have the mapping file in statsd-exporter to map the metric to the exact format required for the prometheus. I have mapped timter type from micrometer spring to summary in prometheus. Prometheus then sends it to GCP with type Cumulative Int64.

I have alert policy for this with conditions

  1. Rolling windows 1 min.
  2. Rolling window function delta
  3. threshold > 0
  4. alert if any time series violates the condition

I see that the alert is generated properly.

Now I have customized the metric to add one more parameter e.g. tenant = “somename“. Also changed the metric name. Now it stopped working.

I can see that the metric is reported on the GCP and also see the delta spikes in the graph. But GCP is not generating the alert.

I am using below code to add the custom metric to spring metric. What could be the reason.

@Component
public class CustomWebMvcTagsContributor implements ObservationFilter {

    @Override
    public Observation.Context map(Observation.Context context) {
        if (context instanceof ServerRequestObservationContext serverContext) {
            // Extract tenant from request attribute (set by JwtTokenFilter)
            String tenant = extractTenantFromAttribute(serverContext);
            if (tenant != null && !tenant.isEmpty()) {
                context.addLowCardinalityKeyValue(io.micrometer.common.KeyValue.of("aabbaabb", tenant));
            }
        }
        return context;
    }
    
    /**
     * Extract tenant from request attribute set by JwtTokenFilter
     */
    private String extractTenantFromAttribute(ServerRequestObservationContext context) {
        try {
            Object tenant = context.getCarrier().getAttribute(CommonConstants.TENANT);
            return tenant != null ? tenant.toString() : null;
        } catch (Exception e) {
            // If extraction fails, just return null
        }
        return null;
    }
    
}