"Urgent Assistance Needed: Frequent Alert Notifications"

Hi All, This is an urgent issue, and I would greatly appreciate your assistance. I’m receiving multiple incidents from a specific alert in GCP—‘VM_instance_application_error(copy)- customized log based metric alert.’. This alert is generating multiple incidents at short intervals (every 5-10 minutes), which is causing significant disruption to our team.

I need guidance on how to reduce the frequency of these alert notifications and the occurrence of incidents. Ideally, I’d like to see only one alert every two to three hours(e.g., one alert per 2-3 hours).

Is there a way to group these incidents or set the alert to trigger only once per hour with consolidated information? The current situation, where incidents are opening every 5 to 10 minutes, is causing significant disruptions for my team. Any suggestions on the best way to address this would be greatly appreciated.

Our goal is to maintain awareness of issues while reducing alert fatigue. Any best practices or configuration suggestions would be greatly appreciated.

Thank you for your prompt attention to this matter.

Hello @Brahathees ,Welcome on Google Cloud Community.

What are your alerting policy settings for such metric ? Did you’ve tried to change “Rolling window” and “threshold value” ?

More info here: https://cloud.google.com/monitoring/api/v3/aggregation?hl=en&_gl=1v2bmnu_gaOTQxMjM5MjU3LjE3MTM4NTQzNjU._ga_WH2QY8WWF5*MTcyNTI1NzE2My4xNjA1LjEuMTcyNTI1NzE3Ni40OS4wLjA.#aggr-intro

Additionally for better granularity, you could try work with secondary data transformation. So first time series will be applied AND then second one.


cheers,
DamianS
LinkedIn medium.com Cloudskillsboost