Memory limit of 4096 MiB exceeded but I can't see it in metrics

Google Cloud Serverless Applications
1

Hello, I’ve Node20 api app with 4GB of ram, And it seems that I’ve memory leak somewhere because I don’t think is possible to the api use over 4gb per container.

So im getting this full error:
Memory limit of 4096 MiB exceeded with 4300 MiB used

Im trying to read the metrics so I can understand at what points I had the memory increase without free and cause the memory leak, But in the metrics the containers never reach over 80% and in some other cases its 90% !

‏لقطة الشاشة 2024-10-25 في 9.53.30 م.png
‏لقطة الشاشة 2024-10-25 في 9.53.30 م.png1384×631 56 KB

2

Hi @IbrahimSluma ,

Welcome to the Google Cloud Community!

Since you mentioned you’re using containers, I’ll assume you’re deploying your app on Cloud Run. From the metrics graph you shared, it looks like you had an out of memory (OOM) error with the instance. For Google Cloud Run instances, a high value (near 80%) for this metric is commonly resolved by provisioning more memory for the instance.

Google Cloud Monitoring uses percentile and distribution-valued metrics for its graphs, with samples taken every one minute. The graph shows that the instance terminated itself as the 99th and 95th percentile memory exceeded the 4096 MiB threshold. You can confirm this by checking the Cloud Logging textPayload for the following error message:

While handling this request the container instance was found to be using too
much memory and was terminated. This is likely to cause a new container instance
to be used for the next request to this revision. If you see this message
frequently, you may have a memory leak in your code or may need more memory.
Consider creating a new revision with more memory.

Do note that the following count towards the available memory of your instance in Cloud Run:

  • Code loaded into memory to run the service
  • Writing to the file system
  • Extra processes running in the container such as an nginx server
  • In-memory caching systems such as the PHP OpCache
  • Per request memory usage
  • Shared in-memory volumes

Here are a few best practices and general tips on managing memory:

  • Delete temporary files. Files written to disk consume memory otherwise available to your service, and can persist between invocations. Failing to delete these files can eventually lead to an out-of-memory error and a subsequent cold start.
  • Optimize memory for services. If you’re using concurrency for your service, you should also increase the memory limit to account for peak usage.
  • Use in-memory volume mounts for services. Consider using volume mounts for memory instead of your instance’s in-memory file system for file reads and writes.
  • Follow general development tips. These tips can help improve your app’s performance on Cloud Run.

I hope this helped!

1 Like
3

Hey @Rhett , Sorry for late reply as I was testing various of different things

Indeed I use fs but I do cleanup files, and I’ve checked and confirmed they being cleaned, After debugging more I found that the issue from rss is keep growing without any release. ( even when I run docker container locally ) so is not related to cloud run.

The weird thing is that when I run my node application locally ( Mac M1 ) the rss clean up in few seconds after requests but don’t clean in docker container. ( I Opened issue here ).

Thank you for your help, I’ll keep this open till I fix the solution