Recently our push queues are timing out after 1 minute.
The setting for the application is to timeout after 5 minutes. Queues were working fine, some running for 2 minutes. No settings were changed, no version re-deployed.
After 1 minute requests are getting the Deadline Exceeded error.
Python3. App Engine Standard Environment. Gunicorn endpoint and Flask.
Hi @JeffK ,
Welcome to Google Cloud community!
Though you’re using Python3 runtime for your App Engine Standard environment, based on the documentation, DeadlineExceeded errors occur when the request reaches its overall time limit and it could be thrown by the original thread that’s handling that request. This means that it could be from any part of code that happens to be executing internally during that time.
Despite having a configured timeout limit set for 5 minutes (such as for task queue and cron job requests), one of the few possible factors that caused a specific deadline to exceed after 1 minute may be because the standard HTTP request is the one calling a timeout after reaching a 60-sec mark.
The first approach for issues like this is to continue observing how frequent this premature timeout happens for your queues. Is this an intermittent outcome for your push queues, or is this the new norm? Notice if network connectivity is a contributing factor here, or if it’s something we can rule out. Also, you can utilize the Log Explorer from Google Cloud console if there’s more behind the premature timeout issue faced by your deployed queues.
You could also consider migrating your push queues to Cloud Tasks. This supports App Engine Standard Environment (2nd gen) and Python 3 runtime. Aside from better configurable timeouts, longer timeouts are more likely to be guaranteed for your use-case. Using Cloud Tasks can have a deadline of 10 minutes for automatic scaling and 24 hours for manual scaling.
If the issue persists, you may also contact Google Cloud support for a deeper investigation on this issue with your App Engine instance.
I hope the above information is helpful.
I was mistaken they are set up as push type Cloud Tasks. Thats why I was surprised some are timing out after 60 seconds.
As was mentioned they should timeout at 10 minutes but after when switched to python3 (January 2024) the tasks were timing after 5 minutes which the gunicorn timeout setting is set at.
As of Nov 1 some are timing out after 1 minute with no change to settings nor was it re-deployed. The app was last deployed mid September.
The app is also using some Legacy Bundled services.
Some of the tasks do send request to Twilio (sms) or Sendgrid (email).