Enterprise Account - Parallelsim:10 & Google Sheets API per-project/per-minute rate limits

@Jose_Arteaga

Today I started getting multiple errors on different tables without having had any changes made to the app. It has been stable for weeks and months, all Virtual Columns well optimized, not one table has more than 400 rows.

Data Sources: Google Sheets

Degree of Parallelism : 10

Average sync time 6.105 seconds for this month so far calculated through Big Query Audit Trail, that’s not only sync but add, edit, export and app sync. The larger sync times on tables are when adding rows, they contain Files and images, so that’s normal..

Average Virtual Column Computation Time is within acceptable range on our app:

ERRORS occurring recently…

Operation: App sync

Message: Error:Error: Data table '***** Table ’ is not accessible due to: Timed out waiting for a resource. The reason is usually a very large/slow loading table or a cross-table cycle in AppSheet formulas..

Because our Enterprise account has the Degree of Parallelism set to 10, initial app loads generate a high volume of concurrent data-fetching requests. Based on our monitoring, AppSheet appears to be hitting standard Google Sheets API per-project/per-minute rate limits (HTTP 429 / 503) during these peak startup bursts.

So for enterprise accounts this doesn’t make sense…

I am also open to some of the Gurus on forum to pitch in if I am perhaps understanding the Parallelism and a Google API rate limits wrong for App cold starts with multiple users, the times it happens seems to make sense why it doesn’t occur during the day but in mornings. All the field workers sync in morning before going to sites around same time these errors come through.

welcome to my world, iam trying to find solution past 3 months
37 tables / 8 spreadsheets (merging from 12) / Core user

last update from support, 5 days ago

"
We want to let you know that our engineering team is currently working on the fix for this ongoing issue. They have addressed a potential issue with database configuration that made it inefficient in processing some traffic. The metrics for this issue are showing improvement, but we’re continuing to monitor and investigate further.
"

I don’t think there is anything in your app causing a problem. It sounds like it is very lightweight data-wise.

These being web apps, there are MANY factors that can impact the flow of your app data OUTSIDE of the user’s device.

Network connectivity is probably the most common culprit. Bad connections slow data transfer rates. We see these messages all the time in our app and it typically is due to users being in the basement floors of a building where the WiFi and cellular connections are not very good. In my experience, users do not typically see these error messages unless they happen to tap the Sync button at the time the error is occurring.

But there are many other points in the data flow route that can slow down the transfer of data. Appsheets servers, Google servers, database servers and/or third-party service servers - any one of these can encounter an issue on any given day and disrupt the transfer of data and result in these messages.

You mentioned “today” you started seeing these messages. I would monitor them and see if they persist for more than a couple hours. If so, check WHO is getting them and where they might be from a connectivity perspective. If multiple people in different areas are logging the messages then start looking at the services your app interacts with to see if any of those are experiencing problems.

Beyond these things, your guess is as good as mine!!

@WillowMobileSys

Thank you for input, I always appreciate your perspective for these things :slightly_smiling_face:

It happens to random users in the mornings, today got exceptionally worse, when all users get ready to sync to go offsite around same time after a toolbox talk, they are on different locations and different internet connections, usually once first user gets this error, all other users still syncing get this error repeatedly, even myself as admin on editor from a different country gets the error…

Then after a minute or two has passed, slowly but surely each person’s device syncs and the error fades away..

This is why I was wondering if some sort of rate limit is hit.

After a minute or two it seems to start coming right, slowly each device syncs and the requests drop off…

So I have been trying to work out if let’s say 20 users all sync at same time and degree of Parallelism is 10 for enterprise, can the sheets handle the multiple requests to hit the sheet simultaneously.

This never occurred when I was on the Core plan and only started getting these errors when we moved to Enterprise and that is what led me to think about this scenario a bit more…on Core it seems to have had all requests pile up in pipeline with less parallelism, so it was slower but never hit limits.

You mentioned sheets.

Do you have any formulas in the sheets? Are there any scripts running against the sheets at the time these users are collectively accessing the app?

If Google sheets, there are several usage rate limits that COULD be hit if the either of the above are true AND there is no efficiency considered when the sheets are being accessed. Remember that access to the data sheets all funnels through the App Owner account!!

As an example, I built an app for a non-profit group and we used charts in Google for dynamic real-time updates. However, those charts performed queries against the source sheets so when this small group of 20 people started up the app and began inserting rows - those charts were pummeling the data sheets resulting in the query rate limit being reached. A re-design of how we grabbed the data fixed the problem.