For anyone else to try. I changed the scheduling trigger name and the bot name itself, and now that particular bot is working again. Not sure which name change did it.
2 Likes
A possible reason (pure speculation!): AppSheet changed how scheduled bots are implemented, the change includes a new configuration structure, you making a change prompted your app to adopt the new configuration structure. If correct, a fix for anyone affected is to make and save a trivial change to an affected bot. You can even undo the change, so long as the changes are saved each time. We have seen this type of fix work in the past.
4 Likes
Just wanted to confirm that the engineering team is investigating this as our top priority. At this time we haven’t been able to identify any code changes on our part that would correlate with this, but the observation that renaming the bot seems to fix it seems important. If anyone else confirms that renaming or some other minor change fixes the issue for you as well, please let us know here.
2 Likes
@Adam-google Just some additional context. My first attempt, I changed my hourly bots minutes after the hour from 10 to 11 for example and then saved. That did not fix it. It was only after I modified the name of the trigger event and the bot to reflect that I had changed the minutes after the hour, and then suddenly it started running. I wish I had done 1 at a time to isolate down whether it was the rename of the trigger event or the bot itself, but maybe this is helpful to you as changing the event trigger time did not solve it.
5 Likes
Could this have been just a coincidence? Your changes occurred at the time AppSheet had resolved the server issue?
@Adam-google
In my case, updating the names didn’t work.
What is interesting, I tried to run those bots manually and now it doesn’t work as well (but it used to work). I am getting a strange error regarding some column that doesn’t exist in my app and actually never even existed:
"Errors": "Error: 42703: column "Settled" does not exist Error: 'Add Row' Data action ''Add Row' Data action 'Action for Create Bills 3'' failed with exception 42703: column "Settled" does not exist 42703: column \"Settled\" does not exist."
I checked all the formulas and references. I do not use it anywhere… Those bots that I am talking about now, were active since 2023. And actually nothing changed since that time.
EDIT
The above issue is on our end but it is caused by the problem with scheduled bots as we are actively rewriting huge amount of logic in a very short time. Sorry for the confusion.
_______
It is beginning of the month, we have quite some workload that needs to be processed manually now in all the apps we maintain. This is a terrible experience. I have no idea what’s going on, when it is going to be resolved, nothing. I really appreciate that this is your “top priority” but it’s been 72 hours and nothing has changed. Please let me know here or via email replying to my ticket when I can expect these errors to be corrected. Apps I am mentioning in all my posts here are Enterprise clients’ apps. We are not free ghost users. For sure not huge companies like the ones you probably take care of but still - we are on the highest possible tiers with those apps. Those errors cost us a lot of money too, actually. It cannot be like this, this is just not fair.
1 Like
Today Scheduled bot not triggered, they took a vacation. No errors or problems, they just don’t start 
1 Like
@Adam-google Also getting this error from Scheduled Bot that is trying to run: “max rows allowed for this app is 10000. Will only process the first 10000 rows”
We have never been limited to 10,000 rows before and this would be a major change if so. Our backing store is AlloyDB so there is no issue on that end.
1 Like
Hello all,
I just got a notice from the engineering team that several bots have been executed successfully today.
Could you please confirm your bots are working fine now?
3 Likes
Not here for sure.
Still waiting for this to work as usual.
2 Likes
The problem relates to an internal system that we rely on for this feature. We have been working with the team that manages this to try to identify the underlying issue and possible remediations. Some attempted fixes had been made over the last few days that turned out to be ineffective, but the most recent ones that were made this afternoon appear to have restored the rate of execution on the order of what we typically saw before the issue, as of about 10 minutes ago. So you may begin to see some improvement now, although it may take some time for the queue to work through the backlog.
9 Likes
It looks like our processes are finally back.
Thank you for help!
Can I have one question?
Have you been aware of this before I started this topic @Adam-google?
4 Likes
Thanks for the confirmation @mateo!
4 Likes
I was made aware of it via another engineer, but it looks like it was initially escalated through our customer support pipeline after a number of people reported the same issue. The community post was cited in that escalation, so the support team was aware of the post and it presumably contributed to their decision to escalate to engineering.
6 Likes
Alright, so it means it is definitely worth posting here.
5 Likes
@Adam-google Is it possible to have proactive monitoring of this system?
Waiting for enough customers to complain I’m sure is not the experience you would want us to have. I don’t know what that system was, but surely there is a way for it to cry uncle when it starts falling over so your team can address in advance of catastrophe like this.
8 Likes
It looks like everything is back on track now. Thanks so much to you and your team for the hard work in getting this sorted out!
3 Likes
Yes, it’s a good point. We do have monitoring and alerting for such things, but it looks like we didn’t have any alerts for a drop in the attempt rate of these scheduled tasks. I just added one, so hopefully we can address it immediately if this happens again.
8 Likes