New features of Gemini in AppSheet Solutions are now generally available

[Arthur_Rallu]

Hello AppSheet Community,

We’re excited to announce that a few new capabilities within Gemini in AppSheet Solutions are now Generally Available (GA). Extract Rows and Summarize are joining Extract and Categorize into GA, as well as a few other changes.

As mentioned in previous announcements, Gemini in AppSheet Solutions helps users automate smarter by integrating capabilities like data extraction and categorization directly into AppSheet workflows, reducing manual work and freeing up a team’s time for higher-value tasks. It’s easy to use with pre-built capabilities that you can adapt to specific business needs. The feature also lets you build with confidence through in-editor task testing and gives admins the tools to deploy AI safely and scalably, ensuring governance and compliance across the organization. In July, two initial AI capabilities were rolled out: Extract and Categorize.

We’re now introducing two new AI capabilities:

• Extract Rows - this AI capability is related and similar to Extract. It lets you extract information from an image or a PDF file and save it into multiple rows, for example for extracting line items in invoices

• Summarize - this new AI capability lets you summarize records based on selected attributes, for example for summarizing the key aspects of a customer account.

In addition, scheduled bots that process multiple records now support all four available AI capabilities, when the output data is saved immediately into your data source.

Use Cases

The new AI capabilities and these updates will let AppSheet creators with AppSheet Enterprise Plus support more use cases. Here are 3 common examples:

• Invoice processing: Finance and audit teams can at the end of the month automatically pull information from invoices and receipts, including line items, reducing processing time and mistakes from manual data entry.

• Monitoring of regulatory changes: compliance and Q&A teams can pull updated and new rules from published regulatory documents to stay on top of constantly changing requirements and avoid being out of compliance.

• Speed up resolution of equipment support requests: As technicians raise support requests for malfunctioning equipment with lots of details, including images, support teams can automate extracting equipment serial numbers from images, route requests to the appropriate teams based on the descriptions of the issues and the characteristics of the equipment, and generate a short actionable summary to speed up resolution.

How to Get Access & Start Building

Gemini in AppSheet Solutions is already available to users with AppSheet Enterprise Plus. Simply add a new task to your automation and select the AI task type.

Learn More & Share Your Feedback

While all these capabilities are now GA, we still want to hear your feedback! Understanding what you are trying to accomplish (sometimes very creatively!) on very concrete examples and seeing examples of what works or doesn’t for you gives us the ability to refine the existing capabilities and offer new ones. You can do that by rating results when using the inline testing feature for AI Task!

We’re incredibly excited to see what you’ll do with this!

Arthur on behalf of the AppSheet team

---

FAQ

What else changed?

We have also made a few improvements:

• Extract and Extract rows were updated to allow

  • files with sizes up to 12 MB

  • Images with sizes up to 5MB

• AI tasks were upgraded to allow 30 requests per min, up from 10.

What are the licensing and pricing requirements?

First, your organization needs to have an AppSheet Enterprise agreement. With it, your organization automatically receives a pooled quota of credits, at no additional charge. In particular, no separate Google Workspace license is needed.

Second, creators need to have an AppSheet Enterprise Plus license to include AI tasks into their automation bots and apps. Then, running an AI task will consume credits from the organization’s pool. Credit consumption will vary based on the complexity of the task.

How can I see how many credits are consumed?

Only AppSheet admins can see their full quota of credits, the overall credit consumption, and how Gemini is used throughout the organization via the AppSheet admin console.

If I run two AI tasks in the same automation, how many credits does this automation consume?

Each AI task will consume credits based on its complexity. The two are independent of each other.

For example, let’s say an automation has an Extract AI task and then a Categorize AI task. Today, if the automation works with a 5-page PDF file, the number of credits consumed during that run is 60 credits total - 50 credits for the Extract AI task and 10 credits for the Categorize AI task.

As another example, if this automation was triggered by a scheduled bot for 10 records and each PDF file was 5 pages long, then the total number of credits consumed would be 600 credits - 60 credits for each record.

Troubleshooting - Why won’t my AI task run?

Check the following:

• Do you have AppSheet Enterprise Plus?

• Is there an admin policy in place that prevents you from running AI tasks?

• Check if you are being rate-limited in the audit logs or the automation monitoring app. Check limits here.

• Check if you are not getting any result back in the audit logs or the automation monitoring app.

Troubleshooting - Why does my AI task give me inaccurate results?

AI tasks are able to accomplish tasks to the extent that Gemini is able to. This means that AI tasks have the same limitations. For example, counting a large number of objects in a picture is typically a difficult task for Generative AI models like Gemini (at least for now).

AI tasks may not be familiar with some industries or tasks. A rule of thumb is that if a topic is niche and is not widely discussed online, odds are that a Generative AI model won’t do well. As an example, AI tasks probably can’t recognize machine parts involved in a very specific manufacturing process from an image.

We strongly encourage creators to provide additional instructions to tweak the behavior of an AI task to get more accurate results. However, creators should not need to provide instructions on what the format of the output should look like (e.g. “Return a date with format YYYY/MM/DD” or “Return a JSON-like output with…”) - AppSheet takes care of that behind the scenes.

[Marc_Dillon]

I’m trying to use the AI extract tool to create child records from a table on a PDF, one record per table row. One of the child record’s fields is a Reference to a table of inventory items. I wish for it to select the correct referenced record based on the value in the PDF, but I’m having trouble getting them to match appropriately. I see the following message in the bot task:

For this Ref column, only the label column “label” from table “stock” is used as input to AI task

Does this mean that only the label value is considered when it selects a record to reference? Am I unable to tell it to match the record based on another more descriptive column in the inventory item table, or a category column? I don’t want to make my label values too long.

On a separate subject, I was also trying to use the testing feature, but it kept telling me that “the table or slice is empty, need at least one row for Ref columns”, but that just wasn’t true, the table had hundreds of rows in it. And it seemed to work fine (except for selecting the wrong records) when the bot was actually triggered.

[Arthur_Rallu]

@Marc_Dillon

Does this mean that only the label value is considered when it selects a record to reference? Am I unable to tell it to match the record based on another more descriptive column in the inventory item table, or a category column?

You are correct. Unfortunately, that’s currently a limitation. Your request makes sense though.

Now let me follow up with 2 questions for you: (a) what would you want to be able to specify instead of the label column? is it another single column, multiple other columns, or the whole record? (b) what’s the use case?

[Arthur_Rallu]

@Marc_Dillon

it kept telling me that “the table or slice is empty, need at least one row for Ref columns”, but that just wasn’t true, the table had hundreds of rows in it. And it seemed to work fine (except for selecting the wrong records) when the bot was actually triggered.

Have you tried to click on “Select Row” and then “Refresh Data”. The tester does not access your data source the same way your app does. If this doesn’t work, can you DM me a video or screenshot. Thank you

[Marc_Dillon]

@Arthur_Rallu

Now let me follow up with 2 questions for you: (a) what would you want to be able to specify instead of the label column? is it another single column, multiple other columns, or the whole record? (b) what’s the use case?

I had expected to be able to provide a plain-language version of any basic SELECT expression in the extra instructions box. Like if I was creating these records elsewhere, for the stock Ref column I would have used something like:

FILTER( stock , AND( [category]=[pdf column1] , [description] = [pdf column2] ) )

In the extra instructions box for the AI extract task, I would have assumed I could have written something like “For the stock Ref column, look up the stock record whose label, description, and category most closely matches the first two column values from the PDF”.

@Arthur_Rallu

Have you tried to click on “Select Row” and then “Refresh Data”. The tester does not access your data source the same way your app does. If this doesn’t work, can you DM me a video or screenshot.

Yes, I selected a row with a PDF from that popup window, then hit the test button at the very bottom (which btw was very hard to discover, being at the very bottom). Perhaps it’s due to the table having a security filter on it that depends on the user accessing the app. I’ll send a screenshot later maybe if I still can’t get it to work with this in mind.

[Marc_Dillon]

I was able to get the tester to work by completely removing the security filter on the stock table. Enabling the “bypass” option in the bot’s event did not work. I’m not sure if there’s maybe another way? I would hope so.

But after that, the tester is actually really nice!

Similarly, I had a security filter on the table that the bot is based on. And I had a required column on the child table, with an initial value formula that depends on the parent record. The testing function seemingly wasn’t even able to access the selected parent record, due to this security filter, because I kept getting the “required column not provided a value” error. Removing the parent’s security filter got rid of this error. So it seems like there’s a lot of room for improvement on this testing feature when security filters are in use.

So then finally I was able to see warning outputs from the extract tester, telling me how it found multiple matching rows for the Ref columns, which leads me to another apparent issue that I would have expected to work differently. As I’ve already described, in the records to be generated there is a “stock” Ref column, to be filled by the extract step. After that, there is another Ref column “stock_uom”, to also be filled by the extract step. stock_uom is a child table of stock. For the stock_uom column, I have a Suggested Values expression so that it only shows stock_uom records associated to the selected stock record. The extract step appears to be searching the entire stock_uom table for a match, instead of just the limited set of records that are associated to the matched stock record. I would have expected it to work within the Suggested Values output, just like an app user would.

[Akkipatel95]

@Arthur_Rallu

However, creators should not need to provide instructions on what the format of the output should look like (e.g. “Return a date with format YYYY/MM/DD” or “Return a JSON-like output with…”) - AppSheet takes care of that behind the scenes.

You have to provide instructions for EnumList column as it’s not processing it correctly.

This field is enumlist. It should have brought ANE57106. But got it like this so I had to give instruction. And I believe Initial Values are not getting generated if you don’t pass it in AI Task for Lookup() or select(). It only works for TODAY(), NOW(),USEREMAIL(), UNIQUEID() etc.

[Arthur_Rallu]

Thanks for highlighting the issues of the tester with security filters. I’ll share that back with the team. My guess is that the tester is implemented outside of the bot and so it’s missing contextual information that the bot would have.

I’m not sure I have the data schema properly in my mind based on your description, so I’ll be curious to double-check that with you so I understand the use case better. I understand the general request though: you essentially want the formulas specified in the data schema (and in particular “Suggested Values”) be taken into account by the extraction AI task. You don’t want Gemini to have to consider all the rows from a table when extracting for a REF column as they are not all relevant and you’ve already done the work of filtering them out!
We don’t have that set up today. The formulas from the data schema are generally not passed to Gemini and the data itself is not pre-processed with these formulas before sending them to the AI task.

I’m glad you’re sharing this though. One of our concerns with allowing to pass any number of columns from a child table was the amount of data passed to Gemini. If your child table has thousands and thousands of records and we let you select a lot of columns, that quickly can become problematic. We limited that by mapping to the “Label” column. For other considerations tied to your app, probably to facilitate ease of use, you implemented the “Suggested Values” to filter out unnecessary (or invalid) records which would also limit the amount data we would pass.

I think what you are asking makes sense and it might be ok w.r.t data volume. We started though with the simplest approach which honestly is to not have to preprocess all the info with the various expressions in the data schema.