Permission Error: 'discoveryengine.documents.import' when importig a text file from a bukcet to a Ve

Google Cloud Cloud Foundations & Onboarding
1

Hello Google Cloud Community,

I’m encountering a permission error while trying to import a document to a a Vertex AI Datastore.

Here are the details:

  • Error Message: 403 Permission ‘discoveryengine.documents.import’
  • Project Details:
    • Using Google Cloud Datastore
    • Attempting to import documents from a bucket
    • Current document count in project: 426,389 (DEV), 426,389 (PRE), 238,618 (PRO)

I saw in the docs that current quotas are:

  • Documents per project: 1,000,000
  • Documents per organization: 5,000,000

Reference: [Quotas and limits | Vertex AI Agent Builder | Google Cloud]

I’ve verified that:

  • The service account has the necessary IAM roles
  • The API is enabled in our project

Questions:

  1. I’ve noticed that my total document count (1,091,396) exceeds the project quota limit of 1,000,000. What are my options to resolve this?
  2. Is it possible to request a quota increase? If so, what is the maximum limit that can be requested?
  3. Are there any alternative solutions or best practices for handling large document volumes beyond the quota limit?

Any guidance would be greatly appreciated. Thank you!

2

Hi @pedropcamellon ,

Welcome to Google Cloud Community!

The 403 permission error means that the service account that you’re using doesn’t have the necessary permissions to import documents. I understand that you have already checked that the service account has the required IAM roles. However, could you confirm if the source bucket is located in the same project? If you’re importing documents from a bucket located in a different project, you need to give the service account service-@gcp-sa-discoveryengine.iam.gserviceaccount.com “Storage Object Viewer” permissions for the Cloud Storage bucket.

In addition, please see my response in line with your questions below:
1. I’ve noticed that my total document count (1,091,396) exceeds the project quota limit of 1,000,000. What are my options to resolve this?

As the project quota limit has been reached, you will need to request for an additional quota to avoid any possible disruption

2. Is it possible to request a quota increase? If so, what is the maximum limit that can be requested?

Yes, it’s possible to request for a quota increase using the Google Cloud console. However, if you find that you can’t request an adjustment from the console, you will have to request the increase from Cloud Customer Care. The maximum limit depends on the business use case. Keep in mind that quota increase requests are reviewed and approved on a case-to-case basis. For more information, you may check this documentation to help you understand quota adjustments.

3. Are there any alternative solutions or best practices for handling large document volumes beyond the quota limit?

For best practices, you may focus on the following:

  • Prioritize Relevant Documents: Select the documents that are most relevant to your use case.
  • Remove Unnecessary Content: Strip out any irrelevant information from documents
  • Filter Out Ineffective Documents like low-quality content or repetitive content

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.