Use of retry in vertexai Gemini

Google Cloud Build with AI Custom ML & MLOps
,
1

When using the google-generativeai SDK in python, I can add retry logic using:

from google.generativeai import GenerativeModel
from google.api_core import retry

model = GenerativeModel(model_name=model_name)

response = model.generate_content(
    prompt,
    request_options=dict(retry=retry.Retry(initial=0.1, multiplier=2, timeout=61)),
)

However, request_options seems not available in vertexai.generative_models.GenerativeModel from google-cloud-aiplatform.

How can I add automatic retrys (exponential backup) using vertexai?

2

Hi @Peterrrrrrr,

Welcome to Google Cloud Community!

The ‘request_options’ isn’t available in the vertexai.generative_models.GenerativeModel from google-cloud-aiplatform. This is because Vertex AI’s API handles retry logic differently than the google-generativeai SDK.

To add automatic retries with exponential backoff using vertex AI library from google-cloud-aiplatform, here are several steps that you might find helpful in implementing retry logic with exponential backoff:

  • Initialize the Vertex AI Client: Set up the Vertex AI client with your project details. This allows you to interact with the Vertex AI services.
  • Define a Retry Strategy: A retry strategy determines how your application will handle retries when a request fails. Key components of a retry strategy include:
  • Initial Delay: The time to wait before the first retry.
  • Maximum Delay: The longest time to wait between retries.
  • Multiplier: The factor by which the delay increases after each retry (exponential backoff).
  • Deadline: The total time allowed for all retries combined.- Apply the Retry Strategy: You need to apply this retry strategy to the function that makes the request to the Vertex AI model. This can be done using a decorator or by manually implementing the retry logic within the function.
  • Handle Exceptions: When making the request, you should handle any exceptions that occur. If a request fails, the retry logic will automatically retry the request according to the defined strategy.
  • Make the Request: Finally, you call the function that makes the request to the Vertex AI model. If the request fails, it will be retried according to the retry strategy until it either succeeds or the total retry time exceeds the deadline.

You can read through this documentation for more information regarding exponential backoff.

I hope the above information is helpful.