Gemini Flash Fine tuning with tools

Google Cloud Build with AI Custom ML & MLOps
,
1

Hi Everyone!

Can’t find this in documentation and in examples around. Is there a ways to fine tune Gemini Flash 1.5 in Vertex AI with tools? I just see how to fine tune it with text and images but nothing about tools invocation.

Thank you!

2

Hi @nikitSPB87,

Welcome to Google Cloud Community!

Currently, Vertex AI does not offer a direct way to fine-tune the gemini-1.5-flash model specifically with tools invocation. The fine-tuning options are limited to text, image, audio and document data. However, there are a few potential workarounds to consider:

Prompt Engineering:

  • Craft detailed and informative prompts that guide the model towards the desired tool usage.
  • Provide clear instructions and examples of how to use the tools within the prompt.
  • Experiment with different prompt structures and phrasings to optimize tool utilization.

RAG (Retrieval Augmented Generation):

  • Incorporate a knowledge base or document store into your application.
  • Use the model to query this knowledge base and retrieve relevant information.
  • Guide the model to use the retrieved information in conjunction with tool invocations to generate more comprehensive and accurate responses.

Custom Model Development:

  • If you have advanced technical expertise, you could consider developing a custom model that integrates tool usage capabilities.
  • This would involve training a model on a dataset that includes examples of tool invocations and their corresponding outputs.
  • This approach requires significant resources and technical knowledge.

You may also want to consider filing this as a feature request. Please note that I can’t provide any details or timelines at this moment. You may keep an eye on the Vertex AI release notes for new releases and updates.

I hope the above information is helpful.