Hi everyone,
Many of you have asked for guidance on Gemini Tuning with Vertex AI. Common questions include: “How do I prepare tuning and preference data?” and “How can I measure improvements in specific use cases to verify that fine-tuning with Gemini is beneficial?”
Together with Haichao and Jessica from the Vertex AI Engineering team, we have published two new tutorials on preparing tuning data for Gemini models and using custom metrics to evaluate the resulting tuned models.
In these guides, you will learn to:
- Inject custom metrics, like the F1 score or JSON validation, directly into the Supervised Fine-Tuning loop.
- Execute custom code during the tuning job for more tailored, task-specific performance evaluation.
- Use the Vertex AI Gen AI Evaluation SDK to automatically score your preference datasets and visualize quality distributions.
- Filter out noisy data by creating a clear quality gap between “chosen” and “rejected” responses before running Direct Preference Optimization (DPO).
As always, let me know if you have questions or feedback.
Happy building!
