Probability or Confidence Score of a Text Query; Gemini/Vertex AI

Google Cloud Build with AI Custom ML & MLOps
, , ,
1

Is it possible to return the probability or confidence score of a text query? I would like to make a threshold for low confidence responses. I would like this to work with a fine-tuned vertex ai model if possible.

Eg, when the generate_content function is called, return the confidence score alongside the text result.

https://ai.google.dev/api/tuning

Such functionality is discussed here, but has anyone actually implemented this or something similar?

https://cloud.google.com/vertex-ai/docs/text-data/classification/evaluate-model

  • Confidence threshold: A confidence score that determines which predictions to return. A model returns predictions that are at this value or higher. A higher confidence threshold increases precision but lowers recall. Vertex AI returns confidence metrics at different threshold values to show how the threshold affects precision and recall.
2

this link/post seemed promising here:

https://t.com/questions/70070421/return-confidence-score-with-custom-model-for-vertex-ai-batch-predictions

anyone get this or anything similar to work? or the other link I had posted above?

https://cloud.google.com/vertex-ai/docs/text-data/classification/evaluate-model

  • Confidence threshold: A confidence score that determines which predictions to return. A model returns predictions that are at this value or higher. A higher confidence threshold increases precision but lowers recall. Vertex AI returns confidence metrics at different threshold values to show how the threshold affects precision and recall.