Please share Gemini tokenize information

Google Cloud Build with AI AI APIs
,
1

hello.

Thank you so much for the recently announced Gemini-Pro API availability.

We use a lot of APIs, and in the case of OpenAI, we expose cl100k_base so that users can pre-calculate the number of tokens and avoid API errors.

But Gemini-Pro doesn’t know anything about the token, so it has to rely on the character count. :sweat_smile:

Is it possible to share token information like OpenAI’s tiktoken?

Thank you for creating good model. :smiley:

3 Likes
2

With the Vertex AI SDK (python) – we compute the number of tokens (and characters) like:

Token Count Docs
It looks like this:

from vertexai.preview.generative_models import GenerativeModel
gemini_pro_model = GenerativeModel(“gemini-pro”)
print(gemini_pro_model.count_tokens(“why is sky blue?”))
I do miss having a local implementation that we could use like tiktoken, and it will be greater if it exists (I am not aware of it) —
I hope it helps.

1 Like
3

Hey!

  • You can now count tokens locally with the Vertex AI SDK for Python (starting with version 1.57.0).
  • Check out this Medium article for details: Counting Gemini text tokens locally.
4 Likes
4

Thank you so much

Now i can request just once not twice anymore ^^

1 Like
5

how to do this with nodejs?

1 Like