Mastering multimodal embeddings mathematics: Cross-modal physics, audio-visual math, and mRAG architecture in Google Vertex AI

[aniketagrawal]

A comprehensive, research-backed engineering deep-dive into the mathematics, backend architecture, and productionization of Google’s Multimodal Embeddings across Text, Image, Video, and Audio.

(Note: For a deep-dive into unimodal text models, Matryoshka dimensions, and text-based semantic dilution, read our Definitive Guide to Enterprise Text Embeddings here).

---

In the first era of Generative AI, enterprise systems operated in a unimodal vacuum: text searched for text. Today, the world’s advanced cognitive engines—powered by Google Cloud’s Vertex AI—operate in a Joint Multimodal Latent Space.

With the release of gemini-embedding-2 and the legacy multimodalembedding@001, Vertex AI maps discrete text tokens, spatial RGB pixel matrices, temporal video frames, and audio frequencies into the exact same continuous geometric hypersphere. This allows a text string (“A red sports car drifting in rain”) to mathematically locate a specific 3-second timestamp inside a 1-hour MP4 video file, completely bypassing metadata, tags, or OCR.

This is the all-out, end-to-end, diamond-quality guide to the lifecycle of Google’s Multimodal Embeddings. We cover theoretical cross-modal physics, GCP-specific API limits, token mathematics, and practical mRAG use-cases.

For engineers and researchers, a dedicated Mathematical Sandbox (Appendix) at the end of this blog contains 9 rigorous, step-by-step calculations proving the physics behind these systems.

---

1. The evolution of multimodal AI in GCP: gemini-embedding-2

Google Cloud offers two primary multimodal embedding models, each with distinct architectural physics:

1. multimodalembedding@001: The legacy powerhouse. It outputs a fixed 1408-dimensional vector. For video, it relies on an intervalSec configuration, dictating pricing tiers (Essential \ge 15s, Standard 8-14s, Plus 4-7s).
2. gemini-embedding-2: The state-of-the-art unified model. It accepts interleaved inputs across image, text, document (PDF), audio, and video modalities. It shares a massive 8,192 token context window across all modalities and outputs a default 3072-dimensional vector (configurable down to 128 via Matryoshka Representation Learning).

Because both models project into the same mathematical space, the L2-normalized vector for the image of a dog \mathbf{v}_{img}, the sound of a bark \mathbf{v}_{aud}, and the text “A dog barking” \mathbf{v}_{txt} share near-identical coordinates.

---

2. The cross-modal geometry: The N-Tower Architecture

To compare fundamentally mismatched data types (1D text, 2D images, 3D video, 1D audio waves), modern multimodal architectures utilize an N-Tower Architecture (building on the ALIGN/CLIP paradigm).

1. The Vision Tower (ViT): An image is sliced into 16 \times 16 pixel “patches”. These patches are flattened into vectors, linearly projected, and processed via self-attention.
2. The Text Tower: A standard Transformer encoder processes text tokens.
3. The Audio Tower: Audio waveforms are converted into 2D Mel-spectrograms (mapping frequency over time) and processed using an Audio Spectrogram Transformer (AST).
4. The Projection Head: All towers output to a dense projection layer (a learned weight matrix W) that forces their final dimensions to match identically.

\mathbf{v}_{joint} = W_{proj} \cdot \text{Encoder}(x)

---

3. The backend workflow: Contrastive Vision-Language-Audio Pretraining

You cannot train a multimodal embedding model on unlabelled data alone. The backend architecture relies on the foundational principles of CLIP (Radford et al., 2021), Google’s ALIGN (Jia et al., 2021), and AudioCLIP (Guzhov et al., 2021).

During training, an N \times N batch of multi-sensory data is processed simultaneously. The model calculates the cosine similarity between every modality in the batch, forming an N \times N similarity matrix.

The InfoNCE Contrastive Loss is applied across the axes (e.g., Image-to-Text):

\mathcal{L}^{(I \rightarrow T)} = - \frac{1}{N} \sum_{i=1}^{N} \log \frac{\exp(\mathbf{I}_i \cdot \mathbf{T}_i / \tau)}{\sum_{j=1}^{N} \exp(\mathbf{I}_i \cdot \mathbf{T}_j / \tau)}

This loss function turns the correct Image-Text pair into opposite magnetic poles (they attract) and turns all other pairs in the batch into identical poles (they repel). (See Appendix: Example 2 for the full matrix calculation).

---

4. Dimensionality and Matryoshka Representation Learning (MRL)

gemini-embedding-2 outputs a default 3072-dimensional vector. However, searching 100 million 3072-dimensional vectors requires immense RAM.

Through Matryoshka Representation Learning (MRL), the model packs the most critical semantic information into the initial dimensions. Using the Vertex AI API, you can pass output_dimensionality=128 or 768. The API seamlessly truncates the vector.

Crucially, GCP documentation notes that for gemini-embedding-2, output embeddings are automatically L2 normalized for non-default dimensions, ensuring cosine similarity calculations remain mathematically sound.

---

5. The physics of image embeddings: OCR and Visual Dilution

How does a continuous image become a discrete vector? The Vision Transformer (ViT) bypasses traditional CNNs by treating image patches exactly like text tokens. (See Appendix: Example 3).

GCP best practice: Prompt engineering for OCR

gemini-embedding-2 acts as a powerful Optical Character Recognition (OCR) engine. However, you must guide the latent space. If you input an image of the word “cat” written on a wall:

• To search for the animal: Prompt with "picture of a cat".
• To search for the typography: Prompt with "the text 'cat'".

Visual Semantic Dilution (the clutter problem)

When the ViT processes patches, it averages (pools) them to create one single vector. If the image contains too many distinct subjects (a car, a dog, a house), the vector is pulled in conflicting directions, sitting at the mathematical centroid of those concepts. (See Appendix: Example 5 for the mathematical proof).

---

6. The physics of video embeddings: Token math and temporal pooling

Embedding a video requires a 3D spatiotemporal approach. gemini-embedding-2 processes video by sampling frames and audio, but it is strictly bound by the 8,192 token context window.

The GCP token consumption formula

All modalities share the context window. Tokens are consumed as follows:

• Video Frame: 66 tokens per frame.
• Audio: 25 tokens per second.
• Timestamps: 10 tokens per second (formatted as “mm:ss”).

If you process a video at 1 FPS with audio extraction enabled, 1 second of video = 101 tokens. Therefore, the absolute maximum video length you can embed in a single request is \approx 81 \text{ seconds}. (See Appendix: Example 4).

To map this sequence into a single vector, the backend performs Temporal Attention Pooling: \mathbf{v}_{video} = \sum_{t=1}^{T} \alpha_t \mathbf{h}_t. (See Appendix: Example 6).

---

7. The physics of audio embeddings: Spectrogram Geometry

How does a continuous 1D soundwave become a 3072-D vector?

1. Fourier Transform: The raw audio waveform is processed using a Short-Time Fourier Transform (STFT) to create a Mel-spectrogram. This converts the 1D audio into a 2D image where the X-axis is Time, the Y-axis is Frequency (Hz), and the color intensity is Amplitude (dB).
2. Patching: This “audio image” is sliced into 16 \times 16 patches.
3. Encoding: The Audio Spectrogram Transformer (AST) processes these patches. (See Appendix: Example 7 for the STFT matrix math).

---

8. Asymmetric Retrieval: Task instructions in gemini-embedding-2

In enterprise search, we face Asymmetric Retrieval. A query is typically a short question (“What is the 401k match?”), while a document is a long passage.

In older models, you used a task_type API field. In gemini-embedding-2, you must use prompt instructions. By appending a task-specific prefix, you project the vector into a specialized geometric subspace.

• Search Query: task: search result | query: {content}
• Search Document: title: {title} | text: {content}
• Classification (Symmetric): task: classification | query: {content}

Note: GCP explicitly warns that the dot product of embeddings is a similarity metric, not a calibrated probability. Use a sigmoid function over the dot product for strict classification tasks.

---

9. Vector compression & indexing: ScaNN & Scalar Quantization

Exact KNN (K-Nearest Neighbors) requires O(N) dot-product calculations. Across 1 billion multimodal vectors, this is physically impossible for real-time latency. Enterprise Vector DBs use Approximate Nearest Neighbors (ANN) via ScaNN and HNSW (Hierarchical Navigable Small World) graphs.

To further compress storage, vector databases mathematically convert 32-bit floats (FP32) to 8-bit integers (INT8) via Scalar Quantization (SQ). This yields a 4x memory footprint reduction with typically less than 0.5\% accuracy loss. (See Appendix: Example 8 for the quantization math).

---

10. The multimodal RAG (mRAG) gold standard architecture

You have 10 million product images, schematic PDFs, and MP4 tutorials. You want a user to ask Gemini: “Which part of the engine is broken in this picture?”

The architecture:

1. Ingestion: Embed all product images, videos, and PDFs using gemini-embedding-2. Index them in Vertex Vector Search using ScaNN.
2. Query: The user uploads an image of a broken engine.
3. Image-to-Anything Search: The system generates a vector for the user’s uploaded image. It searches the Vector DB in <50\text{ms}.
4. Context Assembly: It retrieves the top 5 most similar images, schematic text chunks, and video segments.
5. Reasoning: All retrieved multi-modal assets are passed into Gemini 2.5 Pro (which natively understands interleaved media) via a single prompt.
6. Output: Gemini generates a flawless, reasoned diagnostic response.

---

11. Production reality: Monitoring cross-modal concept drift

Visual and audio data drift faster than text. The image of a “cell phone” in 2005 maps to a totally different visual cluster than a “cell phone” in 2026.

We mathematically detect drift using the Wasserstein Distance (Earth Mover’s Distance) across the principal components of the latent space:

W_1(P, Q) = \inf_{\gamma \in \Pi(P, Q)} \mathbb{E}_{(x, y) \sim \gamma} [\|x - y\|]

If the Wasserstein distance exceeds your standard deviation threshold, an automated alert fires, signaling the need to re-embed your active catalog. (See Appendix: Example 9).

---

12. Engineering implementation: Google GenAI SDK workflow

For modern applications on the Gemini Enterprise Agent Platform, generating Multimodal Embeddings requires the google-genai SDK. Here is how to generate a dimensionally-reduced (128-D) embedding for an audio file.

import numpy as np
from google import genai
from google.genai import types

# 1. Initialize the client
client = genai.Client(vertexai=True, project="YOUR_PROJECT_ID", location="us-central1")

# 2. Define Multimodal Input (Audio)
content = types.Content(
    parts=[
        types.Part.from_uri(
            file_uri="gs://cloud-samples-data/generative-ai/audio/Chirp-3-Docs-Dive.mp3",
            mime_type="audio/mpeg",
        ),
    ],
)

# 3. Generate the 128-D Joint Embedding (MRL Truncation)
response = client.models.embed_content(
    model="gemini-embedding-2",
    contents=[content],
    config=types.EmbedContentConfig(output_dimensionality=128),
)

# 4. Output validation & L2 Norm Check
embedding_values_np = np.array(response.embeddings[0].values)

print(f"Embedding length: {len(embedding_values_np)}") # Outputs 128
print(f"Norm of embedding: {np.linalg.norm(embedding_values_np):.6f}") # Outputs ~1.000000

---

13. Valid references & citation links

1. CLIP (Contrastive Language-Image Pretraining): Radford, A., et al. (2021). Learning Transferable Visual Models From Natural Language Supervision. OpenAI. arXiv:2103.00020
2. ALIGN: Jia, C., et al. (2021). Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision. Google Research. arXiv:2102.05918
3. ViT (Vision Transformer): Dosovitskiy, A., et al. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. Google Brain. arXiv:2010.11929
4. AudioMAE / AST: Gong, Y., et al. (2021). AST: Audio Spectrogram Transformer. MIT. arXiv:2104.01778
5. Vertex AI Official Docs: Google Cloud. (2026). Get Multimodal Embeddings. Google Cloud Architecture Center. View Documentation
6. Wasserstein Distance in ML: Arjovsky, M., et al. (2017). Wasserstein GAN. Courant Institute, NYU. arXiv:1701.07875

---

---

APPENDIX: The mathematical sandbox

This section contains 9 rigorous, research-level mathematical calculations proving the physics of multimodal embeddings. These examples simplify complex tensor operations into understandable, step-by-step arithmetic.

Example 1: The Universal Coordinate Mapping (Cross-Modal Dot Product)

How do different modalities align? Assume a simplified 3-dimensional latent space representing [Animal: Dog, Action: Barking, Environment: Indoor].

• Text Input: “A dog barking” \rightarrow \mathbf{v}_{txt} = [0.88, 0.90, 0.10].
• Audio Input: A .wav file of a bark \rightarrow \mathbf{v}_{aud} = [0.89, 0.92, 0.05].

The Math (Cosine Similarity between Text and Audio):
First, L2 Normalize the vectors (\|\mathbf{v}\| = \sqrt{x^2+y^2+z^2}):

• \|\mathbf{v}_{txt}\| = \sqrt{0.88^2 + 0.90^2 + 0.10^2} = 1.262 \rightarrow \hat{\mathbf{v}}_{txt} = [0.697, 0.713, 0.079]
• \|\mathbf{v}_{aud}\| = \sqrt{0.89^2 + 0.92^2 + 0.05^2} = 1.280 \rightarrow \hat{\mathbf{v}}_{aud} = [0.695, 0.718, 0.039]

Dot Product: (0.697 \times 0.695) + (0.713 \times 0.718) + (0.079 \times 0.039) = 0.484 + 0.511 + 0.003 = \mathbf{0.998}.
Conclusion: The system successfully retrieves the audio file using a text query with a 99.8\% match.

---

Example 2: The InfoNCE Matrix Calculation (Contrastive Loss)

Assume a training batch size of N=3. We have 3 Images (I_1, I_2, I_3) and 3 Texts (T_1, T_2, T_3). The model computes the dot products (logits) for all combinations. Let temperature \tau = 1.0 for simplicity.

\text{Logits} = \begin{bmatrix} I_1 \cdot T_1 (0.8) & I_1 \cdot T_2 (0.2) & I_1 \cdot T_3 (0.1) \\ I_2 \cdot T_1 (0.3) & I_2 \cdot T_2 (0.7) & I_2 \cdot T_3 (0.4) \\ I_3 \cdot T_1 (0.1) & I_3 \cdot T_2 (0.2) & I_3 \cdot T_3 (0.9) \end{bmatrix}

The loss function evaluates Row 1. The correct pair (I_1, T_1) scored 0.8. The negatives scored 0.2 and 0.1.

1. Exponentiate (Numerator): \exp(0.8) = 2.225
2. Sum of Exponentials (Denominator): \exp(0.8) + \exp(0.2) + \exp(0.1) = 2.225 + 1.221 + 1.105 = 4.551
3. Softmax Probability: 2.225 / 4.551 = 0.488 (48.8\%)
4. Negative Log Loss: \mathcal{L} = -\ln(0.488) = \mathbf{0.717}

Through backpropagation, the model updates its weights to push that 48.8\% closer to 100\%, driving the loss to 0.

---

Example 3: Vision Transformer (ViT) Patch Math & Parameter Count

How does an image become a sequence of vectors?

1. Input: A standard 224 \times 224 pixel RGB image.
2. Patching: The ViT slices it into 16 \times 16 pixel patches.
3. Total Patches (N): N = (H \times W) / P^2 = (224 \times 224) / 16^2 = 50176 / 256 = \mathbf{196 \text{ patches}}.
4. Flattening: Each patch has 16 \times 16 \times 3 \text{ (RGB channels)} = \mathbf{768 \text{ raw values}}.
5. Sequence Assembly: The 196 patches are linearly projected to the hidden dimension D (e.g., 768). A learned [CLS] token is prepended. The final input to the Transformer is a matrix of size \mathbf{197 \times 768}.

---

Example 4: Token Consumption Math for Video (gemini-embedding-2)

GCP enforces a strict 8,192 token limit. How much video can you embed?

• Video Frame: 66 tokens/frame.
• Audio: 25 tokens/sec.
• Timestamps: 10 tokens/sec.

If you process a video at 1 FPS with audio extraction enabled:
Consume Rate: 66 + 25 + 10 = \mathbf{101 \text{ tokens per second}}.
Maximum Duration: 8192 \text{ tokens} / 101 \text{ tokens/sec} \approx \mathbf{81.1 \text{ seconds}}.
Conclusion: Any video longer than 81 seconds will be silently truncated by the API.

---

Example 5: Visual Semantic Dilution (The Clutter Problem)

Assume a 3D space: [Apparel: Dress, Color: Red, Object: Dog].

• Query (Q): “Red dress” \rightarrow \mathbf{[0.707, 0.707, 0.0]} (L2 Normalized)
• Clean Image (I_{clean}): Studio photo of just a red dress. \rightarrow \mathbf{[0.707, 0.707, 0.0]}
• Cluttered Image (I_{clutter}): A woman in a red dress walking a dog in a park.
  • The ViT sees 50 patches of dress, 50 patches of dog. Mean pooling averages the features: [0.5, 0.5, 0.7].
  • Normalized: \mathbf{[0.51, 0.51, 0.71]}

Cosine Similarity:

• \text{Sim}(Q, I_{clean}) = (0.707 \times 0.707) + (0.707 \times 0.707) = \mathbf{1.0} (100% Match )
• \text{Sim}(Q, I_{clutter}) = (0.707 \times 0.51) + (0.707 \times 0.51) = \mathbf{0.72} (72% Match )
  Conclusion: Background pixels mathematically dilute the spatial attention.

---

Example 6: Temporal Attention Pooling (Video Math)

Let’s calculate the final embedding for a 5-second video. Latent space: [Action: Hitting, Object: Baseball, Environment: Crowd].

• Sec 1 (Pitcher winding up): \mathbf{h}_1 = [0.2, 0.8, 0.1], Weight \alpha_1 = 0.1
• Sec 2 (Pitch in air): \mathbf{h}_2 = [0.3, 0.9, 0.1], Weight \alpha_2 = 0.1
• Sec 3 (Bat hitting ball): \mathbf{h}_3 = [0.9, 0.9, 0.1], Weight \alpha_3 = 0.6 (Model recognizes peak action)
• Sec 4 (Ball flying): \mathbf{h}_4 = [0.4, 0.9, 0.1], Weight \alpha_4 = 0.1
• Sec 5 (Crowd cheering): \mathbf{h}_5 = [0.1, 0.1, 0.9], Weight \alpha_5 = 0.1

Calculating the Pooled Video Vector (\mathbf{v}_{video}):

\mathbf{v}_{video} = \sum_{t=1}^{5} \alpha_t \mathbf{h}_t

\mathbf{v}_{video} = [0.02, 0.08, 0.01] + [0.03, 0.09, 0.01] + [0.54, 0.54, 0.06] + [0.04, 0.09, 0.01] + [0.01, 0.01, 0.09]

\mathbf{v}_{video} = \mathbf{[0.64, 0.81, 0.18]}

The final vector is heavily weighted towards “Hitting” and “Baseball”, successfully suppressing the irrelevant “Crowd” frame.

---

Example 7: Audio Spectrogram Math (STFT & Mel-Scale)

How big is the “image” generated by a soundwave?

1. Input: A 2-second .wav file sampled at 44.1 \text{ kHz}. Total samples = 44,100 \times 2 = \mathbf{88,200 \text{ samples}}.
2. STFT Parameters: Window size N_{FFT} = 1024, Hop length H = 512.
3. Time Frames (X-axis): T = \lfloor \frac{\text{Total Samples} - N_{FFT}}{H} \rfloor + 1 = \lfloor \frac{88200 - 1024}{512} \rfloor + 1 = 170 + 1 = \mathbf{171 \text{ frames}}.
4. Frequency Bins (Y-axis): The STFT yields (N_{FFT} / 2) + 1 = 513 linear frequency bins. A Mel-filterbank compresses this to 128 Mel bins to mimic human hearing.
5. Result: The audio is now a 2D matrix of size \mathbf{128 \times 171}. This matrix is treated exactly like an image and fed into the Audio Spectrogram Transformer.

---

Example 8: Scalar Quantization (SQ) Memory Footprint & Error Rate

You are building an e-commerce search engine with 10 Million product images using gemini-embedding-2 (3072 dims).

• Uncompressed (FP32): 1 vector = 3072 dimensions \times 4 bytes = 12,288 bytes.
  • 10 Million vectors = 122.88 Gigabytes of RAM.
• Compressed (INT8 SQ): 1 vector = 3072 dimensions \times 1 byte = 3,072 bytes.
  • 10 Million vectors = 30.72 Gigabytes of RAM.

The Quantization Math:
Assume a specific dimension value is x_{fp32} = 0.35. The vector’s min value is -1.0 and max is 1.0.

x_{int8} = \text{round}\left( \frac{0.35 - (-1.0)}{1.0 - (-1.0)} \times 255 \right) = \text{round}\left( \frac{1.35}{2.0} \times 255 \right) = \text{round}(172.125) = \mathbf{172}

Dequantization (At Search Time):

x_{approx} = \left( \frac{172}{255} \right) \times 2.0 - 1.0 = 0.6745 \times 2.0 - 1.0 = 1.349 - 1.0 = \mathbf{0.349}

Error: |0.350 - 0.349| = \mathbf{0.001}. You saved ~92 GB of RAM for a negligible 0.001 coordinate error.

---

Example 9: Calculating 1D Earth Mover’s Distance (Wasserstein Drift)

Imagine your embedding tracks the visual concept of “Bezels on a Phone” on a 1D scale from 0 (Thick Bezels) to 10 (Bezel-less).

• Reference Data (P) from 2015: Distribution is centered at 2.
• Production Data (Q) from 2026: Distribution is centered at 9.

The Wasserstein distance calculates the minimum “work” required to move the mass of distribution P to Q.

W_1(P, Q) = \inf_{\gamma \in \Pi(P, Q)} \mathbb{E}_{(x, y) \sim \gamma} [\|x - y\|]

Work = Mass \times Distance. Assuming a normalized mass of 1, the distance is |9 - 2| = \mathbf{7}.
Because 7 exceeds your drift threshold of 2.0, an automated alert fires. You must re-embed your active catalog to capture the new visual reality of smartphones.

We can’t wait to see what you build. Share your creations and ask questions in the Google Cloud Community. Happy coding!

Let’s keep the conversation going! Share your thoughts, questions, and ideas in the comments.

Note: Should you have any concerns or queries about this post or my implementation, please feel free to connect with me on LinkedIn! Thanks!

[Aarav_chauhan]

nice ide