Vertex AI Mistral OCR — document_annotation_format always returns 500

Google Cloud Build with AI Custom ML & MLOps
,
1

Component: Vertex AI → Model Garden / partner models (MaaS)

Regions tested: us-central1, europe-west4 — identical behavior, reproduced
consistently across multiple days (first observed 2026-07-14, re-verified 2026-07-17).
Project ID available on request.

Summary

publishers/mistralai/models/mistral-ocr-2505:rawPredict fails with
500 {"message":"Internal server error"} for any request that includes
document_annotation_format, including a minimal single-field schema. The identical
request body works against Mistral’s own API (api.mistral.ai/v1/ocr), so the wire
format is correct. Plain OCR (no annotations) works fine on Vertex.

Reproduction

Base request:

POST https://us-central1-aiplatform.googleapis.com/v1/projects/PROJECT/locations/us-central1/publishers/mistralai/models/mistral-ocr-2505:rawPredict
Authorization: Bearer <ADC token>
Content-Type: application/json

{
  "model": "mistral-ocr-2505",
  "document": {
    "type": "document_url",
    "document_url": "data:application/pdf;base64,<any small PDF>"
  }
}

Minimal annotation schema used below:

{
  "type": "object",
  "properties": { "summary": { "type": "string" } },
  "required": ["summary"],
  "additionalProperties": false
}
# Variation on the base request Result
1 No annotation fields (bare OCR) — control 200 OK, correct markdown per page
2 document_annotation_format: { type: "json_schema", json_schema: { name: "summary", strict: true, schema: <minimal schema> } } 500 {"message":"Internal server error"} — every run, any schema, strict true or false
3 document_annotation_format: { type: "text" } 400 from the Mistral server: "Please provide a json_schema for document_annotation_format"
4 Adding document_annotation_prompt: "Summarize the document." (a documented Mistral OCR API field) 422 — proxy validation falls through to ChatCompletionRequest: {"detail":[{"type":"missing","loc":["body","ChatCompletionRequest","messages"],...}]}
5 bbox_annotation_format with a valid schema, text-only PDF 200 (no-op — no images to annotate)
6 bbox_annotation_format with a valid schema, PDF containing an image 404 {"detail":"Not Found"}

Why this is not a malformed request

  • Case 3 shows the request reaches the Mistral model server (Mistral error envelope
    {"object":"Error",...}), which knows the document_annotation_format field and
    validates it — and asks for exactly the json_schema shape we send in case 2.
  • Malformed schema packagings (bare schema without the name/strict wrapper, schema as
    a JSON string, schema_definition instead of schema) are all rejected with 4xx
    validation errors. Only the correct shape gets past validation — and then fails
    with 500 during execution.
  • The identical case-2 body succeeds against api.mistral.ai/v1/ocr (verified
    repeatedly; annotations returned correctly).

Analysis

OCR itself works; every code path that requires the annotation LLM component fails —
500 for document annotations, 404 for bbox annotations when there is actually an image
to annotate — while malformed requests are correctly rejected with 4xx. Case 4
additionally shows the Vertex proxy’s request schema does not know
document_annotation_prompt at all. Together this suggests the annotation-executing
component (the LLM step that Mistral’s Document AI runs after OCR) is missing from the
Vertex deployment.

Expected behavior

document_annotation_format and bbox_annotation_format work as documented for
Mistral OCR (this is core Document AI functionality of the model, and the model card
advertises document understanding).

Impact

Structured extraction on Vertex-hosted Mistral OCR is unusable. This specifically
blocks HIPAA-scoped workloads that chose Vertex MaaS for Google Cloud BAA coverage —
the direct Mistral API is not an option for them precisely because it lacks a BAA.

1 Like