Attaching metadata to chunks in Rag Engine

Google Cloud Build with AI Custom ML & MLOps
1

I have a .jsonl file stored in Google Cloud Storage, and I want to ensure that metadata values are properly attached to every chunk during retrieval. Let’s say my JSONL object looks like this:

{
"id": "string",
"content": "string",
"metadata": {
"title": "string",
"author": "string",
"parentId": "string"
}
}

Here are two sample retrieved_context objects. You’ll notice a difference in structure between them:

{
"retrieved_context": {
"uri": "gs://hello.jsonl",
"title": "hello.jsonl",
"text": "content It was a calm night and I was taken to a hidden place where wise elders awaited. They shared the legacy of a spiritual leader who lived in devotion for over a hundred years.\nmetadata title Mystic Tale\nmetadata author John\nmetadata parentId xxxx-xxxx-xxxx-xxxx",
"rag_chunk": {
"text": "content It was a calm night and I was taken to a hidden place where wise elders awaited. They shared the legacy of a spiritual leader who lived in devotion for over a hundred years.\nmetadata title Mystic Tale\nmetadata author John\nmetadata parentId xxxx-xxxx-xxxx-xxxx"
}
}
},

{
"retrieved_context": {
"uri": "gs://hello.jsonl",
"title": "hello.jsonl",
"text": "id xxxx-xxxx-xxxx-xxxx\ncontent Once upon a time, in a forest of glowing trees, a young fox discovered a stone that whispered secrets. The animals gathered as the wise owl interpreted its message",
"rag_chunk": {
"text": "id xxxx-xxxx-xxxx-xxxx\ncontent Once upon a time, in a forest of glowing trees, a young fox discovered a stone that whispered secrets. The animals gathered as the wise owl interpreted its message"
}
}
}

My Questions are

Why does the text structure vary across chunks?
Some chunks include metadata inside the text field, while others do not.

How can I ensure every chunk includes metadata values, so I can always trace it back to the original document?

Are metadata values actually embedded as plain text inside the string?
If yes, does that mean I’ll need to manually parse and extract them from each chunk?

Is there a more structured or reliable way to attach metadata to every chunk, possibly outside the text body?

As a solo dev, can I purchase standard support? Or is an organisation necessary for that?

2

Hey guys, I have similiar issue here and want to understand if anyone can provide help!

I tried structData and jsonData as recommended in Prepare data for ingesting  |  AI Applications  |  Google Cloud , either content field that point to a gcs file, or content directly inside the jsonl file.

Also followed the PropertyNames requirements thanks to Vertex AI Search Agent Builder: Indexing Failure After Successful Imports , make sure property name correct

Tried with new datastore again many times, and just could not make the metadata work as expected :joy:

I always found the flattened metadata inside content and no customized metadata returned to a RAG retrieval response :smiling_face_with_tear: , which I treated as a signal as ingestion failure(am I making the wrong assumption and flattened in metadata is actually correct?)

Any directions? should I try some low level APIs instead of Ingest in GCP Console?

Any advice will be appreciated, thanks!