Vertex AI RAG Engine: Metadata Filtering not working

[Jeslin_Joseph]

I’m struggling to get metadata filtering working correctly using the Vertex AI RAG engine (google-cloud-aiplatform). My goal is to restrict retrieval results based on a category field assigned to different files within a single RAG corpus.

My Setup:

1. Storage: Files are stored in a GCS bucket folder.

2. Metadata File: I have a metadata.jsonl in the same folder.

   • Example entry: {"file_gcs_uri": "gs://my-bucket/folder/doc1.pdf", "metadata": {"category": "HR"}}

3. Import: I imported the entire folder into the RAG Corpus using the import_files API.

The Problem:

Despite the filter, the retrieval returns results from all categories (ignoring the filter).

Is there a specific schema requirement for the metadata.jsonl? Any guidance on this would be greatly appreciated!

[shuhaib_arkn]

May be the JSON Structure is Not suitable try with simple JSON,
{“file_path”: “gs://my-bucket/hr/policy1.pdf”, “category”: “HR”}

[lk213]

Vertex AI Rag engine file metadata and metadata filtering

I’m trying to import files into rag engine from gcs. I can see I can see in the docs for rag engine import that I can set rag_file_metadata_config. I want to use inline_metadata_schema_source and inline_metadata_source, but I can’t find any examples or docs for what they should look like. Similarly, I can’t find any docs on what metadata.json should look like if I wanted to use a metadata file in gcs instead, though I’d prefer just using the inline options. And for context retrieval filtering, …

I’m also trying to work this out. I tried using the same metadata.json and filter format that worked for vertex ai search, but no luck. The docs I linked in my post seem to indicate it is possible, so I made a support ticket for it with google. They told me the functionality exists in the beta api and works, but isn’t technically released yet, so they refused to tell me how to use it.

What syntax did you try for the filter?

[Edson_Cepeda]

@Jeslin_Joseph hey I was wondering if you could get metadata filtering on rag engine working, and if so how did you do it?