RAG Agent in ADK - EVAL

Google Cloud Build with AI Agents
1

Hello Everyone,

I hope you’re all doing well. I was working on building the RAG agent using the sample provided in the ADK sample. While trying to execute it, I ran into an issue. Specifically, the EVAL FAILED with the error message: eval/test_eval.py::test_eval_full_conversation - ValueError: Input path /home/l/adk-samples/python/agents/RAG/eval/data/conversation.test.json is invalid.

Has anyone encountered a similar issue and, if so, what was the resolution? Upon further research, it seems like the test file needs to be converted to a different format. Any guidance or suggestions you could offer would be greatly appreciated.

Thanks a lot,
Lourdes

2

Hi Lourdes,

Thanks for reaching out with this detailed question. It’s great to see you’re using the evaluation capabilities of the Agent Development Kit (ADK).

The error ValueError: Input path ... is invalid occurs because recent versions of the ADK have adopted a more structured, Pydantic-backed schema for evaluation files. Your conversation.test.json file is likely in an older format, which the evaluator can no longer parse directly.

Fortunately, the ADK provides a straightforward utility to handle this exact scenario. You can use the AgentEvaluator.migrate_eval_data_to_new_schema method to convert your test file to the new, required format.

Here is a simple Python script you can use to perform the migration.

from google.adk.evaluation.agent_evaluator import AgentEvaluator
import os

# Configuration 
# Set the path to your original, old-format evaluation file.
old_file_path = "/home/l/adk-samples/python/agents/RAG/eval/data/conversation.test.json"

# Define the path for the newly converted file.
directory = os.path.dirname(old_file_path)
new_file_name = "conversation.migrated.test.json"
new_file_path = os.path.join(directory, new_file_name)

# Main 
def migrate_evaluation_file():
    """
    Uses the ADK's built-in utility to convert an evaluation file
    to the new schema and saves it to a new location.
    """
    if not os.path.exists(old_file_path):
        print(f"Error: The input file was not found at '{old_file_path}'")
        return

    print(f"Migrating '{os.path.basename(old_file_path)}' to the new EvalSet schema...")

    try:
        AgentEvaluator.migrate_eval_data_to_new_schema(
            old_eval_data_file=old_file_path,
            new_eval_data_file=new_file_path,
        )
        print(f"Migration successful!")
        print(f"   New file saved at: '{new_file_path}'")
        print("\nNext step: Update your test_eval.py to use this new file.")

    except Exception as e:
        print(f"An error occurred during migration: {e}")


if __name__ == "__main__":
    migrate_evaluation_file()

Be sure you create a new Python file named migration_script.py in the root of your adk-samples project. Then, open your terminal, navigate to the project directory, and run the script shell python migration_script.py.

The script will generate a new file named conversation.migrated.test.json in the same directory as your original file.

Finally, open eval/test_eval.py, update the file path to point to this new, migrated file and run your pytest command. This time should work successfully.

Let us know if this resolves the issue for you.

3

Thank you ! I tried this before it wasn’t recognizing the agent_evaluation and in one of the forum they mentioned to use python 3.9.1 to resolve the error. In my case it didn’t work, will try it again and keep you posted.

4

Hi,

Thank you for the script !

The script you provided worked but I am getting below error. In the pip list adk is available do I need to upgrade the python version ?

../../../../.cache/pypoetry/virtualenvs/rag-zv9BR4zn-py3.12/lib/python3.12/site-packages/google/adk/evaluation/agent_evaluator.py:396: ModuleNotFoundErro

============================================================================================================ short test summary info =============================================================================================================
FAILED eval/test_eval.py::test_eval_full_conversation - ModuleNotFoundError: Eval module is not installed, please install via pip install google-adk[eval].
========================================================================================================= 1 failed in 120.62s (0:02:00) ==========================================================================================================

Thank you for the help !

Lourde

5

Hi @Lourdes_Manickam ,

that’s a great step ahead. Please install google-adk[eval] as suggested.

Best

6

I am sorry for responding late, as I mentioned in the earlier message the ADK is already installed. When I execute the command pip list it shows ADK was already installed. I tried uninstalling and reinstalling the package it didn’t help.

Thanks

Lourde