Applying Chain of Thought with Vertex AI Agent

[AnanthM]

I’ve created an agent using Vertex AI Agent Builder to answer queries about investment products. The agent uses a tool to retrieve data from an API, which includes information such as:

• Asset class
• One-year, three-year, and five-year returns
• Liquidity options
• Fees

The agent performs well with direct questions about specific products. However, it struggles with queries that require logic and calculations, such as “Which fund has the highest five-year return?”

To address this, I’ve tried the following:

1. Implemented a chain-of-thought prompt (https://arxiv.org/pdf/2201.11903) in the Instructions section to encourage the agent to break down the problem step-by-step. This approach was unsuccessful. As per my understanding, in theory and as per the paper, chain of thought reasoning should solve the problem. Want to understand why it doesn’t work.
   Here is the prompt I used

If the user asks a question that needs calculation to derive the answer, then break down the problem step by step and solve as per the below instructions. 

Data Extraction:

Identify all relevant funds and their associated values for the requested field.

Create a clear list or table of these funds and values.

Data Validation:

Ensure all values are of the same type 

Check for any missing or anomalous data points.

Sorting:

Arrange the values in ascending or descending order, depending on whether you're looking for the minimum or maximum.

Identification:

For maximum: Select the fund(s) corresponding to the highest value.

For minimum: Select the fund(s) corresponding to the lowest value.

Multiple Occurrences:

If multiple funds share the maximum or minimum value, list all of them.

Result Presentation:

State the extreme value (maximum or minimum).

Name the fund(s) associated with this value.

Provide context by mentioning how this value compares to the average or median, if relevant.

Verification:

Double-check the result by comparing it to the original data set.

Ensure no errors were made in the sorting or selection process.

Explanation:

Briefly explain the process used to arrive at the answer, highlighting key steps.

2. Provided a step-by-step example of how to solve a similar problem, which did improve the agent’s performance.

Questions:

1. Are there any best practices or recommended approaches for improving an agent’s ability to handle queries involving calculations and comparisons?
2. How can I effectively implement chain-of-thought reasoning in my agent without relying solely on specific examples?
3. Are there any built-in functions or tools in Vertex AI Agent Builder that can assist with mathematical operations or data comparison tasks?
4. Am I right in inferring that few-shot prompting with examples works but chain of thought doesn’t.

Any insights or suggestions would be greatly appreciated. Thank you!

[ruthseki]

Hi @AnanthM,

Welcome to Google Cloud Community!

Here’s my input to your questions:

1. Are there any best practices or recommended approaches for improving an agent’s ability to handle queries involving calculations and comparisons?

2. How can I effectively implement chain-of-thought reasoning in my agent without relying solely on specific examples?

Best Practices and Solutions:

Improve the Chain-of-Thought Prompt:

• Specific Guidance: Break down the prompt into concrete steps. Instead of “Identify all relevant funds,” guide the agent to identify relevant fields, then select specific data points.
• Concrete Examples: Provide more targeted examples of how to solve specific problems. This helps the agent understand the desired outcome.
• Logical Flow: Ensure your chain-of-thought steps are logically sequenced. Consider how the agent will process information and make decisions at each step.

You may also refer to leverage chain of thought prompting documentation or by following what’s stated in the image below:

To learn more about effective prompt engineering and usage of Vertex AI Agent, check out the Five Best Practices for Prompt Engineering, Prompting Best Practices and Vertex AI Agents Best Practices.

3. Are there any built-in functions or tools in Vertex AI Agent Builder that can assist with mathematical operations or data comparison tasks?

The supported built-in tool is: Code Interpreter : A Google first-party tool that combines the capability of code generation and code execution and allows the user to perform various tasks, including: data analysis, data visualization, text processing, solving equations or optimization problems.

4. Am I right in inferring that a few-shot prompting with examples works but chain of thought doesn’t?

You are correct that few-shot prompting with examples often works better than pure chain-of-thought prompting in the early stages. This is because examples provide a concrete model for the agent to follow. However, chain-of-thought is still a valuable technique for promoting generalizable reasoning. The key is to use a well-structured and targeted prompt.

I hope the above information is helpful.