Vertex AI Search not returning relevant answers and search results for structured JSONL ticket data

Google Cloud Build with AI Generative AI & Foundational Models
1

Goal

We developed a ticketing system where our customers can submit tickets which are then handled by our support engineers.

We want to assist our support engineers by using AI to generate draft replies to customer tickets.

Given a current ticket in the prompt (including summary, all posts and internal assignments), the system should:

  1. Retrieve similar past tickets
  2. Generate a suggested response based on how those tickets were resolved

The generated response will be reviewed and optionally sent to the customer by the support engineer.

Data

  • 9,302 tickets exported as JSONL (1 ticket per file)
  • Each ticket contains:
    • Metadata (TicketId, Summary, ProductLine, etc.)
    • A list of posts (customer + internal)
    • A list of internal assignments, each with posts

Example of the JSONL file (formatted to multiple lines for better readability):

{
  "TicketId": 123,
  "TicketNumber": "ST000123",
  "Summary": "Hardware breakpoint doesn't work",
  "Description": "Whenever I try to place a hardware breakpoint, the software crashes. I am using version 5.2.3. What is wrong?",
  "CreatedAt": "2022-01-11T14:53:10.592528Z",
  "ReporterEmail": "john.doe@customer.com",
  "ProductLine": "Debuggers",
  "Posts": [
    {
      "EventId": 43,
      "EventType": "Ticket created",
      "Description": "Whenever I try to place a hardware breakpoint, the software crashes. I am using version 5.2.3. What is wrong?",
      "CreatedAt": "2022-01-11T14:53:10.592528Z",
      "CreatorEmail": "john.doe@customer.com",
      "CreatorRole": "Customer"
    },
    {
      "EventId": 112,
      "EventType": "Ticket post submitted",
      "Description": "Have you tried using the JTAG debug probe?",
      "CreatedAt": "2022-01-20T09:42:52.185681Z",
      "CreatorEmail": "jane.smith@company.org",
      "CreatorRole": "Internal"
    },
    {
      "EventId": 183,
      "EventType": "Ticket post submitted",
      "Description": "Yes, but it doesn't work.",
      "CreatedAt": "2022-01-26T08:37:35.155728Z",
      "CreatorEmail": "john.doe@customer.com",
      "CreatorRole": "Customer"
    },
    {
      "EventId": 211,
      "EventType": "Ticket post submitted",
      "Description": "Please update the version of our software to v6. Then, it should work.",
      "CreatedAt": "2022-01-27T09:28:31.126449Z",
      "CreatorEmail": "jane.smith@company.org",
      "CreatorRole": "Internal"
    },
    {
      "EventId": 350,
      "EventType": "Ticket closed",
      "Description": "Thank you, updating the software to v6.0.6 solved my problem.",
      "CreatedAt": "2022-02-10T09:27:48.507694Z",
      "CreatorEmail": "john.doe@customer.com",
      "CreatorRole": "Customer"
    },
  ],
  "Assignments": [
    {
      "AssignmentId": 321,
      "AssignmentNumber": "T0000321",
      "Summary": "Placing hardware breakpoints throws OOM",
      "Description": "The customer is trying to place hardware breakpoints, but the OOM is thrown and the software crashes.",
      "CreatedAt": "2022-01-26T08:37:35.155728Z",
      "ReporterEmail": "jane.smith@company.org",
      "Posts": [
        {
          "EventId": 398888,
          "EventType": "Task created",
          "Description": "The customer is trying to place hardware breakpoints, but the OOM is thrown and the software crashes.",
          "CreatedAt": "2022-01-26T08:37:35.155728Z",
          "CreatorEmail": "jane.smith@company.org",
          "CreatorRole": "Internal"
        },
        {
          "EventId": 398890,
          "EventType": "Task proposed as resolved",
          "Description": "They need to update to at least v6.",
          "CreatedAt": "2022-01-26T09:22:01.283765Z",
          "CreatorEmail": "sam.taylor@company.org",
          "CreatorRole": "Internal"
        },
        {
          "EventId": 405532,
          "EventType": "Task closed",
          "Description": "",
          "CreatedAt": "2022-01-26T14:15:42.630695Z",
          "CreatorEmail": "jane.smith@company.org",
          "CreatorRole": "Internal"
        }
      ]
    }
  ]
}

What we did

  1. Uploaded all 9302 JSONL files to a GCS bucket.
  2. Created a Vertex AI Search structured data store (schema auto-detected). All documents have been successfully indexed.

image
image1587Ă—936 67.6 KB

  1. We created a new application of type “Custom search (general)” and connected it to the previously created structured data store.
  2. We selected “Search with an answer” as the “Search type”. We left all other settings untouched.

Problem

Search results and answers are not relevant at all.

Example:

  • Query: “Summarize ticket ST000123”
  • Result:
    • Generated answer is completely unrelated to the ticket
    • Search results don’t even contain a document where TicketId=123

Questions

  1. What could cause such poor retrieval and answer quality in this setup?
  2. Is Vertex AI Search the right tool for this use case (semantic retrieval over structured ticket history)?
1 Like