From PDF Scrolling to Instant Answers: Building a "Glass Box" Clinical Assistant

[Vijay_Makkakode]

A Deep Dive into Seattle Children’s Pathway Assistant Architecture

Problem

At Seattle Children’s, our Clinical Standard Work (CSW) pathways are the gold standard for patient care, but they are massive—comprising over 2,000 pages of complex logic trees, evidence-based protocols, and intricate visual flowcharts. Understanding these documents is inherently challenging due to their complex, varied layouts. To translate these static PDFs into a machine-readable format that the AI agent can accurately query, the system required a highly accurate multi-step extraction and indexing pipeline.

For a busy clinician at the point of care, extracting actionable guidance from these static PDFs during acute care scenarios, where minutes can alter outcomes, is highly inefficient.

What We Built

We built Pathway Assistant to translate extensive CSW into a machine-readable format that the AI agent can accurately query. Instead of a simple search bar, we created an interactive chatbot that acts like a clinical decision support system. It engages in a dialogue, asks clarifying questions to “ground” itself in the provided pathway, and generates detailed, step-by-step reasoning for its recommendations.

The core intelligence underwent three major iterations to eliminate hallucination errors:

• Version 1 utilized a static RAG design with Gemini 1.5 Pro.

• Version 2 introduced basic agentic chatting with Gemini 2.0 Flash.

• Version 3 achieved strict deterministic curation using Gemini 2.5 Pro.

The Stack & Solution

We architected a fully managed, serverless stack to ensure security, scalability, and ease of deployment.

• Gemini Enterprise Agent Platform: The brain of the operation. We utilized Gemini 2.5 Pro (Thinking) models to process multimodal inputs (text + visual flowcharts). This allowed the model to “read” our complex logic trees just like a human would.

  • We explicitly prompt Gemini to perform text extraction while preserving the original layout and chunking the text in a single operation, effectively bypassing earlier unsuccessful tests with non-GenAI tools like Google Cloud Document AI. Additionally, Gemini’s Visual Q&A scans cover pages to extract administrative metadata (like versioning and expected revision dates) into structured JSON.

• Cloud Run: We containerized the application logic via Artifact Registry here. It provides a serverless, auto-scaling backend that hosts our custom Python code, handling traffic spikes without manual server management.

• Streamlit: The frontend interface. This Python library allowed us to build a chat interface where clinicians can type prompts (e.g., “9-year-old with DKA…”) and view the “thinking” process alongside the final answer.

• Cloud Storage (GCS): Our secure Knowledge Base. We store the raw PDF content and parsed pages here, which the app retrieves (RAG) to ground every answer in our official hospital protocols.

• BigQuery: The analytics engine. We log every interaction and user feedback (conversation score/feedback text) here to continuously monitor performance and refine our prompts.

• Security (IAP & OAuth 2.0): We wrapped the application in Cloud Load Balancing with Identity-Aware Proxy (IAP). This delegates authentication to Google Workspace, ensuring only verified staff can access the tool.

---

Under the Hood

The system doesn’t just “guess”; it follows a rigorous logic path defined by our curation process.

import vertexai
from vertexai.preview.generative_models import GenerativeModel

# Initialize Vertex AI
vertexai.init(project="seattle-childrens-poc", location="us-west1")

# The "Thinking" Model configuration (V3)
# We use Gemini's reasoning capabilities to parse complex flowcharts
model = GenerativeModel("gemini-2.5-pro-preview")

# Interactive Chat Session
# The model is instructed to ask clarifying questions before answering
chat = model.start_chat()
response = chat.send_message(
    [clinician_prompt, pathway_pdf_context],
    generation_config={
        "temperature": 0,          # Strict determinism for medical safety
        "max_output_tokens": 10000 # Extended token limit for "thinking" process
    }
)

Note: This runs inside our secure Cloud Run container, impersonating a service account to access data.

The Technical Wins

• Zero Error Rate: In our latest version (V3), after implementing metadata curation and “thinking” models, we achieved a 0% error rate on our test set, correcting all previous logic errors.

• Rapid Reasoning: The model can synthesize complex inputs (pH, Bicarb, weight, history) and generate a complete management plan (fluids, insulin, labs) in roughly 20-100 seconds. Doing this manually would take significantly longer.

• “Glass Box” Trust: Unlike “black box” AI, our system displays its step-by-step reasoning (e.g., “Patient is <2 months old, therefore excluding Valproic Acid…”). This transparency allows clinicians to verify the logic instantly.

What We Learned

Here are two top takeaways we’d share with any team looking to build similar clinical tools:

1. Curation is the New Coding

• The Old Way: We initially thought we could just “dump” PDFs into the model. This led to a 16.8% error rate because the model struggled with vague flowcharts.

• Our Golden Path: We had to curate the source material. We optimized our flowcharts (e.g., labeling decision branches clearly) and audited 13k lines of JSON metadata. Because the metadata reflects the model’s interpretation, we had to eliminate errors by writing decision points explicitly as questions, defining numerical ranges unambiguously (e.g., changing “2-4” to “2 to 4”), and removing continuous “loops” within branches. First we manually crafted the source material and then with the support of Gemini optimized for consistency, accuracy, and efficiency. Curation of the source material was the key to unlocking zero errors.

2. The “Consultant” Workflow

• The Old Way: A simple “Search” bar. The model would guess answers based on incomplete info (e.g., assuming a patient was in the ICU).

• Our Golden Path: We engineered the bot to behave like a consultant. It now asks clarifying questions (“Is the patient in the ED or Inpatient?”) before giving an answer. This workflow was refined in version 2 by tuning system instructions to mimic a supervising physician and trainee, explicitly instructing the AI to ask about age, setting, and medications, which corrected 204 of 226 previously incorrect answers. This “grounding” ensures the AI is looking at the exact right page of the protocol. Not only that, but Pathway Assistant always allows the user to get feedback in real time from the chatbot about the quality and veracity of the information shared by the model. This feedback is stored in BigQuery and is regularly updated and analyzed to detect remaining possible sources of confusion and fix them.

Want to build something similar?

Explore Gemini Enterprise Agent Platform

Check out the Cloud Run quickstarts

Question: Anyone else using “Thinking” models to replace manual document workflows? Curious how others are handling the “human-in-the-loop” verification for high-stakes decisions.