Imagine a world not very far from now where the primary entry point to businesses is entirely via chat interfaces.
We are entering the era of CUI (Conversational User Interface)—a paradigm shift where natural language, ambient voice notes, and intelligent agentic workflows replace traditional GUI (Graphical User Interface) apps, web forms, and manual catalog navigation. In a CUI, user intent is expressed naturally, while specialized AI agents synthesize user context, authenticate accounts, and execute real-world transactions behind the scenes.
Today, primitive chatbots exist across WhatsApp, RCS, Apple Messages, and Telegram. But they are severely limited—following rigid decision trees, forgetting context, and lacking the security needed to handle real transactions.
What if you could interact with an e-commerce platform—let’s call our prototype HappyDelivery—completely friction-free on your favorite messaging app?
The HappyDelivery Vision
User: “Hey, we are out of milk and eggs. Also, add some snacks for Friday movie night.”
HappyDelivery Agent: "Got it! I’ve added 1 gallon of your usual Whole Milk (4.29)and1dozenFree−RangeEggs(4.29)and1dozenFree−RangeEggs(3.99). Based on your past movie nights, should I add Doritos (4.50)anda6−packofSprite(4.50)anda6−packofSprite(5.99)?"
User: “Swap Sprite for Coke Zero. That’s it.”
HappyDelivery Agent: “Done! Your total is $18.77. I’ll charge your Visa ending in 4242 and deliver Thursday between 5-7 PM to 742 Evergreen Terrace. Sound good?”
User: “Yes, perfect!”
HappyDelivery Agent: "Order confirmed!Order #HD-9821 is placed. I’ll text you when your driver is on the way."
No app downloads. No infinite catalog scrolling. No login screens or checkout forms. Just conversational, frictionless commerce powered by a Multi-Agent Swarm on Google Cloud Platform (GCP).
This document is the definitive engineering blueprint to build this exact system today.
Part 1: High-Level System Design & Harmony
To achieve this experience without hallucinations, high latency, or security flaws, we separate Ingress, State, and Cognition.
Part 2: Step-by-Step Infrastructure & Code Blueprint
Phase 1: Ingress Layer (Messaging APIs & Audio Pipeline)
1. Developer Portal Configuration
- WABA / Meta Developer Portal: Register your business phone number, generate a permanent system user token, and set up Webhooks targeting your ingress URL (
https://api.happydelivery.com/v1/webhooks). - Security Verification: Configure a secret verification token (
WEBHOOK_VERIFY_TOKEN). - Subscriptions: Subscribe to
messages,messaging_postbacks, anduser_pre_approval.
2. High-Throughput Webhook Microservice (FastAPI on GKE)
Because messaging gateways drop connections if an HTTP response takes >3 seconds, never process LLM calls synchronously inside the webhook endpoint.
Code: ingress_service.py
import hmac
import hashlib
import os
from fastapi import FastAPI, Request, HTTPException, Response, BackgroundTasks
from google.cloud import pubsub_v1
app = FastAPI()
PUBLISHER = pubsub_v1.PublisherClient()
TOPIC_PATH = PUBLISHER.topic_path("happydelivery-gcp-project", "inbound-messages")
def get_app_secret() -> bytes:
secret = os.getenv("META_APP_SECRET", "")
return secret.encode("utf-8")
def verify_signature(payload: bytes, signature: str):
expected_hash = hmac.new(get_app_secret(), payload, hashlib.sha256).hexdigest()
if not hmac.compare_digest(f"sha256={expected_hash}", signature):
raise HTTPException(status_code=403, detail="Invalid signature")
@app.get("/v1/webhooks")
async def verify_webhook(request: Request):
params = request.query_params
if params.get("hub.verify_token") == os.getenv("WEBHOOK_VERIFY_TOKEN"):
return Response(content=params.get("hub.challenge"), media_type="text/plain")
raise HTTPException(status_code=400, detail="Verification failed")
@app.post("/v1/webhooks")
async def handle_webhook(request: Request, background_tasks: BackgroundTasks):
signature = request.headers.get("X-Hub-Signature-256", "")
body = await request.body()
verify_signature(body, signature)
payload = await request.json()
# Asynchronously push to Pub/Sub to respond to WhatsApp/messaging in <200ms
background_tasks.add_task(PUBLIS
3. Voice Note Audio Processing Pipeline
If the user sends an audio note (“Hey HappyDelivery, get me some apples”), the Pub/Sub worker processes the audio media ID:
from google.cloud import speech
def transcribe_audio(audio_bytes: bytes) -> str:
client = speech.SpeechClient()
audio = speech.RecognitionAudio(content=audio_bytes)
config = speech.RecognitionConfig(
encoding=speech.RecognitionConfig.AudioEncoding.OGG_OPUS,
sample_rate_hertz=16000,
language_code="en-US",
)
response = client.recognize(config=config, audio=audio)
return " ".join([result.alternatives[0].transcript for result in response.results])
Phase 2: Session Memory Layer (GKE + Cloud Memorystore Redis)
Because chat messages arrive as isolated events, Redis maintains the live user state.
State Data Structure (session:{phone_number})
{
"user_phone": "+15550192834",
"customer_id": "cust_98213",
"auth_token": "bearer_sec_token_abc123",
"cart": [
{"item_id": "prod_milk_01", "name": "Whole Milk 1 Gal", "qty": 1, "price": 4.29},
{"item_id": "prod_eggs_12", "name": "Free-Range Eggs 12ct", "qty": 1, "price": 3.99}
],
"dietary_preferences": ["dairy-full", "gluten-free-preferred"],
"chat_history": [
{"role": "user", "content": "Hey, we are out of milk and eggs."},
{"role": "assistant", "content": "Got it! Added Whole Milk and Eggs to your cart."}
]
}
Phase 3: The LangGraph Multi-Agent Swarm (Core Logic)
We use LangGraph with Vertex AI Gemini 1.5 Pro/Flash to build a deterministic agent graph.
[ Inbound Payload ]
│
▼
┌─────────────────┐
│ Supervisor Node │
└────────┬────────┘
│
┌─────────────────┼─────────────────┐
▼ ▼ ▼
┌────────────────┐ ┌────────────────┐ ┌───────────────────┐
│ Profile Agent │ │Inventory Agent │ │ Transaction Agent │
└────────┬───────┘ └────────┬───────┘ └─────────┬─────────┘
│ │ │
└─────────────────┼───────────────────┘
▼
[ State Update & Redis ]
│
▼
[ WhatsApp Response ]
1. Defining the State in Python
from typing import TypedDict, Annotated, List, Dict, Any
from langchain_core.messages import BaseMessage
import operator
class AgentState(TypedDict):
messages: Annotated[List[BaseMessage], operator.add]
phone_number: str
customer_id: str
cart: List[Dict[str, Any]]
next_step: str
2. Developing the Tools
tools.py
from langchain_core.tools import tool
@tool
def search_catalog(query: str, category: str = None) -> list:
"""Searches the HappyDelivery inventory database for matching products."""
# Simulated connection to ElasticSearch / Postgres ERP
return [
{"item_id": "prod_milk_01", "name": "Whole Milk 1 Gal", "price": 4.29, "in_stock": True},
{"item_id": "prod_eggs_12", "name": "Free-Range Eggs 12ct", "price": 3.99, "in_stock": True},
{"item_id": "prod_doritos_09", "name": "Doritos Nacho Cheese 9oz", "price": 4.50, "in_stock": True},
{"item_id": "prod_coke_zero_6p", "name": "Coke Zero 6-pack", "price": 5.99, "in_stock": True}
]
@tool
def execute_stripe_charge(customer_id: str, amount_cents: int, cart_summary: str) -> dict:
"""Executes payment on Stripe using the tokenized card on file."""
# Strict API call - No LLM generation allowed here
return {"status": "success", "charge_id": "ch_3M498234", "order_id": "HD-9821"}
3. Agent Node Definitions & Prompt Engineering
Node Implementations (nodes.py)
from langchain_google_vertexai import ChatVertexAI
from langchain_core.prompts import ChatPromptTemplate
from pydantic import BaseModel, Field
class RouteResponse(BaseModel):
next_node: str = Field(description="One of: 'inventory', 'profile', 'transaction', 'END'")
llm_flash = ChatVertexAI(model_name="gemini-1.5-flash", temperature=0)
llm_pro = ChatVertexAI(model_name="gemini-1.5-pro", temperature=0.2)
supervisor_prompt = ChatPromptTemplate.from_messages([
("system", """You are the Supervisor of the HappyDelivery Agent Swarm.
Analyze the user conversation and route to the appropriate specialist:
- Route to 'profile' if we need to load dietary preferences or check authentication.
- Route to 'inventory' if the user wants to search, add, or swap products in their cart.
- Route to 'transaction' if the user explicitly confirms payment and checkout.
- Route to 'END' if the turn is complete."""),
("messages", "{messages}")
])
supervisor_chain = supervisor_prompt | llm_flash.with_structured_output(RouteResponse)
inventory_agent_with_tools = llm_pro.bind_tools([search_catalog])
transaction_agent_with_tools = llm_pro.bind_tools([execute_stripe_charge])
def supervisor_node(state: AgentState):
result = supervisor_chain.invoke({"messages": state["messages"]})
return {"next_step": result.next_node}
def profile_node(state: AgentState):
# Hydrates user profile & dietary tags into state
return {"next_step": "supervisor"}
def inventory_node(state: AgentState):
messages = state["messages"]
response = inventory_agent_with_tools.invoke(messages)
return {"messages": [response]}
def transaction_node(state: AgentState):
messages = state["messages"]
response = transaction_agent_with_tools.invoke(messages)
return {"messages": [response]}
4. Assembling the LangGraph Workflow
from langgraph.graph import StateGraph, END
workflow = StateGraph(AgentState)
# Add Nodes
workflow.add_node("supervisor", supervisor_node)
workflow.add_node("profile", profile_node)
workflow.add_node("inventory", inventory_node)
workflow.add_node("transaction", transaction_node)
# Set Entry Point
workflow.set_entry_point("supervisor")
# Conditional Routing Edges
workflow.add_conditional_edges(
"supervisor",
lambda state: state["next_step"],
{
"profile": "profile",
"inventory": "inventory",
"transaction": "transaction",
"END": END
}
)
workflow.add_edge("profile", "supervisor")
workflow.add_edge("inventory", "supervisor")
workflow.add_edge("transaction", END)
app_swarm = workflow.compile()
Phase 4: Secure Account Linking & Authentication
To execute orders securely via WhatsApp/RCS without storing raw credit card details in the chat history:
[Unlinked User Texts Agent] ──> [Profile Agent Detects No Token]
│
▼
[Sends WhatsApp Interactive Button: "Link Account"]
│
▼
[User Clicks ──> Opens Encrypted OAuth Webview (HTTPS)]
│
▼
[Logs in via HappyDelivery Web Portal / Passkey]
│
▼
[Stripe Customer Token (cust_9821) Linked to Phone (+1555...)]
- Initial Handshake: The phone number is cryptographically signed by the WhatsApp API header (
X-Hub-Signature-256). - Tokenized Payments: Payment tokens (
cus_123456) are stored in Cloud Spanner / Secret Manager and tied to the verified phone number. The agent only sees the last 4 digits (e.g., “Visa ending in 4242”). - High-Value Step-Up (OTP): For purchases exceeding a configurable threshold (e.g., >$100), the Transaction Agent pauses execution and triggers an in-chat OTP verification push.
Part 3: Solving Enterprise Real-World Challenges
1. The Latency Problem (“The Swarm Tax”)
Running multiple LLM hops can take 4-8 seconds.
-
Mitigation Strategy (Asynchronous Progress Push): When the GKE Pub/Sub worker realizes the query will take >2 seconds, it sends an immediate low-latency template message via WhatsApp API:
“Looking through our fresh aisles for you…
” Once the LangGraph swarm completes its graph traversal 2 seconds later, it sends the full visual response.
2. Eliminating Financial Hallucinations
An LLM must never compute totals using internal arithmetic or fabricate discounts.
- Mitigation Strategy: All price totals are calculated strictly inside the deterministic backend server during tool execution. The
execute_stripe_chargetool takes thecart_id, calculates tax/shipping via the ERP, and charges the exact amount directly. The LLM only receives and formats the string response returned by the backend tool.
Part 4: Line-by-Line Execution of the HappyDelivery Shopping Flow
Here is how our technical architecture executes the exact conversation from the introduction:
| Turn | User Input / Action | System & Agent Swarm Action under the Hood | Outbound Message |
|---|---|---|---|
| 1 | “Hey, we are out of milk and eggs. Also add movie snacks.” | GKE Webhook: Receives message, posts to Pub/Sub. Redis: Loads profile ( cust_9821).Supervisor: Routes to ProfileAgent → InventoryAgent.Tool Call: search_catalog(["milk", "eggs", "snacks"]).Gemini 1.5 Pro: Filters by past preference (“Whole Milk”, “Doritos”, “Sprite”). |
"Got it! Added Whole Milk (4.29) & Eggs (3.99). Want Doritos (4.50)andSprite(4.50)andSprite(5.99) for movie night?" |
| 2 | “Swap Sprite for Coke Zero.” | Supervisor: Routes to InventoryAgent.Tool Call: search_catalog("Coke Zero").State Update: Swaps item in cart array in Redis state. Total updated to $18.77. |
“Done! Total is $18.77. Charge Visa ending 4242 and deliver Thursday 5-7 PM?” |
| 3 | “Yes, perfect!” | Supervisor: Detects confirmation → Routes to TransactionAgent.Tool Call: execute_stripe_charge(customer_id="cust_9821", amount=1877).Backend: Stripe returns charge_id: "ch_3M49". Order created in ERP. |
"Order confirmed! Order #HD-9821 is placed. See you Thursday!" |
Conclusion
The shift from GUI to CUI is not a design trend—it is an architectural evolution. By replacing static forms and apps with an orchestrated Multi-Agent Swarm powered by Google Cloud Vertex AI, GKE, and LangGraph, businesses can meet users where they already live: in chat.
Let’s build a new era of agentic workflows.