Securing WebSockets for Agentic & Real-Time Architectures

[joel_gauci]

Securing WebSockets for Agentic & Real-Time Architectures: Private Southbound Connectivity from Apigee X to Cloud Run via PSC and Regional ILB

In modern cloud-native and Agentic AI architectures, traditional request/response REST APIs are often no longer enough. Whether you are building real-time financial tickers, collaborative workspaces, or autonomous AI Agents that stream reasoning steps, execute tools over persistent sessions, or process live voice/video streams, WebSockets (wss://) have become a foundational transport layer.

However, exposing WebSocket microservices in an enterprise environment introduces a classic architectural dilemma:

1. Security & Governance: You want Apigee X in front of your WebSocket services to enforce authentication, API governance, traffic management, and header inspection.
2. Zero-Trust Network Isolation: You want your backend Cloud Run services to be 100% private (--ingress=internal-and-cloud-load-balancing), completely inaccessible from the public internet.

In this article, we walk through how to build a production-grade, end-to-end Private Southbound WebSocket architecture connecting Apigee X → Private Service Connect (PSC) → Regional Internal Application Load Balancer → Serverless NEG → Private Cloud Run, along with the subtle Google Cloud and Apigee configuration details that make long-lived WebSocket connections work reliably.

All the source code, Go WebSocket server/client, automated deployment scripts, and Terraform configurations are available in the open-source GitHub repository:
https://github.com/JoelGauci/cloudrun-websocket-echo

---

Why WebSockets Matter in the Era of Agentic AI

AI agents are shifting from simple turn-based chatbots to stateful, event-driven orchestrators:

• Bidirectional Streaming: Agents stream intermediate thoughts, partial tokens, and tool execution requests to clients while simultaneously receiving user interruptions, telemetry, or context updates over a single full-duplex TCP connection.
• Agent-to-Agent & Tool Protocols: Modern agentic protocols increasingly rely on persistent WebSockets to avoid the latency overhead of repeated TLS handshakes and HTTP headers.
• Security at the Edge: When an AI Agent or external client initiates a wss:// handshake, Apigee X acts as the policy enforcement point (PEP)—validating credentials, inspecting or stripping headers, and injecting Google OIDC ID Tokens before routing the traffic privately to the backend.

---

Quick Overview: The Cloud Run WebSocket Echo Application

To demonstrate and test this architecture, the repository (cloudrun-websocket-echo) provides a lightweight microservice written in Go:

• Bidirectional WebSocket Server (main.go):
  • Accepts WebSocket upgrade requests (Upgrade: websocket) on GET /connect (or any custom subpath) and echoes back structured JSON enriched with RFC3339 UTC timestamps, human-readable English time, caller authentication status, and matched path.
  • Sends automatic Ping/Pong keep-alive heartbeats every 54 seconds to prevent idle proxy timeouts.
• Built-in Browser WebSocket Tester UI (GET /):
  • Features a Light / Dark Mode interface.
  • Automatically detects the API Gateway base path (e.g. /v1/wsecho) and provides an interactive Path after base input field so you can test /connect or any custom WebSocket route live from your browser.
• Go CLI Test Client (cmd/client/main.go):
  • Supports custom HTTP headers (-H "Header-Name: Value"), Cloud Run Bearer tokens (-token), and repeated message streaming.

---

Deep Dive: The Private Southbound Architecture

When Cloud Run is configured with --ingress=internal-and-cloud-load-balancing, public traffic is blocked at Google’s edge. Because Apigee X runs in a Google-managed tenant project, we bridge Apigee X into your VPC using Private Service Connect (PSC) fronting a Regional Internal Application Load Balancer (INTERNAL_MANAGED) and a Serverless NEG.

[Client / AI Agent]
   │ wss://${APIGEE_HOST}/v1/wsecho/connect
   ▼
[Apigee X Runtime (Google-Managed Tenant Project)]
   │ Southbound PSC Endpoint Attachment (websocket-echo-ea -> ${ENDPOINT_ATTACHMENT_HOST})
   │ Request/WebSocket Timeout: io.timeout.millis = 3600000 (3600s)
   │ Policy: AM-SetTargetHost (Host: ${CLOUD_RUN_HOST}) + GoogleIDToken
   ▼
[PSC Service Attachment (websocket-echo-service-attachment)]
   │ NAT Subnet: psc-nat-subnet-ws-echo (purpose: PRIVATE_SERVICE_CONNECT)
   ▼
[Regional Internal HTTPS Load Balancer (INTERNAL_MANAGED, Port 443)]
   │ Native WebSocket Support (HTTP/1.1 Upgrade & HTTP/2)
   │ Proxy Subnet: proxy-only-subnet-ew1 (purpose: REGIONAL_MANAGED_PROXY)
   │ Backend Service: websocket-echo-ilb-backend (HTTPS)
   ▼
[Serverless NEG (websocket-echo-neg)]
   ▼
[Cloud Run Service (websocket-echo)]
     --ingress=internal-and-cloud-load-balancing
     --timeout=3600
     --session-affinity

Why the Google Cloud Regional Internal Application Load Balancer (INTERNAL_MANAGED)?

A key advantage of this architecture is that the Google Cloud Regional Internal Application Load Balancer (INTERNAL_MANAGED) natively supports WebSockets out of the box with zero additional configuration required:

• Built on Envoy Proxy: Because the Regional Internal Application Load Balancer is powered by Google-managed Envoy proxies, it automatically recognizes HTTP/1.1 Connection: Upgrade and Upgrade: websocket headers (as well as HTTP/2 extended CONNECT RFC 8441) and seamlessly switches the connection into a full-duplex bidirectional TCP tunnel.
• Native Serverless NEG Integration: Unlike L4 internal passthrough load balancers, the L7 Regional Internal Application Load Balancer natively routes traffic directly to Serverless NEGs (Cloud Run) while maintaining end-to-end TLS encryption (HTTPS protocol between the ILB and Cloud Run).

---

Step-by-Step Implementation & Key Gotchas Solved

1. Private Cloud Run Deployment (--timeout=3600 & --session-affinity)

For WebSocket services on Cloud Run, two settings are critical:

• --timeout=3600: In Cloud Run, the request timeout defines the maximum lifetime of an open WebSocket connection (up to 60 minutes).
• --session-affinity: Ensures that reconnects from the same client are routed to the same container instance.
• --ingress=internal-and-cloud-load-balancing: Blocks all direct internet traffic while permitting requests coming through our Regional Internal Application Load Balancer.

gcloud run deploy websocket-echo \
  --source . \
  --region="${REGION}" \
  --ingress=internal-and-cloud-load-balancing \
  --no-allow-unauthenticated \
  --timeout=3600 \
  --session-affinity

---

2. VPC Subnets for Regional ILB & Private Service Connect

A Regional Internal Application Load Balancer (INTERNAL_MANAGED) and a PSC Service Attachment each require dedicated special-purpose subnets in your VPC:

1. Proxy-Only Subnet (REGIONAL_MANAGED_PROXY): Used by Envoy proxies managed by Google Cloud for the Regional ILB.
2. PSC NAT Subnet (PRIVATE_SERVICE_CONNECT): Used by the PSC Service Attachment to NAT traffic coming from the Apigee tenant project into your VPC.

# 1. Regional Managed Proxy Subnet
gcloud compute networks subnets create proxy-only-subnet-ew1 \
  --network="${VPC_NETWORK}" \
  --region="${REGION}" \
  --range="10.129.0.0/23" \
  --purpose="REGIONAL_MANAGED_PROXY" \
  --role="ACTIVE"

# 2. PSC NAT Subnet
gcloud compute networks subnets create psc-nat-subnet-ws-echo \
  --network="${VPC_NETWORK}" \
  --region="${REGION}" \
  --range="192.168.2.0/24" \
  --purpose="PRIVATE_SERVICE_CONNECT"

---

3. Serverless NEG & Regional Backend Service (Gotcha #1: timeoutSec)

Next, we create a Serverless NEG pointing to the Cloud Run service and attach it to a Regional Backend Service (INTERNAL_MANAGED). Because the Regional Internal Application Load Balancer natively supports WebSockets, there is no special “enable WebSockets” toggle to set on the Backend Service. However, there is a critical timeout rule when using Serverless NEGs:

Gotcha #1 — Do NOT set timeoutSec on a Backend Service with a Serverless NEG

Normally, when configuring a Load Balancer for WebSockets, you increase the backend service timeout (e.g. --timeout=3600s). However, if you attempt to attach a Serverless NEG to a Backend Service with a custom timeoutSec, Google Cloud returns:
Invalid value for field 'resource.timeoutSec': '3600'. Timeout sec is not supported for a backend service with Serverless network endpoint groups.

Why? Serverless NEGs automatically inherit their connection/WebSocket timeout directly from the underlying Cloud Run service (--timeout=3600). Keep the Backend Service timeout at its default (30s).

# Create Serverless NEG
gcloud compute network-endpoint-groups create websocket-echo-neg \
  --region="${REGION}" \
  --network-endpoint-type="serverless" \
  --cloud-run-service="websocket-echo"

# Create Regional Backend Service (keep default timeoutSec!)
gcloud compute backend-services create websocket-echo-ilb-backend \
  --load-balancing-scheme="INTERNAL_MANAGED" \
  --protocol="HTTPS" \
  --region="${REGION}"

gcloud compute backend-services add-backend websocket-echo-ilb-backend \
  --network-endpoint-group="websocket-echo-neg" \
  --network-endpoint-group-region="${REGION}" \
  --region="${REGION}"

---

4. Regional Internal HTTPS Proxy, Forwarding Rule & PSC Service Attachment

We expose the Backend Service internally via a Regional URL Map, a self-signed TLS certificate, a Regional Target HTTPS Proxy, and an internal Forwarding Rule on port 443. Finally, we publish that Forwarding Rule as a Private Service Connect Service Attachment:

# Create PSC Service Attachment (Producer)
gcloud compute service-attachments create websocket-echo-service-attachment \
  --region="${REGION}" \
  --producer-forwarding-rule="websocket-echo-ilb-forwarding-rule" \
  --connection-preference="ACCEPT_AUTOMATIC" \
  --nat-subnets="psc-nat-subnet-ws-echo"

---

5. Apigee Endpoint Attachment & Proxy Configuration (Gotchas #2 & #3)

On the Apigee X side, we create an Endpoint Attachment pointing to our websocket-echo-service-attachment. Once ACTIVE, Apigee allocates a private IP address (${ENDPOINT_ATTACHMENT_HOST}) inside its tenant network that routes directly to our Regional ILB.

To make WebSockets work end-to-end through Apigee X to Cloud Run over PSC, three proxy configurations are essential:

A. Aligning Apigee’s Request Timeout to 3600 Seconds (io.timeout.millis)

By default, Apigee closes connections after 55–60 seconds. To support 1-hour WebSocket sessions matching Cloud Run, set <Property name="io.timeout.millis">3600000</Property> (3,600,000 ms) in both <HTTPProxyConnection> and <HTTPTargetConnection>.

Gotcha #2 — Do NOT set keepalive.timeout.millis on HTTPTargetConnection for WebSockets

If you add <Property name="keepalive.timeout.millis">3600000</Property> inside <HTTPTargetConnection>, Apigee’s target HTTP client forces the outbound header Connection: keep-alive instead of preserving Connection: Upgrade. When the handshake reaches Cloud Run, the Go WebSocket server rejects it with HTTP 400 Bad Request ('upgrade' token not found in 'Connection' header).

Solution: Only configure io.timeout.millis (3600000). Do not set keepalive.timeout.millis.

B. Rewriting the Host Header for Cloud Run GFE (Gotcha #3)

Gotcha #3 — Cloud Run Serverless NEG requires the Cloud Run Host header

When Apigee forwards requests to the private PSC IP (https://${ENDPOINT_ATTACHMENT_HOST}), the default HTTP Host header is the IP address itself. When the Regional ILB forwards a request with an IP Host header to a Serverless NEG, Google Front End (GFE) returns 404 Not Found because no Cloud Run service is mapped to that IP.

Solution: Attach an AssignMessage policy (AM-SetTargetHost) in the TargetEndpoint PreFlow that explicitly sets the Host header to your Cloud Run service hostname, while authenticating with <GoogleIDToken>:

<!-- apiproxy/policies/AM-SetTargetHost.xml -->
<AssignMessage continueOnError="false" enabled="true" name="AM-SetTargetHost">
  <Set>
    <Headers>
      <Header name="Host">${CLOUD_RUN_HOST}</Header>
    </Headers>
  </Set>
  <IgnoreUnresolvedVariables>true</IgnoreUnresolvedVariables>
  <AssignTo createNew="false" transport="http" type="request"/>
</AssignMessage>

<!-- apiproxy/targets/default.xml -->
<TargetEndpoint name="default">
  <PreFlow name="PreFlow">
    <Request>
      <Step>
        <Name>AM-SetTargetHost</Name>
      </Step>
    </Request>
  </PreFlow>
  <HTTPTargetConnection>
    <Properties>
      <Property name="io.timeout.millis">3600000</Property>
    </Properties>
    <SSLInfo>
      <Enabled>true</Enabled>
      <IgnoreValidationErrors>true</IgnoreValidationErrors>
    </SSLInfo>
    <URL>https://${ENDPOINT_ATTACHMENT_HOST}</URL>
    <Authentication>
      <GoogleIDToken>
        <Audience>${CLOUD_RUN_URL}</Audience>
      </GoogleIDToken>
    </Authentication>
  </HTTPTargetConnection>
</TargetEndpoint>

---

Testing the End-to-End Flow

Once deployed using the automated scripts in the repository (./deploy/setup_ilb_psc_apigee.sh and ./deploy/update_apigee_proxy.sh), you can test the private WebSocket tunnel through Apigee X:

1. Using the Go CLI Client with Custom Headers (-H)

go run ./cmd/client \
  -url "wss://${APIGEE_HOST}/v1/wsecho/connect" \
  -insecure \
  -H "X-Agent-Session: agent-123" \
  -H "X-Trace-ID: trace-456" \
  -msg "Hello from AI Agent over Southbound PSC!"

Response:

{
  "echo": "Hello from AI Agent over Southbound PSC!",
  "path": "/connect",
  "received_at_utc": "2026-09-14T09:49:00Z",
  "formatted_time": "Monday, 14-Sep-2026 09:49:00 UTC",
  "authenticated": true
}

2. Using the Built-In Browser Tester (Light & Dark Mode)

Navigate directly to https://${APIGEE_HOST}/v1/wsecho in any browser:

• Toggle between Light Mode and Dark Mode.
• Customize the Path after base (default /connect or any custom route like /agent/stream).
• Click Connect and send real-time messages through Apigee X → Private Service Connect → Regional ILB → Cloud Run.

---

Get Started

Check out the complete repository with source code, shell scripts, and Terraform modules at:
https://github.com/JoelGauci/cloudrun-websocket-echo

---

Ready to build secure, real-time architectures for your AI agents? Join Joel Gauci and Google Cloud experts in person at Apigee AI Horizon in Paris on October 13, 2026, at the Google Paris office to explore how to govern and scale persistent agent workflows using Apigee.

Register today

---

Let’s talk — Google Cloud Sales Specialist awaits