Securing WebSockets for Agentic & Real-Time Architectures

joel_gauci · September 14, 2026, 11:32am

Securing WebSockets for Agentic & Real-Time Architectures: Private Southbound Connectivity from Apigee X to Cloud Run via PSC and Regional ILB

In modern cloud-native and Agentic AI architectures, traditional request/response REST APIs are often no longer enough. Whether you are building real-time financial tickers, collaborative workspaces, or autonomous AI Agents that stream reasoning steps, execute tools over persistent sessions, or process live voice/video streams, WebSockets (wss://) have become a foundational transport layer.

However, exposing WebSocket microservices in an enterprise environment introduces a classic architectural dilemma:

Security & Governance: You want Apigee X in front of your WebSocket services to enforce authentication, API governance, traffic management, and header inspection.
Zero-Trust Network Isolation: You want your backend Cloud Run services to be 100% private (--ingress=internal-and-cloud-load-balancing), completely inaccessible from the public internet.

In this article, we walk through how to build a production-grade, end-to-end Private Southbound WebSocket architecture connecting Apigee X → Private Service Connect (PSC) → Regional Internal Application Load Balancer → Serverless NEG → Private Cloud Run, along with the subtle Google Cloud and Apigee configuration details that make long-lived WebSocket connections work reliably.

All the source code, Go WebSocket server/client, automated deployment scripts, and Terraform configurations are available in the open-source GitHub repository:
https://github.com/JoelGauci/cloudrun-websocket-echo

Why WebSockets Matter in the Era of Agentic AI

AI agents are shifting from simple turn-based chatbots to stateful, event-driven orchestrators:

Bidirectional Streaming: Agents stream intermediate thoughts, partial tokens, and tool execution requests to clients while simultaneously receiving user interruptions, telemetry, or context updates over a single full-duplex TCP connection.
Agent-to-Agent & Tool Protocols: Modern agentic protocols increasingly rely on persistent WebSockets to avoid the latency overhead of repeated TLS handshakes and HTTP headers.
Security at the Edge: When an AI Agent or external client initiates a wss:// handshake, Apigee X acts as the policy enforcement point (PEP)—validating credentials, inspecting or stripping headers, and injecting Google OIDC ID Tokens before routing the traffic privately to the backend.

Quick Overview: The Cloud Run WebSocket Echo Application

To demonstrate and test this architecture, the repository (cloudrun-websocket-echo) provides a lightweight microservice written in Go:

Bidirectional WebSocket Server (main.go):
- Accepts WebSocket upgrade requests (Upgrade: websocket) on GET /connect (or any custom subpath) and echoes back structured JSON enriched with RFC3339 UTC timestamps, human-readable English time, caller authentication status, and matched path.
- Sends automatic Ping/Pong keep-alive heartbeats every 54 seconds to prevent idle proxy timeouts.
Built-in Browser WebSocket Tester UI (GET /):
- Features a Light / Dark Mode interface.
- Automatically detects the API Gateway base path (e.g. /v1/wsecho) and provides an interactive Path after base input field so you can test /connect or any custom WebSocket route live from your browser.
Go CLI Test Client (cmd/client/main.go):
- Supports custom HTTP headers (-H "Header-Name: Value"), Cloud Run Bearer tokens (-token), and repeated message streaming.

Deep Dive: The Private Southbound Architecture

When Cloud Run is configured with --ingress=internal-and-cloud-load-balancing, public traffic is blocked at Google’s edge. Because Apigee X runs in a Google-managed tenant project, we bridge Apigee X into your VPC using Private Service Connect (PSC) fronting a Regional Internal Application Load Balancer (INTERNAL_MANAGED) and a Serverless NEG.

[Client / AI Agent]
   │ wss://${APIGEE_HOST}/v1/wsecho/connect
   ▼
[Apigee X Runtime (Google-Managed Tenant Project)]
   │ Southbound PSC Endpoint Attachment (websocket-echo-ea -> ${ENDPOINT_ATTACHMENT_HOST})
   │ Request/WebSocket Timeout: io.timeout.millis = 3600000 (3600s)
   │ Policy: AM-SetTargetHost (Host: ${CLOUD_RUN_HOST}) + GoogleIDToken
   ▼
[PSC Service Attachment (websocket-echo-service-attachment)]
   │ NAT Subnet: psc-nat-subnet-ws-echo (purpose: PRIVATE_SERVICE_CONNECT)
   ▼
[Regional Internal HTTPS Load Balancer (INTERNAL_MANAGED, Port 443)]
   │ Native WebSocket Support (HTTP/1.1 Upgrade & HTTP/2)
   │ Proxy Subnet: proxy-only-subnet-ew1 (purpose: REGIONAL_MANAGED_PROXY)
   │ Backend Service: websocket-echo-ilb-backend (HTTPS)
   ▼
[Serverless NEG (websocket-echo-neg)]
   ▼
[Cloud Run Service (websocket-echo)]
     --ingress=internal-and-cloud-load-balancing
     --timeout=3600
     --session-affinity

Why the Google Cloud Regional Internal Application Load Balancer (`INTERNAL_MANAGED`)?

A key advantage of this architecture is that the Google Cloud Regional Internal Application Load Balancer (INTERNAL_MANAGED) natively supports WebSockets out of the box with zero additional configuration required:

Built on Envoy Proxy: Because the Regional Internal Application Load Balancer is powered by Google-managed Envoy proxies, it automatically recognizes HTTP/1.1 Connection: Upgrade and Upgrade: websocket headers (as well as HTTP/2 extended CONNECT RFC 8441) and seamlessly switches the connection into a full-duplex bidirectional TCP tunnel.
Native Serverless NEG Integration: Unlike L4 internal passthrough load balancers, the L7 Regional Internal Application Load Balancer natively routes traffic directly to Serverless NEGs (Cloud Run) while maintaining end-to-end TLS encryption (HTTPS protocol between the ILB and Cloud Run).

Step-by-Step Implementation & Key Gotchas Solved

1. Private Cloud Run Deployment (`--timeout=3600` & `--session-affinity`)

For WebSocket services on Cloud Run, two settings are critical:

--timeout=3600: In Cloud Run, the request timeout defines the maximum lifetime of an open WebSocket connection (up to 60 minutes).
--session-affinity: Ensures that reconnects from the same client are routed to the same container instance.
--ingress=internal-and-cloud-load-balancing: Blocks all direct internet traffic while permitting requests coming through our Regional Internal Application Load Balancer.

gcloud run deploy websocket-echo \
  --source . \
  --region="${REGION}" \
  --ingress=internal-and-cloud-load-balancing \
  --no-allow-unauthenticated \
  --timeout=3600 \
  --session-affinity

2. VPC Subnets for Regional ILB & Private Service Connect

A Regional Internal Application Load Balancer (INTERNAL_MANAGED) and a PSC Service Attachment each require dedicated special-purpose subnets in your VPC:

Proxy-Only Subnet (REGIONAL_MANAGED_PROXY): Used by Envoy proxies managed by Google Cloud for the Regional ILB.
PSC NAT Subnet (PRIVATE_SERVICE_CONNECT): Used by the PSC Service Attachment to NAT traffic coming from the Apigee tenant project into your VPC.

# 1. Regional Managed Proxy Subnet
gcloud compute networks subnets create proxy-only-subnet-ew1 \
  --network="${VPC_NETWORK}" \
  --region="${REGION}" \
  --range="10.129.0.0/23" \
  --purpose="REGIONAL_MANAGED_PROXY" \
  --role="ACTIVE"

# 2. PSC NAT Subnet
gcloud compute networks subnets create psc-nat-subnet-ws-echo \
  --network="${VPC_NETWORK}" \
  --region="${REGION}" \
  --range="192.168.2.0/24" \
  --purpose="PRIVATE_SERVICE_CONNECT"

3. Serverless NEG & Regional Backend Service (Gotcha #1: `timeoutSec`)

Next, we create a Serverless NEG pointing to the Cloud Run service and attach it to a Regional Backend Service (INTERNAL_MANAGED). Because the Regional Internal Application Load Balancer natively supports WebSockets, there is no special “enable WebSockets” toggle to set on the Backend Service. However, there is a critical timeout rule when using Serverless NEGs:

Gotcha #1 — Do NOT set timeoutSec on a Backend Service with a Serverless NEG

Normally, when configuring a Load Balancer for WebSockets, you increase the backend service timeout (e.g. --timeout=3600s). However, if you attempt to attach a Serverless NEG to a Backend Service with a custom timeoutSec, Google Cloud returns:
Invalid value for field 'resource.timeoutSec': '3600'. Timeout sec is not supported for a backend service with Serverless network endpoint groups.

Why? Serverless NEGs automatically inherit their connection/WebSocket timeout directly from the underlying Cloud Run service (--timeout=3600). Keep the Backend Service timeout at its default (30s).

# Create Serverless NEG
gcloud compute network-endpoint-groups create websocket-echo-neg \
  --region="${REGION}" \
  --network-endpoint-type="serverless" \
  --cloud-run-service="websocket-echo"

# Create Regional Backend Service (keep default timeoutSec!)
gcloud compute backend-services create websocket-echo-ilb-backend \
  --load-balancing-scheme="INTERNAL_MANAGED" \
  --protocol="HTTPS" \
  --region="${REGION}"

gcloud compute backend-services add-backend websocket-echo-ilb-backend \
  --network-endpoint-group="websocket-echo-neg" \
  --network-endpoint-group-region="${REGION}" \
  --region="${REGION}"

4. Regional Internal HTTPS Proxy, Forwarding Rule & PSC Service Attachment

We expose the Backend Service internally via a Regional URL Map, a self-signed TLS certificate, a Regional Target HTTPS Proxy, and an internal Forwarding Rule on port 443. Finally, we publish that Forwarding Rule as a Private Service Connect Service Attachment:

# Create PSC Service Attachment (Producer)
gcloud compute service-attachments create websocket-echo-service-attachment \
  --region="${REGION}" \
  --producer-forwarding-rule="websocket-echo-ilb-forwarding-rule" \
  --connection-preference="ACCEPT_AUTOMATIC" \
  --nat-subnets="psc-nat-subnet-ws-echo"

5. Apigee Endpoint Attachment & Proxy Configuration (Gotchas #2 & #3)

On the Apigee X side, we create an Endpoint Attachment pointing to our websocket-echo-service-attachment. Once ACTIVE, Apigee allocates a private IP address (${ENDPOINT_ATTACHMENT_HOST}) inside its tenant network that routes directly to our Regional ILB.

To make WebSockets work end-to-end through Apigee X to Cloud Run over PSC, three proxy configurations are essential:

A. Aligning Apigee’s Request Timeout to 3600 Seconds (`io.timeout.millis`)

By default, Apigee closes connections after 55–60 seconds. To support 1-hour WebSocket sessions matching Cloud Run, set <Property name="io.timeout.millis">3600000</Property> (3,600,000 ms) in both <HTTPProxyConnection> and <HTTPTargetConnection>.

Gotcha #2 — Do NOT set keepalive.timeout.millis on HTTPTargetConnection for WebSockets

If you add <Property name="keepalive.timeout.millis">3600000</Property> inside <HTTPTargetConnection>, Apigee’s target HTTP client forces the outbound header Connection: keep-alive instead of preserving Connection: Upgrade. When the handshake reaches Cloud Run, the Go WebSocket server rejects it with HTTP 400 Bad Request ('upgrade' token not found in 'Connection' header).

Solution: Only configure io.timeout.millis (3600000). Do not set keepalive.timeout.millis.

B. Rewriting the `Host` Header for Cloud Run GFE (Gotcha #3)

Gotcha #3 — Cloud Run Serverless NEG requires the Cloud Run Host header

When Apigee forwards requests to the private PSC IP (https://${ENDPOINT_ATTACHMENT_HOST}), the default HTTP Host header is the IP address itself. When the Regional ILB forwards a request with an IP Host header to a Serverless NEG, Google Front End (GFE) returns 404 Not Found because no Cloud Run service is mapped to that IP.

Solution: Attach an AssignMessage policy (AM-SetTargetHost) in the TargetEndpoint PreFlow that explicitly sets the Host header to your Cloud Run service hostname, while authenticating with <GoogleIDToken>:

<!-- apiproxy/policies/AM-SetTargetHost.xml -->
<AssignMessage continueOnError="false" enabled="true" name="AM-SetTargetHost">
  <Set>
    <Headers>
      <Header name="Host">${CLOUD_RUN_HOST}</Header>
    </Headers>
  </Set>
  <IgnoreUnresolvedVariables>true</IgnoreUnresolvedVariables>
  <AssignTo createNew="false" transport="http" type="request"/>
</AssignMessage>

<!-- apiproxy/targets/default.xml -->
<TargetEndpoint name="default">
  <PreFlow name="PreFlow">
    <Request>
      <Step>
        <Name>AM-SetTargetHost</Name>
      </Step>
    </Request>
  </PreFlow>
  <HTTPTargetConnection>
    <Properties>
      <Property name="io.timeout.millis">3600000</Property>
    </Properties>
    <SSLInfo>
      <Enabled>true</Enabled>
      <IgnoreValidationErrors>true</IgnoreValidationErrors>
    </SSLInfo>
    <URL>https://${ENDPOINT_ATTACHMENT_HOST}</URL>
    <Authentication>
      <GoogleIDToken>
        <Audience>${CLOUD_RUN_URL}</Audience>
      </GoogleIDToken>
    </Authentication>
  </HTTPTargetConnection>
</TargetEndpoint>

Testing the End-to-End Flow

Once deployed using the automated scripts in the repository (./deploy/setup_ilb_psc_apigee.sh and ./deploy/update_apigee_proxy.sh), you can test the private WebSocket tunnel through Apigee X:

1. Using the Go CLI Client with Custom Headers (`-H`)

go run ./cmd/client \
  -url "wss://${APIGEE_HOST}/v1/wsecho/connect" \
  -insecure \
  -H "X-Agent-Session: agent-123" \
  -H "X-Trace-ID: trace-456" \
  -msg "Hello from AI Agent over Southbound PSC!"

Response:

{
  "echo": "Hello from AI Agent over Southbound PSC!",
  "path": "/connect",
  "received_at_utc": "2026-09-14T09:49:00Z",
  "formatted_time": "Monday, 14-Sep-2026 09:49:00 UTC",
  "authenticated": true
}

2. Using the Built-In Browser Tester (Light & Dark Mode)

Navigate directly to https://${APIGEE_HOST}/v1/wsecho in any browser:

Toggle between Light Mode and Dark Mode.
Customize the Path after base (default /connect or any custom route like /agent/stream).
Click Connect and send real-time messages through Apigee X → Private Service Connect → Regional ILB → Cloud Run.

Get Started

Check out the complete repository with source code, shell scripts, and Terraform modules at:
https://github.com/JoelGauci/cloudrun-websocket-echo

Topic		Replies	Views
Apigee X Network Connectivity using Private Service Connect (PSC) Knowledge Hub apigee , infrastructure	0	858	November 24, 2022
Exploring Apigee Networking Blueprints Knowledge Hub apigee , api-management , infrastructure , server-configuration	6	1548	April 4, 2025
Connecting Gemini Enterprise Agent Runtime to Apigee Apigee apigee-general	1	99	September 10, 2026