Modernizing and Decoupling Event Ingestion: Apigee as a Gateway for Google Cloud Pub/Sub

[welylau]

Modernizing and Decoupling Event Ingestion: Apigee as a Gateway for Google Cloud Pub/Sub

In modern cloud-native architectures, the ability to decouple producers from consumers is critical for building resilient, “fire-and-forget” asynchronous processing systems. While Google Cloud Pub/Sub provides a highly scalable backbone for these events, exposing it directly to external partners or client applications can introduce security risks and management overhead. Especially, the API consumers have already an existing application identity registered with the Apigee.

This post explores how to leverage Apigee as an intelligent HTTP ingestion point to safely and efficiently proxy messages to GCP Pub/Sub.

1. The Problem Statement: Decoupling Ingestion from Processing

Many enterprises face the challenge of ingesting high volumes of telemetry, order data, or user events from diverse client applications. Direct integration with Pub/Sub poses several hurdles:

• Security Complexity: Direct access requires managing complex IAM permissions for every external client.
• Payload Validation: Pub/Sub accepts any valid message, but backend consumers often require strict schema adherence or payload transformation before processing.
• Traffic Surges: Sudden spikes in ingestion can overwhelm downstream systems if not throttled or managed at the entry point.

By placing Apigee in-front-of the Pub/Sub, you create a managed facade that handles security, mediation, and traffic control before the message ever reaches your internal message bus.

Security options: Direct vs. via Apigee

In the context of publishing messages to Pub/Sub, the publisher would need to have a Google Account (typically a Service Account) with the respective Pub/Sub Publisher permission (roles/pubsub.publisher) in the IAM. This option is generally fine for an internal app within the enterprise.

Apigee comes with much wider security options such as OAuth 2.0, JWS/JWT, VerifyAPIKey, etc. This allows customers to decouple the consumer-app’s identity (app to Apigee) and the service account (Apigee to Pub/Sub).

2. Technical Implementation: Two variations

There are two primary ways to implement this: using the PublishMessage policy in a “target-less” proxy or using the Pub/Sub API as a target endpoint.

1. Using the PublishMessage policy.

Apigee introduces a native PublishMessage policy specifically designed to integrate with Google Cloud services like Pub/Sub.

Example PublishMessage Policy

Here’s the example of the PublishMessage policy and the Proxy Endpoint snippets:

Publish Message policy:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<PublishMessage continueOnError="false" enabled="true" name="PM-apigee-ingest">
 <DisplayName>PM-apigee-ingest</DisplayName>
 <Source>{request.queryparam.message}</Source>
 <CloudPubSub>
   <Topic>projects/{request.header.project}/topics/{request.header.topic}</Topic>
 </CloudPubSub>
</PublishMessage>

Proxy Endpoint:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ProxyEndpoint name="default">
 <Description/>
 <FaultRules/>
 <PreFlow name="PreFlow">
   <Request>
     <Step>
       <Name>PM-apigee-ingest</Name>
     </Step>
   </Request>
   <Response/>
 </PreFlow>
 <PostFlow name="PostFlow">
   <Request/>
   <Response/>
 </PostFlow>
 <Flows/>
 <HTTPProxyConnection>
   <BasePath>/proxytopubsub</BasePath>
   <Properties/>
 </HTTPProxyConnection>
 <RouteRule name="default"/>
</ProxyEndpoint>

In this configuration, the policy takes the incoming request.queryparam.message and publishes it directly to the specified topic. This is typically placed in the ProxyEndpoint Response or PreFlow to ensure the client receives a quick acknowledgment (e.g., a 202 Accepted).

The Pros and Cons

Pros	Cons
Simplicity: Extremely easy to configure. You just drop the policy into the flow and provide the topic name.	Opaque Execution: Since there is no target, standard "Target" analytics (like target response time) aren't captured automatically.
Performance: Usually faster for simple "fire and forget" scenarios because Apigee doesn't have to manage a full HTTP client-target connection lifecycle.	Error Handling: Catching specific Pub/Sub API errors can be slightly more rigid within the policy's fault rules compared to a Target's Error flow.

2. Using the Pub/Sub API as the target endpoint

In this pattern, Apigee treats Pub/Sub like any other backend REST API (pubsub.googleapis.com). You use a standard Target Endpoint configuration.

Here’re the sample code snippets:

For the Proxy Endpoint

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ProxyEndpoint name="default">
 <Description/>
 <FaultRules/>
 <PreFlow name="PreFlow">
   <Request>
     <Step>
       <Name>AM-pubmessage</Name>
     </Step>
   </Request>
   <Response/>
 </PreFlow>
 <PostFlow name="PostFlow">
   <Request/>
   <Response/>
 </PostFlow>
 <Flows/>
 <HTTPProxyConnection>
   <BasePath>/proxy-pubsub-target</BasePath>
   <Properties/>
 </HTTPProxyConnection>
 <RouteRule name="default">
   <TargetEndpoint>default</TargetEndpoint>
 </RouteRule>
</ProxyEndpoint>

For the AssignMessage policy

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<AssignMessage continueOnError="false" enabled="true" name="AM-pubmessage">
 <DisplayName>AM-pubmessage</DisplayName>
 <Properties/>
 <Set>
   <Payload contentType="application/json">
         {
           "messages": [
             {
               "data": "xxx",
               "attributes": {
                 "origin": "apigee",
                 "id": "test123"
               }
             }
           ]
         }
     </Payload>
 </Set>
 <IgnoreUnresolvedVariables>true</IgnoreUnresolvedVariables>
 <AssignTo createNew="false" transport="http" type="request"/>
</AssignMessage>

For the TargetEndpoint

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<TargetEndpoint name="default">
 <Description/>
 <FaultRules/>
 <PreFlow name="PreFlow">
   <Request/>
   <Response/>
 </PreFlow>
 <PostFlow name="PostFlow">
   <Request/>
   <Response/>
 </PostFlow>
 <Flows/>
 <HTTPTargetConnection>
   <Properties/>
   <Authentication>
     <GoogleAccessToken>
       <Scopes>
         <Scope>https://www.googleapis.com/auth/pubsub</Scope>
       </Scopes>
     </GoogleAccessToken>
   </Authentication>
   <URL>https://pubsub.googleapis.com/v1/projects/{project}/topics/{topic}:publish</URL>
 </HTTPTargetConnection>
</TargetEndpoint>

The Pros and Cons

Pros	Cons
Better Analytics: Apigee provides out-of-the-box visibility into Target Latency, Target Errors, and Success rates, making it easier to debug backend performance issues.	Slightly Higher Overhead: Requires managing authentication (Google OAuth tokens) manually via a Service Account or the GoogleAccessToken element.
Consistent: It’s arguably easier to modify or replace the target endpoint should in the future, you decide to replace Pub/Sub endpoint with some other services.	More "Moving Parts": Small changes to the Pub/Sub API require you to update your Target Endpoint logic, whereas the Policy handles much of this under the hood.

3. Securing the Flow: The Service Account

For Apigee to publish messages to your Pub/Sub topic, it must act as a legitimate identity within Google Cloud. This is achieved using a Service Account.

1. Create the Service Account: Generate a Service Account in the GCP project where Apigee is provisioned.
2. Grant Permissions: Assign the Pub/Sub Publisher role to this Service Account for the specific topic or project.
3. Associate with Apigee: The Service Account must be associated with the Apigee environment or specified during the proxy deployment.

4. Conclusion and Summary

While direct access (from the consumer to the Pub/Sub) is possible, the benefits of the Apigee-first approach are significant:

Feature	Direct Pub/Sub Access	Apigee + Pub/Sub
Authentication	IAM-based (Complex for external clients)	API Key, OAuth2, OIDC, or HMAC
Traffic Control	Limited to Pub/Sub quotas	Fine-grained Quotas and Spike Arrest per Client
Mediation	None (Ingests raw payload)	Transform JSON to XML, strip headers, or inject metadata
Analytics	Infrastructure-level metrics	Business-centric reports (e.g., Traffic by Partner App)
Visibility	Log monitoring	Real-time Debug/Trace of the message flow

By utilizing Apigee, you transform a simple message bus into a managed API product. This provides your platform team with the visibility and control needed to scale event-driven architectures without sacrificing security or performance.

Sample codes can be found here.

Thanks to my peer Joey Wong for the peer-review.

Tailor your solution with help from a Google Cloud Sales Specialist