Apigee AI Gateway can help unify & add governance to many parts of your AI landscape, from models, to tools and agents.
In this tutorial we’re going to focus on deploying an LLM Routing Routing Proxy to Apigee, and routing between many leading model providers from a common endpoint. Let’s jump right in!
Step 1: Create an Apigee Proxy
An Apigee Proxy is just an endpoint that receives any type of traffic, including LLM and model traffic, and routes, secures and transforms it to any type of backend service. In Apigee you can create proxies in the Google Cloud Console, Terraform, OpenAPI specs, or through Apigee YAML templates, which we will use in this tutorial.
Here is the designer view of the proxy in the console. As you can see in the Proxy Endpoint definition, we will be receiving traffic at the /llm path of our load balancer, and then routing and forwarding traffic either to the target services Anthropic, Google Cloud Model Garden, Gemini, or OpenAI.
The YAML definition looks like this, with our primary endpoint, and routing rules to each of our target services.
name: LLM Router
displayName: LLM Router
type: feature
description: This proxy routes LLM traffic between Google Cloud Model Garden, Gemini, Anthropic and OpenAI.
defaultEndpoint:
name: default
basePath: /llm
routes:
- name: anthropic
condition: proxy.pathsuffix MatchesPath "/v1/messages/**"
target: anthropic
- name: googlecloud
condition: proxy.pathsuffix MatchesPath "/v1/projects/**"
target: googlecloud
- name: openai
condition: proxy.pathsuffix MatchesPath "/v1/responses/**"
target: openai
- name: gemini
condition: proxy.pathsuffix MatchesPath "/v1/models/**"
target: gemini
flows: []
faultRules: []
endpoints: []
targets:
- name: anthropic
url: https://api.anthropic.com
flows: []
faultRules: []
httpTargetConnection:
properties:
property:
metadata:
name: io.timeout.millis
value: "18000000"
url: https://api.anthropic.com
- name: gemini
url: https://generativelanguage.googleapis.com
flows: []
faultRules: []
httpTargetConnection:
properties:
property:
metadata:
name: io.timeout.millis
value: "18000000"
url: https://generativelanguage.googleapis.com
- name: googlecloud
url: https://aiplatform.googleapis.com
flows: []
faultRules: []
httpTargetConnection:
properties:
property:
metadata:
name: io.timeout.millis
value: "18000000"
url: https://aiplatform.googleapis.com
- name: openai
url: https://api.openai.com
flows: []
faultRules: []
httpTargetConnection:
properties:
property:
metadata:
name: io.timeout.millis
value: "18000000"
url: https://api.openai.com
policies: []
resources: []
As you can see, the proxy above tests the incoming request for the type of API call being made, and routes to the appropiate provider based on the messaging protocol being used. Currently our proxy doesn’t have any policy checks or transformations, but we can add that in later to match our security & API requirements.
The important thing is that users get a unified LLM endpoint, with full streaming & model support for any provider, and we get analytics, tracing, auditing & finops support, with no changes needed to the clients or user environments, the proxy can add this seamlessly to the traffic requests and streams.
Step 2: Deploy the Proxy
To deploy the proxy, we can simply use the Apigee Feature Templater tool to deploy directly to our Apigee X project in Google Cloud.
GOOGLE_CLOUD_PROJECT=YOUR_PROJECT_ID
aft -i LLM-Router.yaml -o $GOOGLE_CLOUD_PROJECT:LLM-Router
Step 3: Test LLM Router Proxy
After we have deployed the proxy, we can test with various LLM API calls.
# Set the Apigee host address, usually a Google Cloud Load Balancer
APIGEE_HOST=YOUR_APIGEE_LB_HOST
# Call Gemini with an AI Studio AI key
GEMINI_API_KEY=YOUR_GEMINI_KEY
curl -i "https://$APIGEE_HOST/llm/v1/models/gemini-3.5-flash:generateContent" -H "x-goog-api-key: $GEMINI_API_KEY" -H 'Content-Type: application/json' -X POST -d '{
"contents": [
{
"parts": [
{
"text": "Explain how AI works in a few words"
}
]
}
]
}'
# Call Anthropic with an Anthropic key
ANTHROPIC_API_KEY=YOUR_ANTHROPIC_KEY
curl https://$APIGEE_HOST/llm/v1/messages \
--header "x-api-key: $ANTHROPIC_API_KEY" \
--header "anthropic-version: 2023-06-01" \
--header "content-type: application/json" \
--data '{"model": "claude-haiku-4-5", "max_tokens": 1024,
"messages": [{"role": "user", "content": "Why is the sky blue?"}]}'
# Call Google Cloud Model Garden models with default identity
GOOGLE_CLOUD_PROJECT=YOUR_PROJECT_ID
GOOGLE_CLOUD_LOCATION=global
curl -i -X POST "https://$APIGEE_HOST/llm/v1/projects/$GOOGLE_CLOUD_PROJECT/locations/$GOOGLE_CLOUD_LOCATION/publishers/google/models/gemini-flash-latest:generateContent" \
-H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
-H "Content-Type: application/json" \
--data-binary @- << EOF
{
"contents": [
{
"role": "USER",
"parts": [
{
"text": "Why is the sky blue?"
}
]
}
]
}
EOF
If you start a Apigee Debug Session, you will see every step of the proxy, and how the traffic is recieved and routed to appropriate LLM provider.
As a next step we can start adding policies to validate & authorize the user, do a token exchange between a user token and the LLM provider token or key, and start monitoring & shaping the LLM analytics & cost management. Check out some more AI Gateway Labs in GitHub, stay tuned for more tutorials here, and let us know of any questions or suggestions in the comments!
Ready to try it yourself? Get in touch with a Google Cloud Sales Specialist