This article is an extension of an existing article that details the ways Hybrid Runtime Ingress can be exposed for consumption. This article focuses on the steps that need to be executed for exposing Apigee Hybrid Ingress on HTTPS and HTTP via L7 external load balancer.
This execution is validated on Apigee Hybrid 1.9 deployed in GKE. With minor tweaks the same can be done for other Kubernetes platforms.
Prerequisites
Complete Apigee Hybrid Installation as detailed here. Deploy an apigee proxy with base path apigee-hybrid-helloworld via management console
Setup the following variables:
export PROJECT_ID= export DNS_PROJECT_ID=$PROJECT_ID export ORG_NAME=$PROJECT_ID export ENVIRONMENT_NAME= export ENV_GROUP= export ENV_GROUP_HOSTNAME= export INGRESS_NAME= # Directory locations variables used in the hybrid install as defined here # Confirm the values are set correctly echo $APIGEECTL_BASE echo $APIGEECTL_HOME echo $HYBRID_FILES gcloud config set project $PROJECT_ID |
|---|
Service definition to enable services exposed on the GKE nodes.
|
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
name: $ENV_GROUP-ingrs-service
namespace: apigee
annotations:
cloud.google.com/app-protocols: â{âhttpsâ:âHTTPSâ,âhttp2â:âHTTPâ}â
cloud.google.com/backend-config: â{âportsâ: {â443â: âhttp-hc-configâ,â80â: âhttp-hc-configâ}}â
cloud.google.com/neg: â{âingressâ: true}â
spec:
ports:
- name: status-port
port: 15021
protocol: TCP
targetPort: 15021
- name: http2
port: 80
protocol: TCP
targetPort: 8080
- name: https
port: 443
protocol: TCP
targetPort: 8443
selector:
app: apigee-ingressgateway #required
ingress_name: $INGRESS_NAME
org: $ORG_NAME
type: NodePort
EOF
|
| - |
Create static external IP address which will be assigned to the Load balancer.
gcloud compute addresses create apigee-global --global export EXTERNAL_IP_ADDR=$(gcloud compute addresses describe apigee-global <br> âproject $PROJECT_ID --format=âget(address)â --global); echo $EXTERNAL_IP_ADDR; |
|---|
Create Backendconfig object, this creates the needed Healthcheck for the Load balancer backend services.
|
cat <<EOF | kubectl apply -f -
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
name: http-hc-config
namespace: apigee
spec:
healthCheck:
checkIntervalSec: 15
requestPath: /healthz/ingress
EOF
|
| - |
HTTP External Load balancer to HTTP Container ports
Starting with Apigee Hybrid release 1.9.1 HTTP (Port 80) Ingress is disabled. From security standpoint use of non-secure ingress should be avoided. As per release notes. "Removed port 80 from the default Kubernetes service of Apigee Ingress Gateway."
The Ingress resource will create the external load balancer with the external address created in the above step.
|
cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.global-static-ip-name: apigee-global
name: xlb-http-apigee-ingress
namespace: apigee
spec:
defaultBackend:
service:
name: $ENV_GROUP-ingrs-service
port:
number: 80
EOF
|
| - |
ApigeeRoute definition will enable non-SNI client to reach the Ingress. The loadbalancer healthchecks does not have SNI in the request. This enables Ingress to respond to the healthcheck calls.
|
cat <<EOF | kubectl apply -f -
apiVersion: apigee.cloud.google.com/v1alpha1
kind: ApigeeRoute
metadata:
name: wildcard-gateway-apigee
namespace: apigee
spec:
hostnames:
- â*â
ports:
- number: 80
protocol: HTTP
selector:
app: apigee-ingressgateway
enableNonSniClient: true
EOF
|
| - |
In the $HYBRID_FILES/overrides/overrides.yaml add the following:
additionalGateways: [âwildcard-gateway-apigeeâ] under the virtualHosts block under env-group name and run apigeectl apply.
| cd $HYBRID_FILES ${APIGEECTL_HOME}/apigeectl apply -f $HYBRID_FILES/overrides/overrides.yaml |
|---|
Test the endpoint:
(make sure $ENV_GROUP_HOSTNAME value is added as hostname for the Apigee environment group)
| curl http://$ENV_GROUP_HOSTNAME/apigee-hybrid-helloworld --resolve $ENV_GROUP_HOSTNAME:80:$EXTERNAL_IP_ADDR |
|---|
A successful response from the above request validates the traffic is reachable from external loadbalancer to the deployed runtime pods and executed.
HTTPS - L7 - External Loadbalancer
Cert Provisioning from Letâs Encrypt
For the purpose of this article we will use Cert-Manager issuer for Letâs Encrypt and automatically provision certificates on your Apigee hybrid ingress load balancer. Thanks to Daniel Strebel on this article to configure Letâs Encrypt cert provisioning. Some of the steps are modified from the original article to work for version 1.9
Follow this doc to setup a DNS zone. Create record set type A for the hostname (ENV_GROUP_HOSTNAME) that points to your Apigee hybrid ingress IP address (GCP address created in the prerequisites step). Create record set type CAA for the hostname(ENV_GROUP_HOSTNAME) with Routing data value <0 issue âletsencrypt.orgâ> (just copy the value within the angle brackets).
| gcloud config set project $PROJECT_ID gcloud services enable --project=${PROJECT_ID} dns.googleapis.com |
|---|
| gcloud iam service-accounts create dns01-solver --display-name âdns01-solverâ --project $PROJECT_ID gcloud projects add-iam-policy-binding $DNS_PROJECT_ID <br>âmember serviceAccount:dns01-solver@$PROJECT_ID.iam.gserviceaccount.com <br>ârole roles/dns.admin gcloud iam service-accounts keys create dns-sa-key.json <br>âiam-account dns01-solver@$PROJECT_ID.iam.gserviceaccount.com |
|---|
| kubectl create secret generic clouddns-dns01-solver-svc-acct <br>âfrom-file=dns-sa-key.json -n apigee |
|---|
|
cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: cloud-dns-issuer
namespace: apigee
spec:
acme:
email: $YOUR_EMAIL_ADDRESS
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: cloud-dns-issuer-account-key
solvers:
- dns01:
cloudDNS:
project: $DNS_PROJECT_ID
serviceAccountSecretRef:
name: clouddns-dns01-solver-svc-acct
key: dns-sa-key.json
EOF
|
| - |
|
cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: cert-manager-$PROJECT_ID-default
namespace: apigee
spec:
secretName: cert-manager-$PROJECT_ID-tls
issuerRef:
name: cloud-dns-issuer
commonName: â.$ENV_GROUP_HOSTNAMEâ
dnsNames:
- $ENV_GROUP_HOSTNAME
- '.$ENV_GROUP_HOSTNAMEâ
EOF
|
| - |
(wait for 2 mts, the below command should show status as True)
| kubectl -n apigee get Certificate cert-manager-$PROJECT_ID-default kubectl get secret cert-manager-$PROJECT_ID-tls -n apigee -o yaml |
|---|
If the Certificate Kubernetes object does not have the âReadyâ attribute marked as âTrueâ beyond 5 minutes, use the below links as source to troubleshoot the issue:
https://cert-manager.io/docs/troubleshooting/acme/
https://cert-manager.io/v1.6-docs/faq/acme/
https://www.techrepublic.com/article/how-to-add-a-certificate-authority-authorization-record-in-google-domains/
HTTPS External Load Balancer to HTTP Container ports
The Ingress resource will create the external load balancer with the external address created in the above step. (Confirm the service referenced in this Ingress is created as defined in the steps above)
|
cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.allow-http: âfalseâ
kubernetes.io/ingress.global-static-ip-name: apigee-global
name: xlb-https-apigee-ingress
namespace: apigee
spec:
defaultBackend:
service:
name: $ENV_GROUP-ingrs-service
port:
number: 80
tls:
- hosts:
- $ENV_GROUP_HOSTNAME
secretName: cert-manager-$PROJECT_ID-tls
EOF
|
| - |
Edit ApigeeRoute to indicate the container is enabled to accept traffic on port and to enable non-SNI clients (healthchecks from loadbalancer)
|
cat <<EOF | kubectl apply -f -
apiVersion: apigee.cloud.google.com/v1alpha1
kind: ApigeeRoute
metadata:
name: wildcard-gateway-apigee
namespace: apigee
spec:
hostnames:
- â*â
ports:
- number: 80
protocol: HTTP
selector:
app: apigee-ingressgateway
enableNonSniClient: true
EOF
|
| - |
In the overrides.yaml add additionalGateways: [âwildcard-gateway-apigeeâ] under the virtualHosts block under env-group name.
| cd $HYBRID_FILES ${APIGEECTL_HOME}/apigeectl apply -f $HYBRID_FILES/overrides/overrides.yaml |
|---|
Test the endpoint
| curl https://$ENV_GROUP_HOSTNAME/apigee-hybrid-helloworld |
|---|
A successful response from the above request validates the traffic is reachable from external loadbalancer to the deployed runtime pods and executed.
HTTPS External Load Balancer to HTTPS Container ports
If you are using the cert generated by âLetsEncryptâ as detailed in this doc. Use the below to extract the cert and key and initialize the variables.
|
cd $HYBRID_FILES/certs
kubectl -n apigee get secret cert-manager-$PROJECT_ID-tls -o json | <br>jq -r â.data.âtls.keyââ | <br>base64 -d > $HYBRID_FILES/certs/letsencrypt_keystore_$ENV_GROUP.key
kubectl -n apigee get secret cert-manager-$PROJECT_ID-tls -o json | <br>jq -r â.data.âtls.crtââ | <br>base64 -d > $HYBRID_FILES/certs/letsencrypt_keystore_$ENV_GROUP.pem
export CERT_FILE=$HYBRID_FILES/certs/letsencrypt_keystore_$ENV_GROUP.pem
export KEY_FILE=$HYBRID_FILES/certs/letsencrypt_keystore_$ENV_GROUP.key
|
| - |
Update the overrides file for the env virtualhost section, the cert and the key file. Replace the sslCertPath and sslKeyPath values as below and run apigeectl apply on the overrides file.
virtualhosts: - name: ENVIRONMENT_GROUP_NAME selector: app: apigee-ingressgateway ingress_name: INGRESS_NAME sslCertPath: $CERT_FILE sslKeyPath: $KEY_FILE |
|---|
| cd $HYBRID_FILES ${APIGEECTL_HOME}/apigeectl apply -f $HYBRID_FILES/overrides/overrides.yaml |
|---|
|
cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.allow-http: âfalseâ
kubernetes.io/ingress.global-static-ip-name: apigee-global
name: xlb-https-apigee-ingress
namespace: apigee
spec:
defaultBackend:
service:
name: $ENV_GROUP-ingrs-service
port:
number: 443
tls:
- hosts:
- $ENV_GROUP_HOSTNAME
secretName: cert-manager-$PROJECT_ID-tls
EOF
|
| - |
Edit ApigeeRoute âwildcard-gateway-apigeeâ to contain 443 in the port list. This makes the container to accept traffic from loadbalancer only on port 443.
|
cat <<EOF | kubectl apply -f -
apiVersion: apigee.cloud.google.com/v1alpha1
kind: ApigeeRoute
metadata:
name: wildcard-gateway-apigee
namespace: apigee
spec:
hostnames:
- â*â
ports:
- number: 443
protocol: HTTPS
tls:
credentialName: $PROJECT_ID-$ENV_GROUP
mode: SIMPLE
selector:
app: apigee-ingressgateway
enableNonSniClient: true
EOF
|
| - |
In the overrides.yaml add additionalGateways: [âwildcard-gateway-apigeeâ] under the virtualHosts block under env-group name.
| cd $HYBRID_FILES ${APIGEECTL_HOME}/apigeectl apply -f overrides/overrides.yaml |
|---|
Test the endpoint
| curl https://$ENV_GROUP_HOSTNAME/apigee-hybrid-helloworld |
|---|
A successful response from the above request validates the traffic is reachable from external loadbalancer to the deployed runtime pods and executed.