Direct VPC egress from Cloud Run to GKE Enterprise cluster internal Knative service

Google Cloud Serverless Applications
1

I have tried so many things!

I have a GKE Enterprise autopilot private cluster running in a Fleet in the same GCP. The cluster was created with https://github.com/terraform-google-modules/terraform-google-kubernetes-engine/tree/master/modules/beta-autopilot-private-cluster. I can run kubectl commands through the bastion host proxy.

The fleet has the CloudRun (i.e. Knative) feature enabled, and Knative is configured on the cluster.

I followed https://cloud.google.com/kubernetes-engine/enterprise/knative-serving/docs/setup#setting_up_a_private_internal_network:

  1. Created an internal passthrough load-balancer, following https://cloud.google.com/service-mesh/docs/operate-and-maintain/gateways#managed-istiod using the deployment yamls from https://github.com/GoogleCloudPlatform/anthos-service-mesh-packages/tree/main/samples/gateways/istio-ingressgateway
  2. Opened port 15017 as per https://cloud.google.com/service-mesh/docs/operate-and-maintain/private-cluster-open-port (though my autopilot private cluster does not have a master-ipv4-cidr, so I set source ranges for my subnet, the Cluster Pod IPv4 range, and the IPv4 service range).
  3. Created a sample “helloworld” Knative service with external ingress allowed.
  4. Created a sample “helloworld-internal” Knative service with external ingress blocked (using networking.knative.dev/visibility: cluster-local).
  5. Created proxy-only subnet and firewall rule as per https://cloud.google.com/kubernetes-engine/docs/how-to/internal-load-balance-ingress#deploy-ingress
  6. Created a Gateway as per https://cloud.google.com/service-mesh/docs/operate-and-maintain/gateways#gateway_selectors

Going back to https://cloud.google.com/kubernetes-engine/enterprise/knative-serving/docs/setup#setting_up_a_private_internal_network, I created a Compute Engine virtual machine (VM) in the same zone as the GKE cluster. From this VM I can curl to the “helloworld” service using its external DNS name - curl -H Host:<external_url> http:<ilb-ip> - but I can not curl to the “helloworld-internal” service.

With my deployment steps coming from multiple Google docs sources I’m sure I’m missing something, but no idea what.

My end goal is to use Cloud Run (managed) with Direct VPC access to call my internal GKE Knative services.

2

Generally speaking, you will not be able to access Services deployed within a GKE cluster from outside the cluster unless you expose them using an internal (or external load balancer). The Service IP range is only routable within the cluster itself and not accessible from outside the cluster. Accessing the service via the ILB (which is only accessible from within your VPC) is the right approach.

3

Thanks @garisingh .

I’m unclear on what resources I need to add.
I have the internal load balancer, and have tested it by running curl to the ilb from a GCE VM in the same VPC to access the helloworld Knative service that has an external address:

curl -H Host:<hw_service_external_url> http:<ilb-ip>

However, for my helloworld-internal:

labels:
    networking.knative.dev/visibility: cluster-local

Running curl from the same GCE VM gets a 404.

curl -H Host:sample-internal.default.svc.cluster.local http:<ilb-ip>

Are there k8s resource I need to add to connect the ilb to sample-internal?

I am using the ilb deployment from anthos-service-mesh repository: /samples/gateways/istio-ingressgateway Which creates an ILB that is not explicitly linked to any Knative service Pod.

Relevant yaml from that repo folder:

apiVersion: v1
kind: Service
metadata:
  name: istio-ingressgateway
labels:
  app: istio-ingressgateway
  istio: ingressgateway
spec:
  ports:
  # status-port exposes a /healthz/ready endpoint that can be used with GKE Ingress health checks
  - name: status-port
    port: 15021
    protocol: TCP
    targetPort: 15021
  # Any ports exposed in Gateway resources should be exposed here.
  - name: http2
    port: 80
  - name: https
    port: 443
  selector:
    istio: ingressgateway
    app: istio-ingressgateway
  type: LoadBalancer

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: istio-ingressgateway
spec:
  replicas: 3
  selector:
    matchLabels:
      app: istio-ingressgateway
      istio: ingressgateway
  template:
    metadata:
      annotations:
        # This is required to tell Anthos Service Mesh to inject the gateway with the
        # required configuration.
        inject.istio.io/templates: gateway
      labels:
        app: istio-ingressgateway
        istio: ingressgateway
    spec:
      containers:
      - name: istio-proxy
        image: auto # The image will automatically update each time the pod starts.
        resources:
          limits:
            cpu: 2000m
            memory: 1024Mi
          requests:
            cpu: 100m
            memory: 128Mi
      serviceAccountName: istio-ingressgateway

The ILB deployments are in a mygw namespace.
Both helloworld and helloworld-internal are in the default namespace.
Both namespaces have the label istio-injection: enabled