Hello there,
I’m having a hard time to debug why cloudrun is not allowed to make outbound http to the public internet.
My setup follow a Global loadbalancer (Classic) → CloudRun internal-loadbalancer. I use the serverless VPC connector explained here:
https://cloud.google.com/vpc/docs/configure-serverless-vpc-access#create-connector
I can reach my Cloudrun service just fine by hitting the URL in the browser, however my server is making an http request to a public endpoint which fails with a timeout.
I’ve also tested by hitting a different public url (Wikipedia) and the same error happens. So it is definitely an egress issue.
I’ve added a firewall rule to allow Egress to all ports and ips as following:
But didn’t make the difference.
Here are my routes:
Please any help would be appreciated!
djs_75
August 10, 2022, 1:09pm
2
1 Like
Hi @djs_75 , thanks for getting back to me
Yes, I have routed the egress traffic to the VPC connector as follows:
annotations:
run.googleapis.com/client-name: gcloud
generated-by: terraform
client.knative.dev/user-image: [MY_IMAGE]
run.googleapis.com/client-version: 397.0.0
autoscaling.knative.dev/minScale: '1'
run.googleapis.com/vpc-access-egress: all-traffic
autoscaling.knative.dev/maxScale: '2'
run.googleapis.com/vpc-access-connector: central-serverless
Here is my entire network configuration:
Note: i also tried to change allow-internal to allow all IPs (0.0.0.0/0) the egress rule was just a test to see if it made any difference
Here is the entire Yaml definition from cloudrun (with some info redacted) if it helps:
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
name: web-app
namespace: '447640861145'
selfLink: /apis/serving.knative.dev/v1/namespaces/447640861145/services/web-app
uid: 27843a4c-1ec0-4a67-9d31-1a8325d4ef5d
resourceVersion: AAXl44oBy9Q
generation: 26
creationTimestamp: '2022-08-09T13:47:03.660740Z'
labels:
cloud.googleapis.com/location: europe-west3
annotations:
serving.knative.dev/creator: XXXX
serving.knative.dev/lastModifier: XXXX
run.googleapis.com/vpc-access-egress: all-traffic
run.googleapis.com/vpc-access-connector: central-serverless
run.googleapis.com/ingress: internal-and-cloud-load-balancing
run.googleapis.com/ingress-status: internal-and-cloud-load-balancing
spec:
template:
metadata:
annotations:
run.googleapis.com/client-name: terraform
generated-by: terraform
autoscaling.knative.dev/minScale: '1'
run.googleapis.com/vpc-access-egress: all-traffic
autoscaling.knative.dev/maxScale: '2'
run.googleapis.com/vpc-access-connector: central-serverless
spec:
containerConcurrency: 80
timeoutSeconds: 120
serviceAccountName: web-app-cloudrun@alpha-shopping-mall.iam.gserviceaccount.com
containers:
- image: XXXX
ports:
- name: http1
containerPort: 8080
env:
- name: CLOUD_RUN_PROJECT_HASH
value: XXXX
- name: SERVER_PORT
value: '8080'
- name: AUTH_URL
value: XXX
- name: AUTH_REALM
value: XXXX
- name: AUTH_APP_BASE_URL
value: XXXX
- name: AUTH_CLIENT_ID
value: web-app
resources:
limits:
cpu: 1000m
memory: 512Mi
traffic:
- percent: 100
latestRevision: true
status:
observedGeneration: 26
conditions:
- type: Ready
status: 'True'
lastTransitionTime: '2022-08-10T14:04:07.930411Z'
- type: ConfigurationsReady
status: 'True'
lastTransitionTime: '2022-08-10T14:04:00.301442Z'
- type: RoutesReady
status: 'True'
lastTransitionTime: '2022-08-10T14:04:07.930411Z'
latestReadyRevisionName: web-app-s9xlc
latestCreatedRevisionName: web-app-s9xlc
traffic:
- revisionName: web-app-s9xlc
percent: 100
latestRevision: true
url: [https://XXXX.app](https://XXXX.app)
address:
url: [https://XXXXX-ey.a.run.app](https://XXXXX-ey.a.run.app)
djs_75
August 10, 2022, 2:10pm
5
Based on the information above there does not appear to be an external ip address - i would configure an Endpoint
djs_75
August 10, 2022, 2:12pm
6
If you have access to Network Intelligence center - you can check the route from the internal to ensure there are no firewalls inhibiting it - but if there is no external IP to associate the service with - you won’t be able to hit it externally. If you’re using NAT to get out - you would need a External Load Balancer or Endpoint defined to service traffic
@djs_75 mmmh,
I thought I didn’t need a static IP, since I’d be ok with a dynamic IP from a pool to make external requests
I think i see your point now, I would try to setup a NAT to get out, since i already have an external loadbalancer in place to serve public traffic
djs_75
August 10, 2022, 2:26pm
8
If you have an external load balancer and it pulled a ip from the pool - look at that and see what External it assigned - ensure your Cloud Run is in the backend
@djs_75 You can’t imagine how thankful I am!
After I setup a NAT with a static IP my instances are now able to go to the public internet
Wish you the best, seriously!
1 Like
