AE Flex requires Private Google Access even with Cloud NAT

[anonymous]

Hi all! So I’m facing an issue when deploying an App Engine Flexible service (It’s a networking issue so I’m posting here, do not hesitate to tell me if it should go elsewhere).

We have encountered a new requirement for our service to use a static IP address for outgoing communications. To do that, a specific subnet was put in place on our VPC Network with a Cloud Router and a Cloud NAT using a reserved external IP.

Now, after following the documentation it was made clear that the easy-to-use vpc_access_connector is only available to App Engine Standard. Since we use Flexible, we had to go to another solution, using the network settings instead, specifying the network and subnet, as well as specifying the ‘internal’ IP mode, as Cloud NAT does not support services with ephemeral external adresses.

Here comes my issue. Since we have a Cloud NAT configured, Private Google Access is disabled on the subnet, as it would simply be useless. (The console even shows the following message: Private Google Access is in effect (even though it has not been enabled manually) for packets sent from this subnet’s primary and secondary IP ranges because Cloud NAT is configured for those ranges.)

Still, when deploying, we encounter the following error:

Error Response: [9] Flex operation projects/PROJECT_NAME/regions/REGION/operations/OPERATION_ID error [FAILED_PRECONDITION]: An internal error occurred while processing task /app-engine-flex/insert_flex_deployment/flex_create_resources>RESSOURCE_ID: Subnetwork ‘SUBNET_NAME’ does not have Private Google Access enabled. App Engine flexible instances using internal-only IP addresses require that Private Google Access be enabled in the target subnetwork. Have a Network Admin enable Private Google Access as described in: https://cloud.google.com/vpc/docs/configure-private-google-access#enabling-pga

My question being: Since a Cloud NAT is set up (even the console acknowledges it), why can’t GAE Flex be deployed without Private Google Access? Is this a safeguard error on Google’s side or a misconfiguration on our side?

(By the way for anyone looking at a way to use a static IP address with Flex, the documentation is unclear and many comments on the internet say it’s not supported, but I can confirm the above configuration works in our tests, just make sure you enable Private Google Access or it won’t deploy)

[Marvin_Lucero]

Hi @tristolivi ,

Is there any guide or documentation that you used when you mentioned :

@tristolivi

Since we use Flexible, we had to go to another solution, using the network settings instead, specifying the network and subnet, as well as specifying the ‘internal’ IP mode, as Cloud NAT does not support services with ephemeral external adresses.

If there is, can you share any link/s of it?

You are seeing the error below since this is how Cloud NAT and PGA work with one another. You can check this documentation for more details.

@tristolivi

Error Response: [9] Flex operation projects/PROJECT_NAME/regions/REGION/operations/OPERATION_ID error [FAILED_PRECONDITION]: An internal error occurred while processing task /app-engine-flex/insert_flex_deployment/flex_create_resources>RESSOURCE_ID: Subnetwork ‘SUBNET_NAME’ does not have Private Google Access enabled. App Engine flexible instances using internal-only IP addresses require that Private Google Access be enabled in the target subnetwork. Have a Network Admin enable Private Google Access as described in: https://cloud.google.com/vpc/docs/configure-private-google-access#enabling-pga

[anonymous]

Hi Marvin,

As far as documentation goes; this is the reference I used for the following parts:

Flex not supporting Serverless VPC Access:

https://cloud.google.com/vpc/docs/serverless-vpc-access#supported_services

Using ‘internal’ IP mode for Flex:

https://cloud.google.com/appengine/docs/flexible/reference/app-yaml?tab=node.js#network_settings

Cloud NAT not “useful” when App Engine has an external IP:

https://cloud.google.com/nat/docs/overview#specifications

Which helped me figure that deploying the Flex instance in ‘internal’ IP mode, inside a subnet which was configured with a Cloud Router and Cloud NAT (in turn configured with a static IP address for the subnet), would allow my Flex instance to communicate with the outside world using the static IP (and indeed it works).

I have taken a look at your documentation but I’m still unsure on why it would prevent deployment. As per the documentation you shared: “Instead, Google Cloud automatically enables Private Google Access for a subnet IP address range when you configure a Cloud NAT gateway to apply to that subnet range, either primary or secondary.” So shouldn’t deployment be aware that even though PGA is not explicitly enabled, it is “active” per the Cloud NAT for the subnet it is being deployed to? (Note that I’m aware I might not be making any sense )

Thanks for taking the time!

[Marvin_Lucero]

Hi @tristolivi ,

Thank you for your reply!

I understand where you’re coming from. Your setup is working as intended, it’s just that, the reminder or information popping out just like what is shown below should be changed or edited as it causes confusion if Cloud NAT is already enabled and PGA must be toggled to ON manually.

I happen to check a related an internal bug where it was reported to be fixed, but it seems to be showing again. You may file a bug by accessing this link. There is no ETA for the fix but it will be worked on by our engineers.