Cloud SQL Private IP Route Not Created Automatically (Private Services Access / VPC Peering)

Google Cloud Compute Infrastructure
, ,
1

Hello community,

I’m dealing with a persistent and very strange issue when trying to set up a private IP for a Cloud SQL instance. Although the instance is assigned an internal IP address from the allocated range, the necessary route in my VPC’s route table is never created automatically. This makes the private IP completely unreachable.

Symptoms:

  • The Cloud SQL instance shows a correctly assigned “Internal IP address”.

  • In VPC Network > Routes, the expected route with the next hop servicenetworking-googleapis-com is completely missing.

  • Any connection attempt to the private IP from a VM in the same VPC results in a Connection timed out.

Troubleshooting steps I’ve already taken without success:

  • I have verified that the Service Networking API is enabled.

  • I have correctly configured Private Services Access (PSA), allocating an IP range and creating the private connection.

  • I have forced the recreation of the private services connection (by deleting and recreating it) multiple times.

  • I have completely deleted the project and started over with a new one, but the issue persisted in the exact same way.

  • I have reviewed the active Organization Policies; none seem related (Restrict VPC Peering Usage is inactive).

  • I have attempted to create the instance in different regions, using the pre-created subnets of the default network, with the same result.

When trying to delete the connection via the gcloud CLI, I receive the following error, which points to a “ghost” resource locking the connection:

ERROR: (gcloud.services.vpc-peerings.delete) The operation “…” resulted in a failure “Failed to delete connection; Producer services (e.g. CloudSQL, Cloud Memstore, etc.) are still using this connection. Help Token: AeNz4PiJ4lKkNOgYOw-DoV8V_7ZEew1TVjdbNeALH1NKk86iG-ha0XvHcZZk-lr7JbELumkGTkLAUrTLKlQVTBVwqPvHf-cszuqDU1Hf–gw5mBl”.

My questions to the community are:

  1. Has anyone encountered this “ghost” issue before, where the route fails to create despite a seemingly correct configuration?

  2. Are there any other diagnostic steps or workarounds I could try?

I appreciate any help or guidance you can offer in advance.

2

Hi @jmsaavedra,

I can see that you have tried different troubleshooting steps to resolve your issue, sharing with you this Public Issue Tracker that highlights the same issue you’re encountering.

As per Google staff:

When you create a new Cloud SQL instance with a private IP (or any new resource that uses a private services connection), GCP performs the following steps. If a step has already been performed (perhaps due to a similar resource having been created), then it skips that step and continues with the process:

  1. Create a VPC network in the producer’s project.
  2. Establish a VPC Network Peering from the producer’s VPC network to your VPC network.
  3. Establish a VPC Network Peering from your VPC network to the producer’s VPC network.
  4. Create a subnet in the producer’s VPC network such that the subnet uses a CIDR that comes from the allocated range.
  5. Create the resource in that subnet.

The peering route in your VPC network (that corresponds to the subnet route in the producer’s VPC network) is only created once the fourth step has finished. So it’s perfectly normal to not see a peering route for a subnet even if there’s an active VPC network peering. Note also that steps 2, 3, and 4 present a bit of a simplified view of the order of operations, but they work for this reply.

  • The purpose of the allocated range is to serve as a “block” to creation of subnet routes or custom routes in your VPC network. Other portions of the allocated range can be used by other subnets created in the producer’s VPC network or even in another VPC network from the same producer. This is why we require at least a /20 for the allocated range (with a /16 being preferred). All subnets of all producer VPC networks from the same producer and peered with your network use IP addresses from the allocated range for the private connection. Back to your example, 10.10.0.0/16 is the set of IP addresses that Google (in its capacity as a producer) can use for offering services (like Cloud SQL) via private services access.

Additionally, for your references, you may check:

Also, based on the documentation for Cloud SQL, Cloud Shell doesn’t work with a private IP address.

Private services access configuration for VPC network involves allocating an IP address range and creating a private connection to the Google Cloud VPC network where your Cloud SQL instance resides. You may also verify that there are no conflicting routes in your VPC network that might interfere with the peering. List the routes using: gcloud compute routes list

3

Thank you for the detailed explanation and the link to the Public Issue Tracker. This confirms my suspicion that the issue is on the backend. The 5-step process you shared is especially helpful, as it indicates the process is failing at Step 4 on the producer’s side. Since I am on the Basic support plan, I cannot open a case with my Help Token. Is there any other way to escalate this backend issue?

4

Hi @jmsaavedra,

You may also use the Public Issue tracker to escalate your issue.

Describe the backend issue, including the specific GCP service. Include error messages and logs. Be as detailed as possible and avoid sharing sensitive information.

Specify the impact and you may also mention you’re on the Basic Support plan to clarify your support level.