Issue with "Couldn't find a free block in allocated IP ranges" when scheduling a notebook pipeline

Google Cloud Compute Infrastructure
1

Hello,
I am encountering an issue when trying to schedule a pipeline job from Google Cloud Workbench that runs a notebook in the us-central1 region within my default VPC network. The error I am receiving is:

​For the peered network projects/236570057022/global/networks/default, couldn’t find a free block in allocated IP ranges. This is needed to create the cluster. Please allocate new ranges for the peered network or wait for existing jobs, using the allocated range to finish.

Currently, in my network VPC, I have Private Services Access (10.66.0.240/20) using servicenetworking-googleapis-com. I also added it to Private Connections to Services, assigned both locations the default-ip-range and the private-ip-range-extra (the one I created). I created it automatically with the prefix /20 below /24 to have plenty of IPs. However, I can’t get the new IPs to be assigned correctly when creating the notebook scheduling.
I also tried creating a subnet with a range of IPs within the subnets tab of the VPC Network, but that didn’t work. I also tried adding a range of Secondary IPv4s to the region I want, us-central1, but it still doesn’t work.
I’ve had conversations with Google “experts” (and I put “experts” in quotes because the people who helped us didn’t know what to do and always keep sending things to other real experts, and they take forever). In this case, we’ve been in my company for over 2 months with endless conversations and people who don’t know what’s going on.
I need your help since I don’t really know what’s going on either. Although I like networking, I understand that it should be added this way, and when I create the instance for the scheduling, it should assign the IP correctly, but I don’t see that it’s working.
Questions:

  • Do I have to create a separate VPC for this?

  • Can’t I add more IPs to use scheduling in the Default VPC?

  • Is there a problem with Workbench?

  • How can it be that three instances of Workbench, which I have running in that region, are taking up the entire IP range?

  • How can I expand my IP range without affecting existing services?

  • Is there a limit on the number of IP ranges I can use in my network for scheduled jobs?

  • This worked before October of last year, but it stopped working then, and it has to be done manually now, which is difficult.

I feel frustrated, thank you very much for reading me and for your time.

2

Hi @DaGoGi ,

Welcome to Google Cloud Community!

You might see this error because the allocated range is not sufficient for your usage, or because a custom static or dynamic route is preventing the allocated range from being fully used. Each service producer requires a minimum IP address range size. For Google, the minimum size is a single /24 block (256 addresses), but the recommended size is a /16 block (65,536 addresses).

If you don’t have a contiguous /16 block, you can start with a smaller allocation and add new ones if you need more IP addresses later.

Answering Your Questions:

Do I have to create a separate VPC for this?

While technically possible, we don’t recommend creating a separate VPC as it complicates things with peering and extra rules. The standard solution is to properly configure Private Services Access (PSA) and manage its allocated IP ranges within your existing default VPC

Can’t I add more IPs to use scheduling in the Default VPC?

Yes, absolutely. You do this by adding more allocated IP ranges to the allocated range to the existing private connection.

Is there a problem with Workbench?

The issue isn’t with Workbench itself, which merely starts the pipeline. The IP address shortage occurs later in the process when Vertex AI attempts to secure resources from the PSA ranges to actually execute the pipeline job.

How can it be that three instances of Workbench… are taking up the entire IP range?

Workbench instances rely on subnet IPs. The PSA IP range depletion is due to concurrent Vertex AI Pipeline jobs consuming those dedicated addresses.

How can I expand my IP range without affecting existing services?

To safely increase the number of IPs available for Private Services Access (PSA), simply add new IP ranges to your servicenetworking-googleapis-com connection. Make sure these new ranges are unique and don’t overlap with any existing VPC subnets or other PSA ranges. Adding ranges won’t cause disruptions, and services will automatically start using the expanded pool. However, it’s critical not to change or remove any existing PSA range that’s currently being used, as that could break connected services.

Is there a limit on the number of IP ranges I can use in my network for scheduled jobs?

The Vertex AI Workbench service itself doesn’t directly restrict the quantity of distinct IP ranges your notebook code can work with or connect to.

This worked before October…

Since this issue only appeared after October, something likely shifted. Possible explanations include an increase in the number of concurrent pipeline jobs, jobs taking longer to run (which ties up IPs longer), or that the initial /20 range, while adequate previously, is just too small for your current usage patterns.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.