I enabled Netapp api. Then when I went to create my first storage-pool and via the GUI some first time network work was done by walking through the GUI, which was: Allocating a IP range for a private network, and creating private services access connection for peering. When that was done and the volume was working here’s how networking looked:
A few weeks later I installed BACKUP AND DR which also required Allocating a IP range for a private network, and creating private services access connection for peering like as with NETAPP above. During the networking setup for Backup, I choose to use the network items that were created for NetApp above, since these were being offered I assume it would be OK. But in hindsight if you are setting up anything new that requires these network items I think it might be best to allocate a new IP range and peering for that application. Here is how networking looks after installing BACKUP AND DR:
Notice in picture below that subnet mask has changed
from 172.24.224.0/28 and 172.24.224.0/28 to 172.24.224.0/28 and 172.24.224.64/26 <-Note this change did not occur until the Backup appliance VM was deployed during the BACKUP install process and this is what broke routing to the Netapp volume IP address!
SOLUTION
To fix the fact that the Netapp volume IP addresses is no longer pingable from our VM:
I disabled the BACKUP api, then I deleted the servicenetworking-googleapi-com VPC NETWORK PEERING that was created during BACKUP install. I did notice that deleting the servicenetworking-googleapi-com VPC NETWORK PEERING also had the effect of removing its corresponding PRIVATE CONNECTIONS TO SERVICES.
After those were removed the netapp vol was working again, and we could ping the volume IP address and the routes looked like before.