What is Managed kafka in GCP under the hood?

Hi @dheerajpanyam ,

Welcome to Google Cloud Community!

Here’s the Network architecture for Google Cloud Managed Service for Apache Kafka:

Flexible networking

The service makes a cluster accessible from any VPC securely. This includes access from multiple VPCs, projects, and regions.

To configure networking for a cluster, you provide the set of subnets where the cluster is accessible. The service provisions private IP addresses for the bootstrap servers and brokers in each subnet. It also sets up private Cloud DNS with URLs for each IP address. The bootstrap servers have a load balancer, so there is a single bootstrap URL per cluster. The URLs are the same across all VPCs so client configurations can be consistent across environments.

This level of flexibility is achieved thanks to Private Service Connect (PSC). Each IP address allocated for a cluster requires a PSC endpoint. The endpoints are provisioned automatically.

Note that Google Cloud Managed Service for Apache Kafka cluster in GCP is under Pre-GA.

Other relevant references for Google Cloud Managed Service for Apache Kafka can be found here:

I hope this helps!

2 Likes