Mastering fleet-wide governance on GKE using fleet level OPA

Google Cloud Community Articles
, ,
1

Maintaining security, compliance, and operational standards across multiple Kubernetes clusters is one of the most significant challenges platform engineering teams face today. Without proper guardrails, managing configurations across a growing fleet of clusters can quickly lead to security vulnerabilities, configuration drift, and bloated cloud bills.

In this blog post, we will explore how you can establish centralized governance across your Google Kubernetes Engine (GKE) clusters using GKE Policy Controller and compare it with the open-source OPA Gatekeeper project.

Centralized governance: Fleet-level OPA setup

Manually installing and managing policy engines on a cluster-by-cluster basis is highly inefficient and error-prone. To solve this, Google Cloud allows you to define policy configurations at the fleet level.

Once fleet-level defaults are enabled, any new GKE cluster registered to the fleet automatically inherits:

  • Automatic Installation: Policy Controller is deployed automatically on registration.

  • Version Management: The fleet centrally manages the controller version.

  • Global Settings: Global settings such as audit intervals, exempted namespaces, and referential constraints are consistently applied.

Managed policy bundles

Google Cloud provides ready-to-use Policy Bundles mapped to industry standards. You can apply these to all member clusters simultaneously to achieve instant compliance:

  • CIS Kubernetes Benchmark: Align cluster settings with the Center for Internet Security standards.

  • Regulated Industries (PCI-DSS / NIST / HIPAA): Compliance-specific bundles designed to meet rigorous regulatory controls.

  • Cost & Reliability: A bundle that proactively flags wasteful resource configurations or missing high-availability setups (like missing PodDisruptionBudgets).

Fleet dashboard & observability

Within the GKE Enterprise console, you get a unified dashboard offering complete visibility:

  • Violation Summaries: Identify which clusters have the highest compliance risk.

  • Security Command Center (SCC) Integration: Violations are pushed to SCC as “findings,” enabling security operations teams to track Kubernetes risks alongside general cloud infrastructure.

[image]

GKE Policy Controller vs. OSS OPA Gatekeeper

While GKE Policy Controller is built on the open-source OPA Gatekeeper engine, Google Cloud adds significant enterprise management capabilities:
Feature
OSS OPA Gatekeeper
GKE Policy Controller (Fleet)
Management
Manual (per cluster)
Centralized (Fleet-wide)
Bundles
Community-maintained
Google-maintained & updated
Installation
Helm / Kubectl
One-click or CLI Fleet enablement
Observability
Prometheus / Grafana
GCP Console + Security Command Center
Support
Community
Google Enterprise Support

Getting started: Enabling Fleet-level policies

Enabling Policy Controller across your GKE fleet is straightforward. Using the gcloud CLI, you can declare your configuration in a YAML file and roll it out instantly.

Step 1: Create a Fleet default configuration file

Create a file named fleet-default.yaml to specify the default configuration and any compliance bundles you wish to pre-install:

policyControllerHubConfig:

installSpec: INSTALL_SPEC_ENABLED

policyContent:

bundles:

  cis-k8s-v1.5.1:

    exemptedNamespaces: \["kube-system"\]

Note: The CLI commands above are provided for explanation, testing, and experiment purposes only. In production environments, policy configurations should be managed using GitOps tools like Config Sync.

Step 2: Apply the configuration to the Fleet

Run the following CLI command to enable Policy Controller using your default configuration:

gcloud container fleet policycontroller enable \

--fleet-default-member-config=fleet-default.yaml

Step 3: Verify the deployment status

You can describe the fleet state to verify that the policies have been successfully rolled out and are active across all member clusters:

gcloud container fleet policycontroller describe

10 Real-world guardrail scenarios

Here are ten common scenarios where Policy Controller helps you maintain strict control over GKE workloads:

1. Enforce organizational labeling

  • What it does: Requires every Namespace or Deployment to have specific labels (e.g., team-id, cost-center, or env).

  • Why it matters: Essential for accurate cost attribution, department chargebacks, and owner tracing in multi-tenant environments.

2. Restrict ingress domains

  • What it does: Restricts Ingress resources to only expose services on pre-approved corporate domains (e.g., hostnames ending in *.yourcompany.com).

  • Why it matters: Prevents “shadow IT” and accidental public exposure of internal services on arbitrary or unmapped domains.

3. Image registry whitelisting

  • What it does: Blocks any Pod from starting if its container image comes from an untrusted public registry.

  • Why it matters: Forces teams to pull verified, scanned container images strictly from your private Artifact Registry.

4. Mandatory resource limits

  • What it does: Denies Pods that do not explicitly define CPU/memory limits and requests.

  • Why it matters: Prevents “noisy neighbor” issues where a single unconstrained container consumes all node resources, risking cluster stability.

5. Prevent privilege escalation

  • What it does: Blocks containers configured with privileged: true, direct hostPath mounts, or running as root.

  • Why it matters: Eliminates critical container escape risks that could allow a compromised pod to take control of the host GKE node.

6. Ban specific image tags

  • What it does: Enforces version pinning by rejecting volatile image tags like :test or :latest.

  • Why it matters: Ensures deployments are predictable, repeatable, and easily rolled back by enforcing immutable digest or semantic tags.

7. Limit service types (No public LoadBalancers)

  • What it does: Prevents the creation of Services with type: LoadBalancer, steering teams toward secure internal Ingress pathways.

  • Why it matters: Avoids unnecessary external IP costs and secures the cluster’s network perimeter from direct, unauthorized public ingress.

8. Enforce network policies

  • What it does: Requires every new Namespace to contain a default “deny-all” NetworkPolicy object on creation.

  • Why it matters: Establishes a zero-trust network posture inside the cluster right from the start.

9. Prohibit NodePort Services

  • What it does: Disallows Kubernetes Services from exposing node-level ports (type: NodePort).

  • Why it matters: Eliminates security gaps by preventing arbitrary, high-numbered ports from being opened on GKE worker nodes.

10. Service port naming for observability

  • What it does: Mandates that Service port names follow standard conventions (e.g., starting with http- or grpc-).

  • Why it matters: Ensures service meshes (such as Istio/Cloud Service Mesh) and monitoring tools can automatically discover and trace protocol-specific traffic.

Step-by-step: Testing a new constraint

Let’s walk through how to deploy and test a policy that prevents the use of unapproved public container registries.

Step 1: Verify the constraint template exists

Policy Controller comes pre-installed with a large template library. Verify that the K8sAllowedRepos template is available in your cluster:

kubectl get constrainttemplates k8sallowedrepos

Step 2: Define your constraint configuration

Create a file named deny-public-images.yaml. This file configures the K8sAllowedRepos template to watch Pod resources and only allow images originating from your private Google Artifact Registry path:

apiVersion: constraints.gatekeeper.sh/v1beta1

kind: K8sAllowedRepos

metadata:

name: allow-only-internal-registries

spec:

enforcementAction: deny

match:

kinds:

  - apiGroups: \[""\]

    kinds: \["Pod"\]

excludedNamespaces:

  - kube-system

  - gatekeeper-system

parameters:

repos:

  - "gcr.io/your-project-id/"

  - "us-central1-docker.pkg.dev/your-project-id/"

Step 3: Apply the constraint

For a single cluster test, apply the file directly via kubectl:

kubectl apply -f deny-public-images.yaml

(Note: In a production environment, it is best practice to commit this file to a Git repository managed centrally by Config Sync to synchronize policies across all GKE clusters automatically).

Step 4: Test the constraint

Attempt to deploy a Pod using a public image from Docker Hub (which is outside your allowlist):

kubectl run test-public-image --image=nginx:test-tag

Expected result

The cluster’s admission controller should reject your request with a descriptive error message:

Error from server (Forbidden): admission webhook “validation.gatekeeper.sh” denied the request: [allow-only-internal-registries] container has an invalid image repo nginx:test-tag, allowed repos are [“gcr.io/your-project-id/”, …]

[image]

Note: The direct kubectl commands and manual constraint setups shown above are intended for explanation, testing, and experiment purposes only.

Managing the impact of constraints on running workloads

Because Policy Controller acts as an Admission Controller, it acts as a “bouncer” at the door of your cluster. It evaluates requests during the creation or update phase, meaning it will not retroactively terminate pods that are already running.

Here is how a newly applied constraint affects existing workloads:

1. Immediate impact: “Audit” mode

Since the Pod is already running, it continues to run. However, the background Audit engine flags the workload as a violation:

  • You will see the violation reported under GKE > Posture Management > Policy in the Google Cloud Console.

  • This allows you to identify non-compliant workloads without causing production outages.

Pro Tip: When launching a new constraint, set enforcementAction: dryrun first. This lets you observe violations in the dashboard before switching to deny to block deployments.

2. Future impact: Admission blocking

The next time the workload is modified or restarted, the policy will intercept and block the deployment in the following cases:

  • Node Restarts / Pod Crashes: If a node goes down or a pod crashes, Kubernetes will try to recreate the Pod. The Policy Controller will block recreation if the pod does not meet the policy.

  • Updates: Modifying labels, environment variables, or other deployment specifications will fail if the underlying Pod template violates the policy.

  • Autoscaling: GKE’s Cluster Autoscaling or Horizontal Pod Autoscaling (HPA) will fail to spin up new pods if the templates are non-compliant.

3. Checking active violations from CLI

You can view a list of active violations directly using kubectl:

kubectl get k8sallowedrepos allow-only-internal-registries -o yaml

Under the status.violations section of the output, you will find the namespace and name of every non-compliant Pod currently running.

Scaling policy management with Config Sync

While manual setup via direct CLI commands works for testing, scaling policy management across an entire enterprise requires a declarative, version-controlled approach. Organizations leverage Config Sync to implement GitOps-driven, fleet-wide policy management.

By storing constraints and constraint templates in a central Git repository, Config Sync continuously monitors and automatically reconciles configurations across all member clusters in the fleet. This ensures that policies remain consistent, auditable through standard git workflows, and resistant to configuration drift across dev, staging, and production environments.

Conclusion

Standardizing policy enforcement across your GKE environments is key to achieving robust security and cost-efficiency at scale. By leveraging GKE Policy Controller, platform teams can easily automate enforcement of industry-standard security frameworks across their entire GKE Fleet.

Further reading

4 Likes