GCP keeps using the default compute SA (which was deleted by error and replaced)

Google Cloud Serverless Applications
1

Hi, Terraform sends a POST API call to create GKE clusters it gets and error:

Error: googleapi: Error 404: Generic::not_found: Not found; Gaia id not found for email xxxx-compute@developer.gserviceaccount.com., notFound
│
│ with module.gke_cluster.google_container_cluster.primary,
│ on modules/gke_cluster/main.tf line 2, in resource “google_container_cluster” “primary”:
│ 2: resource “google_container_cluster” “primary” {

This SA was deleted by error two years ago and replaced by another SA using:
gcloud alpha compute project-info set-default-service-account --service-account compute-sa@tufin-securecloud-dev.iam.gserviceaccount.com

For some reason old deleted service account still being used in a sporadic way. Every day we have this error during deployments of part of our clusters, but not al of them.

How can we completely remove any sign for the old SA?

Thanks

2

Make a Google Group with the name gke-security-groups@yourdomain.com in your domain. The name of the group must exactly be gke-security-groups. Ensure that the “View Members” permission for “Group Members” is granted to the gke-security-groups group.

Here is a documentation that might help you with the issue, and you can also review this GitHub repository with the same issue you are getting.

Additionally, you can also review this Terraform - Google Service Account. As stated, any IAM roles that a service account previously had must be reapplied if it is deleted and then recreated.

3

Hi, thanks for the reply. I’ve tried creating the group according to the docs, but this didn’t help. Actually the issue getting worse. While deployments for few clusters were successful, now none of them is getting deployed. When Terraform sends the request it gets the following error:

{
  "error": {
    "code": 404,
    "message": "Generic::not_found: Not found; Gaia id not found for email 824745837848-compute@developer.gserviceaccount.com.",
    "errors": [
      {
        "message": "Generic::not_found: Not found; Gaia id not found for email 824745837848-compute@developer.gserviceaccount.com.",
        "domain": "global",
        "reason": "notFound"
      }
    ],
    "status": "NOT_FOUND"
  }
}: timestamp=2022-10-02T05:40:22.770Z

The happens on the first request to create the cluster. Not seem to be related to cluster RBAC.
As stated, “824745837848-compute@developer.gserviceaccount.com” was deleted two years ago by error and was replaced by another SA using google alpha command: gcloud alpha compute project-info set-default-service-account --service-account compute-sa@tufin-securecloud-dev.iam.gserviceaccount.com

For some reason google still picks this deleted default compute SA instead of using the cloudbuild SA that actually deploys the clusters.

Please see attached debug:

---[ REQUEST ]---------------------------------------
POST /v1beta1/projects/tufin-securecloud-dev/locations/europe-west3-b/clusters?alt=json&prettyPrint=false HTTP/1.1
Host: container.googleapis.com
User-Agent: google-api-go-client/0.5 Terraform/1.2.3 (+https://www.terraform.io) Terraform-Plugin-SDK/2.10.1 terraform-provider-google-beta/dev
Content-Length: 1161
Content-Type: application/json
X-Goog-Api-Client: gl-go/1.16.14 gdcl/0.82.0
Accept-Encoding: gzip

{
 "cluster": {
  "addonsConfig": {
   "httpLoadBalancing": {
    "disabled": true
   },
   "istioConfig": {
    "auth": "AUTH_MUTUAL_TLS",
    "disabled": true
   }
  },
  "authenticatorGroupsConfig": {
   "enabled": true,
   "securityGroup": "gke-security-groups@tufin.com"
  },
  "autopilot": {
   "enabled": false
  },
  "autoscaling": {
   "enableNodeAutoprovisioning": false
  },
  "binaryAuthorization": {
   "enabled": false
  },
  "initialClusterVersion": "1.23.8-gke.1900",
  "initialNodeCount": 1,
  "ipAllocationPolicy": {
   "clusterIpv4CidrBlock": "/18",
   "servicesIpv4CidrBlock": "/24",
   "useIpAliases": true
  },
  "legacyAbac": {
   "enabled": false
  },
  "maintenancePolicy": {
   "window": {}
  },
  "masterAuth": {
   "clientCertificateConfig": {}
  },
  "masterAuthorizedNetworksConfig": {},
  "name": "securecloud-cluster-zvika",
  "network": "projects/tufin-securecloud-dev/global/networks/securecloud-development-vpc",
  "networkConfig": {},
  "networkPolicy": {
   "enabled": true,
   "provider": "CALICO"
  },
  "nodeConfig": {
   "machineType": "e2-micro",
   "preemptible": true,
   "serviceAccount": "securecloud-cluster@tufin-securecloud-dev.iam.gserviceaccount.com"
  },
  "notificationConfig": {
   "pubsub": {}
  },
  "shieldedNodes": {
   "enabled": false
  },
  "subnetwork": "projects/tufin-securecloud-dev/regions/europe-west3/subnetworks/securecloud-zvika-subnet"
 }
}

-----------------------------------------------------: timestamp=2022-10-02T05:40:20.567Z
2022-10-02T05:40:22.770Z [INFO]  provider.terraform-provider-google-beta_v4.27.0_x5: 2022/10/02 05:40:22 [DEBUG] Google API Response Details:
---[ RESPONSE ]--------------------------------------
HTTP/2.0 404 Not Found
Cache-Control: private
Content-Type: application/json; charset=UTF-8
Date: Sun, 02 Oct 2022 05:40:22 GMT
Server: ESF
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

{
  "error": {
    "code": 404,
    "message": "Generic::not_found: Not found; Gaia id not found for email 824745837848-compute@developer.gserviceaccount.com.",
    "errors": [
      {
        "message": "Generic::not_found: Not found; Gaia id not found for email 824745837848-compute@developer.gserviceaccount.com.",
        "domain": "global",
        "reason": "notFound"
      }
    ],
    "status": "NOT_FOUND"
  }
}

-----------------------------------------------------: timestamp=2022-10-02T05:40:22.770Z
2022-10-02T05:40:22.770Z [INFO]  provider.terraform-provider-google-beta_v4.27.0_x5: 2022/10/02 05:40:22 [DEBUG] Retry Transport: Stopping retries, last request failed with non-retryable error: googleapi: got HTTP response code 404 with body: HTTP/2.0 404 Not Found
Cache-Control: private
Content-Type: application/json; charset=UTF-8
Date: Sun, 02 Oct 2022 05:40:22 GMT
Server: ESF
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

{
  "error": {
    "code": 404,
    "message": "Generic::not_found: Not found; Gaia id not found for email 824745837848-compute@developer.gserviceaccount.com.",
    "errors": [
      {
        "message": "Generic::not_found: Not found; Gaia id not found for email 824745837848-compute@developer.gserviceaccount.com.",
        "domain": "global",
        "reason": "notFound"
      }
    ],
    "status": "NOT_FOUND"
  }
}: timestamp=2022-10-02T05:40:22.770Z
2022-10-02T05:40:22.770Z [INFO]  provider.terraform-provider-google-beta_v4.27.0_x5: 2022/10/02 05:40:22 [DEBUG] Retry Transport: Returning after 1 attempts: timestamp=2022-10-02T05:40:22.770Z
2022-10-02T05:40:22.770Z [INFO]  provider.terraform-provider-google-beta_v4.27.0_x5: 2022/10/02 05:40:22 [DEBUG] Unlocking "google-container-cluster/tufin-securecloud-dev/europe-west3-b/securecloud-cluster-zvika": timestamp=2022-10-02T05:40:22.770Z
2022-10-02T05:40:22.770Z [INFO]  provider.terraform-provider-google-beta_v4.27.0_x5: 2022/10/02 05:40:22 [DEBUG] Unlocked "google-container-cluster/tufin-securecloud-dev/europe-west3-b/securecloud-cluster-zvika": timestamp=2022-10-02T05:40:22.770Z
2022-10-02T05:40:22.773Z [ERROR] vertex "module.gke_cluster.google_container_cluster.primary" error: googleapi: Error 404: Generic::not_found: Not found; Gaia id not found for email 824745837848-compute@developer.gserviceaccount.com., notFound
╷
│ Error: googleapi: Error 404: Generic::not_found: Not found; Gaia id not found for email 824745837848-compute@developer.gserviceaccount.com., notFound
│ 
│   with module.gke_cluster.google_container_cluster.primary,
│   on modules/gke_cluster/main.tf line 2, in resource "google_container_cluster" "primary":
│    2: resource "google_container_cluster" "primary" {
│ 
╵
2022-10-02T05:40:23.446Z [DEBUG] provider.terraform-provider-google-beta_v4.27.0_x5: 2022/10/02 05:40:23 [DEBUG] [core] [Server #1 ListenSocket #2] ListenSocket deleted 
2022-10-02T05:40:23.446Z [DEBUG] provider.terraform-provider-google-beta_v4.27.0_x5: 2022/10/02 05:40:23 [DEBUG] [transport] transport: http2Server.HandleStreams failed to read frame: read unix /tmp/plugin067716049->@: use of closed network connection 
2022-10-02T05:40:23.447Z [DEBUG] provider.terraform-provider-google_v4.27.0_x5: 2022/10/02 05:40:23 [DEBUG] [core] [Server #1 ListenSocket #2] ListenSocket deleted 
2022-10-02T05:40:23.447Z [DEBUG] provider.terraform-provider-google_v4.27.0_x5: 2022/10/02 05:40:23 [DEBUG] [transport] transport: http2Server.HandleStreams failed to read frame: read unix /tmp/plugin662543252->@: use of closed network connection 
2022-10-02T05:40:23.447Z [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/google/4.27.0/linux_amd64/terraform-provider-google_v4.27.0_x5 pid=1027
2022-10-02T05:40:23.447Z [DEBUG] provider: plugin exited
2022-10-02T05:40:23.447Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2022-10-02T05:40:23.447Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2022-10-02T05:40:23.447Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2022-10-02T05:40:23.447Z [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/google-beta/4.27.0/linux_amd64/terraform-provider-google-beta_v4.27.0_x5 pid=1018
2022-10-02T05:40:23.447Z [DEBUG] provider: plugin exited
2022-10-02T05:40:23.447Z [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/null/3.1.1/linux_amd64/terraform-provider-null_v3.1.1_x5 pid=1004
2022-10-02T05:40:23.447Z [DEBUG] provider: plugin exited
terraformApply failed with exit code 1
4

The original reply may be unrelated to your problem.

Can you try using binary authorization with evaluation mode to disable it, instead of leaving it default / false.

binary_authorization {
  evaluation_mode = "DISABLED"
}

I’ve had some success with this in my testing for a relevant problem.

5

Hi, thanks for replying. Indeed the original reply wasn’t related to my problem. So at some point we’ve decide to move to aa new project. It was a quite a painful move, but we’re at the end now. Thanks anyway.