Failed to create Google Anthos on Bare metal "hybrid" cluster

Google Cloud Serverless Applications
1

I am trying to create Google Anthos on Bare metal “hybrid” cluster. It is showing following error. ANy help would be appreciated.

“msg”=“Operation failed, retrying with backoff” “Cause”=“pod "cert-manager-96d7b5768-twhml" is not ready”

Detailed Logs as follows:

mdrahman@bm-wkst:~$ cd baremetal/bmctl-workspace/bm-cluster1/log/create-cluster-20210803-102743/
mdrahman@bm-wkst:~/baremetal/bmctl-workspace/bm-cluster1/log/create-cluster-20210803-102743$ ls
create-cluster.log
mdrahman@bm-wkst:~/baremetal/bmctl-workspace/bm-cluster1/log/create-cluster-20210803-102743$ cat create-cluster.log
Log file created at: 2021/08/03 10:27:43
Running on machine: bm-wkst
Binary: Built with gc go1.15.13 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0803 10:27:43.742998 150017 logs.go:57] “msg”=“Start Extracting manifests files to .manifests”
I0803 10:27:52.267932 150017 logs.go:57] “msg”=“End Extracting manifests files to .manifests”
I0803 10:27:52.271244 150017 validate.go:85] Validating Cluster.Authentication authentication:
I0803 10:27:52.271299 150017 file_check.go:20] [Runtime Check] Start Check for file --sshPrivateKeyPath=/home/mdrahman/.ssh/id_rsa.
I0803 10:27:52.271309 150017 file_check.go:24] [Runtime Check] End Check for file --sshPrivateKeyPath=/home/mdrahman/.ssh/id_rsa.
I0803 10:27:52.271311 150017 file_check.go:20] [Runtime Check] Start Check for file --gkeConnectAgentServiceAccountKeyPath=bmctl-workspace/.sa-keys/encqor-ilab-03-anthos-baremetal-connect.json.
I0803 10:27:52.271319 150017 file_check.go:24] [Runtime Check] End Check for file --gkeConnectAgentServiceAccountKeyPath=bmctl-workspace/.sa-keys/encqor-ilab-03-anthos-baremetal-connect.json.
I0803 10:27:52.271322 150017 file_check.go:20] [Runtime Check] Start Check for file --gkeConnectRegisterServiceAccountKeyPath=bmctl-workspace/.sa-keys/encqor-ilab-03-anthos-baremetal-register.json.
I0803 10:27:52.271326 150017 file_check.go:24] [Runtime Check] End Check for file --gkeConnectRegisterServiceAccountKeyPath=bmctl-workspace/.sa-keys/encqor-ilab-03-anthos-baremetal-register.json.
I0803 10:27:52.271327 150017 file_check.go:20] [Runtime Check] Start Check for file --gcrKeyPath=bmctl-workspace/.sa-keys/encqor-ilab-03-anthos-baremetal-gcr.json.
I0803 10:27:52.271331 150017 file_check.go:24] [Runtime Check] End Check for file --gcrKeyPath=bmctl-workspace/.sa-keys/encqor-ilab-03-anthos-baremetal-gcr.json.
I0803 10:27:52.271333 150017 file_check.go:20] [Runtime Check] Start Check for file --cloudOperationsServiceAccountKeyPath=bmctl-workspace/.sa-keys/encqor-ilab-03-anthos-baremetal-cloud-ops.json.
I0803 10:27:52.271336 150017 file_check.go:24] [Runtime Check] End Check for file --cloudOperationsServiceAccountKeyPath=bmctl-workspace/.sa-keys/encqor-ilab-03-anthos-baremetal-cloud-ops.json.
I0803 10:27:52.271339 150017 shell_cmd_check.go:20] [Runtime Check] Start Environment check for command docker.
I0803 10:27:52.271357 150017 shell_cmd_check.go:24] [Runtime Check] End Environment check for command docker.
I0803 10:27:52.271359 150017 docker.go:52] [Runtime Check] Start Check docker version.
I0803 10:27:52.336689 150017 docker.go:67] [Runtime Check] Start Check if user is in right docker group.
I0803 10:27:52.337418 150017 docker.go:110] [Runtime Check] End Check if user is in right docker group.
I0803 10:27:52.834605 150017 service_account.go:67] [Runtime Check] Start: Check for service account projects/encqor-ilab-03/serviceAccounts/anthos-baremetal-register@encqor-ilab-03.iam.gserviceaccount.com
I0803 10:27:53.204691 150017 service_account.go:94] [Runtime Check] End: Check for service account projects/encqor-ilab-03/serviceAccounts/anthos-baremetal-register@encqor-ilab-03.iam.gserviceaccount.com
I0803 10:27:53.759516 150017 service_account.go:67] [Runtime Check] Start: Check for service account projects/encqor-ilab-03/serviceAccounts/anthos-baremetal-cloud-ops@encqor-ilab-03.iam.gserviceaccount.com
I0803 10:27:54.013177 150017 service_account.go:94] [Runtime Check] End: Check for service account projects/encqor-ilab-03/serviceAccounts/anthos-baremetal-cloud-ops@encqor-ilab-03.iam.gserviceaccount.com
I0803 10:27:54.013845 150017 kustomize.go:1041] Building manifest for cert-manager/base
I0803 10:27:55.258861 150017 kustomize.go:1041] Building manifest for cluster-api/base
I0803 10:27:55.541924 150017 kustomize.go:1041] Building manifest for kubeadm-bootstrap-operator/base
I0803 10:27:55.927046 150017 kustomize.go:1041] Building manifest for cluster-api-provider/overlays/bmctl
I0803 10:27:56.049647 150017 kustomize.go:1041] Building manifest for cluster-operator/overlays/bmctl
I0803 10:27:56.967201 150017 kustomize.go:1041] Building manifest for static-provisioner/overlays/bmctl
I0803 10:27:57.022249 150017 kustomize.go:1041] Building manifest for multinet/base/webhooks/overlays/bmctl
I0803 10:27:57.365011 150017 kind.go:211] Start: create a KIND cluster bmctl.
I0803 10:27:57.417418 150017 kind.go:245] Using image registry: “gcr.io/anthos-baremetal-release”
I0803 10:27:57.460536 150017 kind.go:563] Detected MTU 1500.
I0803 10:27:57.573906 150017 kind.go:541] Docker network created.
I0803 10:27:57.608748 150017 kind.go:166] Creating cluster “bmctl” …
I0803 10:27:57.608764 150017 kind.go:166] • Ensuring node image (gcr.io/anthos-baremetal-release/kindest/node:v0.10.0-gke.4-v1.20.5-gke.1301) :framed_picture: …
I0803 10:27:57.642150 150017 kind.go:166] Pulling image: gcr.io/anthos-baremetal-release/kindest/node:v0.10.0-gke.4-v1.20.5-gke.1301 …
I0803 10:28:28.409613 150017 kind.go:166] ✓ Ensuring node image (gcr.io/anthos-baremetal-release/kindest/node:v0.10.0-gke.4-v1.20.5-gke.1301) :framed_picture:
I0803 10:28:28.409630 150017 kind.go:166] • Preparing nodes :package: …
I0803 10:28:32.111063 150017 kind.go:166] ✓ Preparing nodes :package:
I0803 10:28:32.197126 150017 kind.go:166] • Writing configuration :scroll: …
I0803 10:28:32.694180 150017 kind.go:166] ✓ Writing configuration :scroll:
I0803 10:28:32.694196 150017 kind.go:166] • Starting control-plane :joystick: …
I0803 10:30:16.902898 150017 kind.go:166] ✓ Starting control-plane :joystick:
I0803 10:30:17.747612 150017 kustomize.go:1041] Building manifest for cni/base
I0803 10:30:17.948385 150017 apply.go:47] Create Kind Cluster “msg”=“customresourcedefinition.apiextensions.k8s.io/networkloggings.networking.gke.io created\n”
I0803 10:30:17.957570 150017 apply.go:47] Create Kind Cluster “msg”=“serviceaccount/cilium created\n”
I0803 10:30:17.967024 150017 apply.go:47] Create Kind Cluster “msg”=“serviceaccount/cilium-operator created\n”
I0803 10:30:17.970546 150017 apply.go:47] Create Kind Cluster “msg”=“clusterrole.rbac.authorization.k8s.io/cilium created\n”
I0803 10:30:17.975790 150017 apply.go:47] Create Kind Cluster “msg”=“clusterrole.rbac.authorization.k8s.io/cilium-operator created\n”
I0803 10:30:17.979672 150017 apply.go:47] Create Kind Cluster “msg”=“clusterrolebinding.rbac.authorization.k8s.io/cilium created\n”
I0803 10:30:17.982701 150017 apply.go:47] Create Kind Cluster “msg”=“clusterrolebinding.rbac.authorization.k8s.io/cilium-operator created\n”
I0803 10:30:17.987808 150017 apply.go:47] Create Kind Cluster “msg”=“configmap/cilium-config created\n”
I0803 10:30:17.997863 150017 apply.go:47] Create Kind Cluster “msg”=“configmap/vars-d9k2m46kd4 created\n”
I0803 10:30:18.007752 150017 apply.go:47] Create Kind Cluster “msg”=“deployment.apps/anet-operator created\n”
I0803 10:30:18.016151 150017 apply.go:47] Create Kind Cluster “msg”=“daemonset.apps/anetd created\n”
E0803 10:30:18.016162 150017 apply.go:143] Create Kind Cluster “msg”=“apply run failed” “error”=“unable to recognize "/tmp/kout226194152": no matches for kind "NetworkLogging" in version "networking.gke.io/v1alpha1"”
I0803 10:30:23.018192 150017 kustomize.go:1041] Building manifest for cni/base
I0803 10:30:23.202721 150017 apply.go:47] Create Kind Cluster “msg”=“customresourcedefinition.apiextensions.k8s.io/networkloggings.networking.gke.io unchanged\n”
I0803 10:30:23.204478 150017 apply.go:47] Create Kind Cluster “msg”=“serviceaccount/cilium unchanged\n”
I0803 10:30:23.206185 150017 apply.go:47] Create Kind Cluster “msg”=“serviceaccount/cilium-operator unchanged\n”
I0803 10:30:23.207984 150017 apply.go:47] Create Kind Cluster “msg”=“clusterrole.rbac.authorization.k8s.io/cilium unchanged\n”
I0803 10:30:23.209959 150017 apply.go:47] Create Kind Cluster “msg”=“clusterrole.rbac.authorization.k8s.io/cilium-operator unchanged\n”
I0803 10:30:23.211388 150017 apply.go:47] Create Kind Cluster “msg”=“clusterrolebinding.rbac.authorization.k8s.io/cilium unchanged\n”
I0803 10:30:23.213064 150017 apply.go:47] Create Kind Cluster “msg”=“clusterrolebinding.rbac.authorization.k8s.io/cilium-operator unchanged\n”
I0803 10:30:23.215220 150017 apply.go:47] Create Kind Cluster “msg”=“configmap/cilium-config unchanged\n”
I0803 10:30:23.219344 150017 apply.go:47] Create Kind Cluster “msg”=“configmap/vars-d9k2m46kd4 configured\n”
I0803 10:30:23.222455 150017 apply.go:47] Create Kind Cluster “msg”=“deployment.apps/anet-operator unchanged\n”
I0803 10:30:23.226948 150017 apply.go:47] Create Kind Cluster “msg”=“daemonset.apps/anetd unchanged\n”
I0803 10:30:23.264714 150017 apply.go:47] Create Kind Cluster “msg”=“networklogging.networking.gke.io/default created\n”
I0803 10:31:26.782601 150017 kind.go:424] End: create a KIND cluster bmctl.
I0803 10:31:26.785213 150017 kustomize.go:1041] Building manifest for cert-manager/base
I0803 10:31:28.321396 150017 apply.go:47] Create Kind Cluster “msg”=“namespace/cert-manager created\n”
I0803 10:31:28.345919 150017 apply.go:47] Create Kind Cluster “msg”=“customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created\n”
I0803 10:31:28.390593 150017 apply.go:47] Create Kind Cluster “msg”=“customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created\n”
I0803 10:31:28.549320 150017 apply.go:47] Create Kind Cluster “msg”=“customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created\n”
I0803 10:31:28.809173 150017 apply.go:47] Create Kind Cluster “msg”=“customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created\n”
I0803 10:31:29.028037 150017 apply.go:47] Create Kind Cluster “msg”=“customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created\n”
I0803 10:31:29.328203 150017 apply.go:47] Create Kind Cluster “msg”=“customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created\n”
I0803 10:31:29.353961 150017 apply.go:47] Create Kind Cluster “msg”=“serviceaccount/cert-manager created\n”
I0803 10:31:29.510974 150017 apply.go:47] Create Kind Cluster “msg”=“serviceaccount/cert-manager-cainjector created\n”
I0803 10:31:29.926331 150017 apply.go:47] Create Kind Cluster “msg”=“serviceaccount/cert-manager-webhook created\n”
I0803 10:31:30.134264 150017 apply.go:47] Create Kind Cluster “msg”=“role.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created\n”
I0803 10:31:30.301794 150017 apply.go:47] Create Kind Cluster “msg”=“role.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created\n”
I0803 10:31:30.311035 150017 apply.go:47] Create Kind Cluster “msg”=“role.rbac.authorization.k8s.io/cert-manager:leaderelection created\n”
I0803 10:31:30.319142 150017 apply.go:47] Create Kind Cluster “msg”=“clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector created\n”
I0803 10:31:30.334588 150017 apply.go:47] Create Kind Cluster “msg”=“clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates created\n”
I0803 10:31:30.369775 150017 apply.go:47] Create Kind Cluster “msg”=“clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges created\n”
I0803 10:31:30.386104 150017 apply.go:47] Create Kind Cluster “msg”=“clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created\n”
I0803 10:31:30.395964 150017 apply.go:47] Create Kind Cluster “msg”=“clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created\n”
I0803 10:31:30.410257 150017 apply.go:47] Create Kind Cluster “msg”=“clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers created\n”
I0803 10:31:30.421857 150017 apply.go:47] Create Kind Cluster “msg”=“clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders created\n”
I0803 10:31:30.427425 150017 apply.go:47] Create Kind Cluster “msg”=“clusterrole.rbac.authorization.k8s.io/cert-manager-edit created\n”
I0803 10:31:30.439830 150017 apply.go:47] Create Kind Cluster “msg”=“clusterrole.rbac.authorization.k8s.io/cert-manager-view created\n”
I0803 10:31:30.459323 150017 apply.go:47] Create Kind Cluster “msg”=“rolebinding.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created\n”
I0803 10:31:30.467760 150017 apply.go:47] Create Kind Cluster “msg”=“rolebinding.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created\n”
I0803 10:31:30.472360 150017 apply.go:47] Create Kind Cluster “msg”=“rolebinding.rbac.authorization.k8s.io/cert-manager:leaderelection created\n”
I0803 10:31:30.477732 150017 apply.go:47] Create Kind Cluster “msg”=“clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector created\n”
I0803 10:31:30.482646 150017 apply.go:47] Create Kind Cluster “msg”=“clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates created\n”
I0803 10:31:30.487521 150017 apply.go:47] Create Kind Cluster “msg”=“clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges created\n”
I0803 10:31:30.493084 150017 apply.go:47] Create Kind Cluster “msg”=“clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created\n”
I0803 10:31:30.497531 150017 apply.go:47] Create Kind Cluster “msg”=“clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created\n”
I0803 10:31:30.503641 150017 apply.go:47] Create Kind Cluster “msg”=“clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers created\n”
I0803 10:31:30.508921 150017 apply.go:47] Create Kind Cluster “msg”=“clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders created\n”
I0803 10:31:30.517341 150017 apply.go:47] Create Kind Cluster “msg”=“configmap/vars-d44fgcm8g8 created\n”
I0803 10:31:30.527946 150017 apply.go:47] Create Kind Cluster “msg”=“service/cert-manager created\n”
I0803 10:31:30.541576 150017 apply.go:47] Create Kind Cluster “msg”=“service/cert-manager-webhook created\n”
I0803 10:31:30.550966 150017 apply.go:47] Create Kind Cluster “msg”=“deployment.apps/cert-manager created\n”
I0803 10:31:30.564584 150017 apply.go:47] Create Kind Cluster “msg”=“deployment.apps/cert-manager-cainjector created\n”
I0803 10:31:30.608174 150017 apply.go:47] Create Kind Cluster “msg”=“deployment.apps/cert-manager-webhook created\n”
I0803 10:31:30.647488 150017 apply.go:47] Create Kind Cluster “msg”=“mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created\n”
I0803 10:31:30.676578 150017 apply.go:47] Create Kind Cluster “msg”=“validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created\n”
I0803 10:36:30.710939 150017 logs.go:57] “msg”=“Operation failed, retrying with backoff” “Cause”=“pod "cert-manager-96d7b5768-twhml" is not ready”
I0803 10:36:31.335160 150017 kustomize.go:1041] Building manifest for cert-manager/base

1 Like
2

Hi,

Can you confirm which version you are using? Have you also reviewed the troubleshooting documents[1] for corresponding version? If not, then please do so to start troubleshooting the issue first. Hoping that helps you.

[1]https://cloud.google.com/anthos/clusters/docs/bare-metal/1.6/troubleshooting

1 Like
3

Thanks @Sadik_M for your input.

We are using 1.8 version. I already looked into the troubleshooting guide but did not find useful info related to the “error” I am getting.

1 Like
4

Hi @encqor

Did you find the solution for this issue? I’m also encountering the same and been looking for fix.

Thanks

MD

5

Hi @encqor ,

I did finally clear my same issue, just need to enable the google component and services from the installation docs.

6

Hi @encqor ,

I believe @muanang did resolve the issue by enabling the services mentioned under the prerequisites for theGCP project.

For an additional information on other errors or limitations you might encounter with Anthos clusters on bare metal - V 1.8, you may need to follow the help center article about known issues.

As Google Cloud offers a variety of support packages to accommodate your support needs, you could always contact the support team for extra assistance for the version specific support.

1 Like
7

Muanang,

Could you please give more detail on what “enable the google component and services from installation docs” entails? I cannot find what you are suggesting.

Thank you!

8

@ccouch here

gcloud services enable --project PROJECT_ID \
    anthos.googleapis.com \
    anthosaudit.googleapis.com \
    anthosgke.googleapis.com \
    cloudresourcemanager.googleapis.com \
    gkeconnect.googleapis.com \
    gkehub.googleapis.com \
    serviceusage.googleapis.com \
    stackdriver.googleapis.com \
    monitoring.googleapis.com \
    logging.googleapis.com \
    storage.googleapis.com \
    opsconfigmonitoring.googleapis.com \
    container.googleapis.com \
    servicemanagement.googleapis.com \
    servicecontrol.googleapis.com

Just changed the PROJECT_ID with your actual id.
**After enabling it, you need to bind it to your IAM SA