GCS FUSE CSI driver with GKE autopilot cluser - sidecar container

[David_Straub]

Hi community!

I’m running a GKE autopilot cluster and would like to add persistent ReadWriteMany storage. After evaluating the options (Filestore is too expensive at this point), I found the GCS FUSE CSI driver to be potentially a good fit for my use case.

Unfortunately, I haven’t been able to get it to work so far. I’v been following the documentation closely, but I am always getting the following error:

MountVolume.SetUp failed for volume “gcs-fuse-csi-pv” : rpc error: code = FailedPrecondition desc = failed to find the
sidecar container in Pod spec

I checked that

• the pod is running on a node which has a running gcsfusecsi pod
• the PersistentVolume and PersistentVolumeClaim both have status BOUND

This my minimal example for the deployment & PV:

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: gcs-fuse-csi-pv
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 5Gi
  storageClassName: standard-rwx
  claimRef:
    namespace: default
    name: gcs-fuse-csi-static-pvc
  mountOptions:
    - implicit-dirs
  csi:
    driver: gcsfuse.csi.storage.gke.io
    volumeHandle: grampshub-fuse
    volumeAttributes:
      gcsfuseLoggingSeverity: warning

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: gcs-fuse-csi-static-pvc
  namespace: default
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  volumeName: gcs-fuse-csi-pv
  storageClassName: standard-rwx
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gcs-fuse-debug
  namespace: default
  annotations:
    gke-gcsfuse/volumes: "true"
    gke-gcsfuse/cpu-limit: "10"
    gke-gcsfuse/memory-limit: 10Gi
    gke-gcsfuse/ephemeral-storage-limit: 10Gi
    gke-gcsfuse/cpu-request: 500m
    gke-gcsfuse/memory-request: 1Gi
    gke-gcsfuse/ephemeral-storage-request: 5Gi
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gcs-fuse-debug
  template:
    metadata:
      labels:
        app: gcs-fuse-debug
    spec:
      containers:
      - name: debug-container
        image: busybox
        command: ["sh", "-c", "sleep 3600"]
        resources:
          limits:
            ephemeral-storage: 1Gi
            memory: 2Gi
            cpu: 100m
          requests:
            ephemeral-storage: 1Gi
            memory: 2Gi
            cpu: 100m
        volumeMounts:
        - name: gcs-fuse-volume
          mountPath: /mnt/gcs
      volumes:
      - name: gcs-fuse-volume
        persistentVolumeClaim:
          claimName: gcs-fuse-csi-static-pvc

This blog post mentions that implementing the sidecar containers was not easy for autopilot clusters, but it sounds as if this challenge was resolved, so I understood the approach as per the docs should work also with autopilot clusters.

Any hint would be appreciated, thanks!

[francislouie]

Hi @David_Straub,

Welcome to Google Cloud Community!

Upon checking on the configuration, it seems like the metadata annotation is misconfigured. The gke-gcsfuse/volumes: "true” annotation should be under the pod template - .spec section.

To learn more about Kubernetes deployment specs, check out these resources:

• Writing a Deployment Spec
• Creating a deployment

Please see image below to see exactly what’s been modified:

Here’s a working test deployment I’ve created using the configuration shown above.

Note: For demonstration purposes, some annotations from the original configuration have been removed from the modified version. Feel free to add them back to your own configuration as needed.

I hope the above information is helpful.

[David_Straub]

Hi @francislouie ,

somehow I didn’t get a notification for your reply and only saw it now. You’re right, that was the mistake, and with the change it works like a charm! Thanks so much!

David