This section introduces the Qumulo Container Storage Interface (CSI) driver and explains how you can connect your Kubernetes cluster to your Qumulo cluster by using the Qumulo CSI driver.

To automate container storage, enable dynamic volumes, and help you scale your application container images based on usage and workflows, Qumulo uses the CSI driver to connect the Kubernetes orchestrator to Qumulo persistent storage. (In comparison, for example, the NFS CSI Driver for Kubernetes requires unprivileged NFS access for dynamic volumes and doesn’t support volume sizing and expansion.)

For general driver information, see the Container Storage Interface (CSI) Specification.

Supported Features

The Qumulo CSI Driver supports:

  • Static and dynamic (expansion) provisioning over NFSv3

  • The following Persistent Volume Claim access modes:

    • ReadOnlyMany

    • ReadWriteMany

    • ReadWriteOnce

    • ReadWriteOncePod

  • NFSv4.1

Unsupported Features

Requirements

  • A Qumulo cluster

  • Kubernetes 1.22 (and higher)

Connecting Your Qumulo Cluster to Kubernetes

This section explains how you can configure, provision, and mount Qumulo storage for each Pod (a logical wrapper for a container) on Kubernetes by using dynamic provisioning. This gives you more control over persistent volume capacity.

Step 1: Install the Qumulo CSI Driver

  1. Connect to a machine that has kubectl and can access your Kubernetes cluster.

  2. Download the .zip file or use one of the following commands.

    • S3

      aws s3 cp s3://csi-driver-qumulo/deploy_v1.2.0.zip ./

    • HTTP with wget (Linux)

      wget https://csi-driver-qumulo.s3.us-west-2.amazonaws.com/deploy_v1.2.0.zip

    • HTTP with curl (macOS)

      curl -O https://csi-driver-qumulo.s3.us-west-2.amazonaws.com/deploy_v1.2.0.zip

  3. Extract the contents of the .zip file.

  4. Run the shell script and specify the current release version. For example:

    • Linux:

      cd deploy_v1.2.0
chmod +x install-driver.sh
./install-driver.sh

    • Windows:

      cd deploy_v1.2.0
install-driver.bat

    The script configures Qumulo’s prebuilt Elastic Container Registry (ECR) image (from public.ecr.aws/qumulo/csi-driver-qumulo:v1.2.0) and installs it on your Kubernetes system.

Step 2: Configure Volume and NFS Export Paths

To prepare your Qumulo cluster for connecting to your Kubernetes cluster, you must first configure your volume and NFS export paths on your Qumulo cluster by setting the following parameters for each storage class that you define.

  1. For storeRealPath, from the root of the Qumulo file system, create a directory for storing volumes on your Qumulo cluster, for example /csi/volumes1.

  2. For storeExportPath, create the NFS export for hosting the persistent volume.

  3. If your cluster has more than one tenant, specify the tenant ID that contains your NFS export for the tenantId parameter.

Step 3: Configure Credentials

To connect your Kubernetes cluster to your Qumulo cluster, you must either use an existing account or create a new account for the CSI driver to communicate with the Qumulo API.

  1. Configure a username and password for a user on your Qumulo cluster.

  2. The configured username must have the following file permissions:

    • Lookup on storeRealPath

    • Create directories in storeRealPath

    • Create and modify quotas:

      • PRIVILEGE_QUOTA_READ

      • PRIVILEGE_QUOTA_WRITE

    • Read NFS exports: PRIVILEGE_NFS_EXPORT_READ

    • Perform TreeDelete operations on volume directories: PRIVILEGE_FS_DELETE_TREE_WRITE

For more information, see Role-Based Access Control (RBAC) with Qumulo Core on Qumulo Care.

Step 4: Create and Configure Secrets

To allow the CSI driver to operate with your Qumulo cluster, you must create and configure Secrets. You can use one of the following methods:

  • Basic authentication with a username and password

    For example:

    kubectl create secret generic cluster1-login \
  --type="kubernetes.io/basic-auth" \
  --from-literal=username=myusername \
  --from-literal=password=mypassword \
  --namespace=kube-system

  • A bearer token (or access token)

    For example:

    TOKEN='access-v1:zNTc5D0zWTdNi/KsZo620fu71TweGh47u+S/5NbV...'
kubectl create secret generic cluster1-login \
  --from-literal=access_token="$TOKEN" \
  --namespace=kube-system

    For more information, see Creating and Using Bearer Tokens to Authenticate Qumulo REST API Calls.

Configuring the CSI Driver to Verify Your Cluster’s TLS Certificate

The CSI driver communicates with your Qumulo cluster by using the Qumulo REST API over HTTPS. By default, the CSI driver accepts any TLS certificate, including self-signed certificates, without verifying the identity of the cluster that presents it.

To configure verification, add one of the following optional keys to the Secret that you created. The following table outlines the CSI driver’s behavior and key configuration process for each type of cluster certificate issuer.

Cluster Certificate Issuer CSI Driver Behavior Key Configuration

Public CA

The CSI driver verifies the certificate against standard public CAs.

Set the tls_insecure key to false. For example:

kubectl create secret generic cluster1-login \
  --type="kubernetes.io/basic-auth" \
  --from-literal=username=myusername \
  --from-literal=password=mypassword \
  --from-literal=tls_insecure=false \
  --namespace=kube-system

Self-Signed or Your Organization's Internal CA

The CSI driver verifies the certificate against only the .pem bundle that you supply.

Specify the .pem certificate by using the ca_cert key.

If a certificate isn't available, you can retrieve it by using the openssl tool. For example:

openssl s_client \
  -connect cluster.example.com:8000 \
  -showcerts </dev/null \
  2>/dev/null | openssl x509 > cluster1-ca.pem

When you create your Secret, specify a path to the certificate.

For example:

kubectl create secret generic cluster1-login \
  --type="kubernetes.io/basic-auth" \
  --from-literal=username=myusername \
  --from-literal=password=mypassword \
  --from-file=ca_cert=cluster1-ca.pem \
  --namespace=kube-system

Any Issuer

The CSI driver skips verification and doesn't log the warning.

Set the tls_insecure key to true.

Step 5: Create a Storage Class

To link your Kubernetes cluster to your Qumulo cluster, you must create a storage class on your Kubernetes cluster.

  1. Begin with the example Qumulo storage class configuration.

    ---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: cluster1
provisioner: qumulo.csi.k8s.io
parameters:
  server: 203.0.113.0
  storeRealPath: "/regions/4234/volumes"
  storeExportPath: "/some/export"
  csi.storage.k8s.io/provisioner-secret-name: cluster1-login
  csi.storage.k8s.io/provisioner-secret-namespace: kube-system
  csi.storage.k8s.io/controller-expand-secret-name: cluster1-login
  csi.storage.k8s.io/controller-expand-secret-namespace: kube-system
reclaimPolicy: Delete
volumeBindingMode: Immediate
mountOptions:
  - nolock
  - proto=tcp
  - vers=3
allowVolumeExpansion: true

  2. Edit the configuration for your Qumulo cluster.

    1. Name your storage class.

    2. Specify server and storeRealPath.

    3. Specify storeExportPath.

    4. (Optional) Specify tenantId.

    5. Configure the following parameters to point to the Secrets that you have created and configured in the namespace in which you installed the CSI driver:

      • controller-expand-secret-name

      • controller-expand-secret-namespace

      • provisioner-secret-name

      • provisioner-secret-namespace

    6. Specify the NFS mountOptions. For example:

      mountOptions:
  - nolock
  - proto=tcp
  - vers=3

    7. To create the class, apply the configuration. For example:

    kubectl create -f storageclass-qumulo.yaml

Step 6: Create a Persistent Volume Claim (PVC) and Apply it to a Pod

To apply a PVC to a Pod dynamically, you must first configure and create it.

  1. Begin with the example PVC configuration.

    ---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: claim1
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: cluster1
  resources:
    requests:
      storage: 1Gi

  2. Edit the configuration for your PVC.

    1. Name your claim.

    2. Change storageClassName to the name of your storage class.

    3. Specify the capacity in spec.resources.requests.storage. This parameter lets you create a quota on your Qumulo cluster.

    4. To create the claim, apply the configuration. For example:

      kubectl apply -f dynamic-pvc.yaml

  3. Use the claim in a Pod or a Deployment. For example:

    ---
apiVersion: v1
kind: Pod
metadata:
  name: claim1-pod
spec:
  volumes:
    - name: cluster1
      persistentVolumeClaim:
        claimName: claim1
  containers:
    - name: claim1-container
      image: ...
      volumeMounts:
        - mountPath: "/cluster1"
          name: cluster1

  4. You can launch and use your container image.

Upgrading the Qumulo CSI Driver

To upgrade the CSI driver, you can install a new version without removing the previous installation.

Prerequisites

Before you begin, confirm that your Kubernetes nodes can pull images from registry.k8s.io.

To Upgrade the Qumulo CSI Driver

  1. Download the .zip file for the new driver version and extract its contents.

  2. Run the installation script.

    cd deploy_v1.2.0
chmod +x install-driver.sh
./install-driver.sh

  3. (Optional) To monitor the installation process, run the following commands.

    kubectl -n kube-system rollout status \
  deployment/csi-qumulo-controller

kubectl -n kube-system rollout status \
  daemonset/csi-qumulo-node

  4. To verify the upgrade, create a test Persistent Volume Claim against one of your storage classes, confirm that it reaches the Bound state, and then delete it.

    Any ReadWriteOncePod claims that an earlier driver version left in the Pending state complete automatically after the upgrade.

  5. (Optional) If you created the access-secrets role and the default-to-secrets role binding while creating and configuring Secrets for an earlier version of the CSI driver, you may remove them. Version 1.2.0 (and higher) grants this access within the CSI driver’s own RBAC configuration.

    kubectl delete rolebinding \
  default-to-secrets --namespace kube-system

kubectl delete role \
  access-secrets --namespace kube-system

To roll back to the previous driver version, run install-driver.sh from the previous version’s deployment files.