kubernetes storage 101

Understanding how storage works

Storage concepts from k8 documentation

Kubernetes, a container orchestration engine had been built for stateless systems. These are generally the kinds of applications we commonly build.

applying deployment configuration for applications does help with this effectively. But, there may be cases where one wants to preserve state in a pod.

k8 simple node anatomy

Generally in a container, any volume mounted is ephemeral in nature (that is, it only lasts the lifetime of the pod). This is achieved by retaining the volume information within the host node.

The problem in this model is that, if the node associated to this pod crashes and if the pod is managed by a deployment then it may be rescheduled onto another node . During this time, the volume would be back to its initial state.

In the event this volume information needs to be preserved across nodes, then an abstraction of volume type can be chosen during the volume configuration.

sample list of supported volume types

awsElasticBlockStorage or gcePersistentDisk or azureDisk were the most commonly used storage until, Persistent Volume Claim (PVC) came into existence.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-deployment-1
spec:
  replicas: 1
  selector:
    matchLabels:
      role: test1
  template:
    metadata:
      labels:
        role: test1
    spec:
      volumes:
      - name: existing-pvc
        gcePersistentDisk:
          pdName: nfs-test-disk1
          fsType: ext4
      containers:
      - name: test-container
        image: gcr.io/google_containers/volume-nfs:0.8
        volumeMounts:
          - mountPath: /data
            name: existing-pvc

The problem with using any of the above storage would pollute the k8 config with cloud vendor specific configuration and make the config non-portable to another environment. Also, a devops admin would typically manage the storage disks, while may not manage the config of pod. And hence the abstraction.

Persistent Volume (PV) — Persistent Volume Claim (PVC)

A volume can be added into a Container of a pod using PersistentVolumeClaim (PVC).
A PVC is created using an abstraction called PersistentVolume (PV).

A PV provides a configuration to use GCEPersistentDisk or AWSElasticBlockStore or … to provide cloud specific storage. This provisioning can be simplified by using StorageClass

StorageClass provides an abstraction to provide a template of storage to use. The following is an example of StorageClass provided by GKE whenever a new cluster is created. It uses the PersistentDisk to be created whenever a PVC is requested.

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  creationTimestamp: "2020-03-24T03:19:07Z"
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
    kubernetes.io/cluster-service: "true"
  name: standard
  resourceVersion: "270"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/standard
  uid: 2822ba7c-f5ef-40ac-9f2d-487ec56ceb19
parameters:
  type: pd-standard
provisioner: kubernetes.io/gce-pd
reclaimPolicy: Delete
volumeBindingMode: Immediate

One can tweak the default StorageClass to provide SSD backed volume (for gcp).

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: faster
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-ssd

kubectl get sc to see all the StorageClass available in a cluster. To use this with pvc

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc1
spec:
  storageClassName: faster
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 30Gi

A sample deployment using the above PVC.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-client-1
spec:
  replicas: 1
  selector:
    matchLabels:
      role: client-1
  template:
    metadata:
      labels:
        role: client-1
    spec:
      volumes:
      - name: client-1-v
        persistentVolumeClaim:
          claimName: pvc1
      containers:
      - name: test-container
        image: debian
        command: ["sh", "-c", "tail -f /dev/null"]
        volumeMounts:
          - mountPath: /data
            name: client-1-v

⛑ Suggestions / Feedback ! 😃