Why storage is a separate layer

A Pod’s root filesystem lives on the node and dies with the Pod. That is fine for stateless web tiers; it is wrong for databases, queues, and anything that must survive rescheduling. Kubernetes models durable disk through the volume API: cluster administrators (or cloud integrations) expose capacity; application teams request it; the scheduler only places Pods once the request can be satisfied.

Three API objects carry most of the mental load:

Object	Scope	Role
PersistentVolume (PV)	Cluster	Represents a piece of storage—NFS export, cloud disk, iSCSI LUN, etc.
PersistentVolumeClaim (PVC)	Namespace	Application’s request for storage (size, access mode, class)
StorageClass	Cluster	Template for dynamic provisioning: “when someone asks for fast-ssd, create disks like this”

For control-plane context, see Kubernetes architecture in simple terms. For mounting volumes in Pods, hands-on Part 3 is a practical companion.

The binding lifecycle (static vs dynamic)

Storage flows in one direction: PVC → PV → node → Pod.

1. You create a PVC specifying size, accessModes, and optionally storageClassName.
2. The control plane finds or creates a matching PV and sets the PVC phase to Bound.
3. When a Pod references the PVC in spec.volumes, the scheduler must pick a node that can attach that volume (for many cloud disks, only one node at a time in ReadWriteOnce mode).
4. The kubelet mounts the volume into the container filesystem at the path you declare in volumeMounts.

Static provisioning: an admin creates PVs ahead of time (or they exist from a prior integration). A PVC binds to a PV that matches size, access mode, storage class, and selector labels.

Dynamic provisioning: no pre-created PV. The PVC names a StorageClass; a provisioner (usually a CSI driver) creates a PV automatically and binds it. This is what managed Kubernetes (EKS, GKE, AKS) expects day to day.

PersistentVolumeClaim (PVC) in detail

A PVC is how a team asks for disk without caring whether the backing store is EBS, Azure Disk, or NFS—only the contract:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-pvc
  namespace: app
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: gp3
  resources:
    requests:
      storage: 20Gi

Important fields:

• resources.requests.storage — Minimum size. For many provisioners you can expand later with allowVolumeExpansion: true on the StorageClass (not all drivers support shrink).
• accessModes — Must be compatible with the PV (see below).
• storageClassName — Selects which StorageClass provisions the volume. Omitting it uses the cluster’s default StorageClass (if one is annotated).
• volumeMode — Filesystem (default, formatted and mounted) or Block (raw device for databases that manage their own layout).
• selector — Optional label selector to bind only to PVs with matching labels (common in static setups).

PVC phases: Pending (no suitable PV yet), Bound, Lost (underlying PV disappeared—data may be gone), Released (claim deleted but reclaim policy still owns the PV).

PersistentVolume (PV) in detail

A PV is cluster-scoped storage inventory. Example fragment for a statically provisioned NFS volume:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv-001
spec:
  capacity:
    storage: 50Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: nfs
  nfs:
    server: 10.0.0.50
    path: /exports/data

PVs can also embed cloud-specific sources (awsElasticBlockStore, gcePersistentDisk, etc.), but modern clusters prefer CSI drivers that create PVs dynamically with a csi stanza instead of in-tree volume types.

Key PV fields:

• capacity.storage — Advertised size (must be ≥ PVC request).
• claimRef — Set when bound; links to the PVC.
• persistentVolumeReclaimPolicy — What happens when the PVC is deleted (see reclaim policies).
• mountOptions — Passed to the mount (e.g. NFS nfsvers=4.1).
• nodeAffinity — Restricts which nodes can use the volume (zones/regions for zonal disks).

StorageClass in detail

A StorageClass is a named recipe for dynamic disks:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  encrypted: "true"
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
reclaimPolicy: Delete

Fields that matter in production:

• provisioner — Identifies the CSI driver or legacy in-tree provisioner (prefer CSI for new clusters).
• parameters — Driver-specific: disk type, IOPS, filesystem, encryption, etc.
• reclaimPolicy — Delete or Retain for PVs this class creates (overrides default on dynamically provisioned PVs).
• volumeBindingMode — Immediate (provision as soon as PVC exists) vs WaitForFirstConsumer (delay until a Pod needs the volume so scheduling can respect topology/zone).
• allowedTopologies — Limits zones where volumes may be created.
• mountOptions — Default mount flags for volumes of this class.

Mark exactly one class as default with storageclass.kubernetes.io/is-default-class: "true" so PVCs without storageClassName still work—or require explicit classes to avoid surprise bills on premium tiers.

Access modes (who can mount how)

Mode	Abbrev	Meaning
ReadWriteOnce	RWO	Single node read/write. Typical for block storage (EBS, PD). Multiple Pods on the same node can share if the driver allows.
ReadOnlyMany	ROX	Many nodes read-only. Shared config or static assets.
ReadWriteMany	RWX	Many nodes read/write. Requires NFS, EFS, Azure Files, CephFS, etc.—not plain zonal block disks.
ReadWriteOncePod	RWOP	Single Pod only (Kubernetes 1.22+). Stricter than RWO for some CSI drivers.

The PVC’s requested modes must be a subset of the PV’s modes. Mismatch leaves the PVC Pending forever.

Reclaim policies (what happens after delete)

• Delete — PVC deleted → PV and often the underlying cloud disk are removed. Default for dynamic classes in dev; verify backups first in prod.
• Retain — PVC deleted → PV object may go to Released but the disk/data remains. Admins must manually reattach or clean up. Safer for production databases when you need forensic recovery.
• Recycle — Deprecated; do not use on new clusters.

Operational rule: production databases on Retain plus documented cleanup runbooks; ephemeral dev namespaces on Delete to control cost (see also FinOps and architecture for orphaned volume hygiene).

CSI: how provisioners actually work today

The Container Storage Interface (CSI) is the standard way volume drivers integrate with Kubernetes. Components you will see:

• CSI driver controller — Creates/deletes volumes, snapshots (cluster-level).
• CSI node plugin — Mounts volumes on the kubelet’s node.
• External provisioner sidecar — Watches PVCs and calls CreateVolume on the driver.
• External attacher — Attaches block volumes to nodes in the cloud.

When you kubectl get storageclass on EKS, GKE, or AKS, the PROVISIONER column shows names like ebs.csi.aws.com or pd.csi.storage.gke.io. Installing a driver (Helm chart or add-on) is a platform task; application teams consume it through StorageClass names. For how CSI fits next to CRI at the kubelet boundary, see CRI and CSI deep dive.

Wiring storage into a Pod

PVCs do nothing until a Pod uses them:

apiVersion: v1
kind: Pod
metadata:
  name: app-with-data
spec:
  containers:
    - name: app
      image: myapp:1.0
      volumeMounts:
        - name: data
          mountPath: /var/lib/data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: data-pvc

Notes:

• claimName must exist in the same namespace as the Pod.
• subPath can mount a subdirectory of a volume (use carefully with concurrent writers).
• readOnly: true on the volume reference prevents writes at the kubelet layer.
• For block mode, use volumeDevices instead of volumeMounts and a privileged or capable container.

StatefulSets and volumeClaimTemplates

StatefulSets give each replica a stable network identity and, via volumeClaimTemplates, a dedicated PVC per Pod:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: db
spec:
  serviceName: db
  replicas: 3
  selector:
    matchLabels:
      app: db
  template:
    metadata:
      labels:
        app: db
    spec:
      containers:
        - name: db
          image: postgres:16
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: gp3
        resources:
          requests:
            storage: 100Gi

Kubernetes creates PVCs named data-db-0, data-db-1, … bound to Pods db-0, db-1, …. Scale-down keeps PVCs by default—an intentional data-safety choice that also causes orphaned disks if you forget cleanup. For end-to-end PostgreSQL and Redis manifests (headless Services, probes, secrets), see StatefulSets with PostgreSQL and Redis.

Snapshots and cloning (brief)

VolumeSnapshot and VolumeSnapshotClass (snapshot.storage.k8s.io API) capture point-in-time copies if the CSI driver supports them. Restore by creating a PVC from a dataSource pointing at a snapshot—useful for backups and cloning environments. Treat snapshots as part of your DR story, not a substitute for application-consistent backup where databases need quiescing.

Troubleshooting cheat sheet

Symptom	Likely cause	What to check
PVC Pending	No StorageClass, wrong class, no provisioner, insufficient quota	kubectl describe pvc Events; get sc; driver pods in kube-system
Pod Pending, volume events	Volume in another AZ, RWO already attached elsewhere	describe pod; WaitForFirstConsumer topology; node labels
Pod running, empty dir	Wrong mountPath, subPath, or app writes elsewhere	Exec and inspect; verify volumeMounts
Permission denied on mount	UID/fsGroup mismatch, root-squash on NFS	securityContext.fsGroup; NFS export options
Data gone after delete	reclaimPolicy: Delete on dynamic PV	StorageClass reclaim policy; backups

Useful commands:

kubectl get pvc,pv,sc
kubectl describe pvc data-pvc -n app
kubectl get events -n app --field-selector involvedObject.name=data-pvc
kubectl describe pod app-with-data -n app

Production practices

• Name StorageClasses by intent — fast-ssd, standard, archive—not by cloud SKU alone, so apps stay portable at the YAML level.
• Use WaitForFirstConsumer for zonal block storage so volumes land in the same zone as the Pod.
• Size with expansion in mind — Enable allowVolumeExpansion where supported; monitor disk usage like CPU.
• Separate dev and prod reclaim policies — Delete in dev sandboxes; Retain plus snapshots for prod data.
• Encrypt by default — StorageClass parameters or cloud account policies (aligns with cluster RBAC for who can create PVCs).
• GitOps the templates — PVCs and StorageClasses in version control; avoid one-off kubectl apply in production (GitOps principles).
• Audit orphaned PVs — Released PVs and unattached cloud volumes are a recurring cost leak.

Local and lab clusters (kind, minikube)

Local clusters often ship with a default StorageClass backed by hostPath or a lightweight provisioner (e.g. rancher.io/local-path, standard on minikube). Good for learning binding and mounts; not representative of zone topology, attach limits, or performance. Practice the same PVC → Pod flow from hands-on Part 1 before trusting production cloud classes.

A practical learning path

1. List StorageClasses and create a PVC without a Pod; watch it bind (or stay Pending with WaitForFirstConsumer).
2. Deploy a Pod that mounts the PVC; write a file; delete the Pod; reschedule and confirm data persists.
3. Change reclaim policy on a test class and observe what happens when you delete the PVC.
4. Deploy a small StatefulSet with volumeClaimTemplates and scale up/down; inspect per-replica PVC names.
5. Intentionally request ReadWriteMany on a block-only class and read the Events—then fix it.

Further reading

• Kubernetes documentation — Storage, Persistent Volumes, Storage Classes, Volume Snapshots
• Kubernetes documentation — CSI Volume Cloning and Snapshots
• CNCF CSI driver list — vendor drivers for AWS, Azure, GCP, Ceph, NFS, etc.
• Kubernetes documentation — StatefulSets (identity and stable storage)

Blog index · Kubernetes architecture · CRI and CSI · Cluster RBAC · Hands-on Part 3

Back to blog list

Back to blog list