K8S 持久化存储机制详解:PV 与 PVC 实践指南
K8S 持久化存储机制详解:PV 与 PVC 实践指南
目录
1. 基于 NFS 实现持久化存储
1.1 环境准备与 NFS 配置
在生产环境中,NFS 是最常用的网络存储方案之一。以下为实验环境的节点规划:
| 角色 | 主机地址 |
|---|---|
| NFS 服务端 | master(192.168.10.20) |
| NFS 客户端 | node01(192.168.10.30),node02(192.168.10.40) |
所有节点执行 NFS 安装:
# RedHat 7 系列
yum install -y nfs-common nfs-utils
# RedHat 8 系列
yum install -y nfs-utils
在 master 节点创建共享目录:
[root@master ~]# mkdir -p /shared/vol{1..5}
[root@master ~]# chmod 777 -R /shared/*
编辑 exports 配置文件:
[root@master ~]# vim /etc/exports
/shared/vol1 *(insecure,rw,no_root_squash,sync,no_subtree_check)
/shared/vol2 *(insecure,rw,no_root_squash,sync,no_subtree_check)
/shared/vol3 *(insecure,rw,no_root_squash,sync,no_subtree_check)
/shared/vol4 *(insecure,rw,no_root_squash,sync,no_subtree_check)
/shared/vol5 *(insecure,rw,no_root_squash,sync,no_subtree_check)
# 使配置生效
[root@master ~]# exportfs -rv
按顺序启动 rpcbind 和 nfs 服务:
[root@master ~]# systemctl start rpcbind && systemctl enable rpcbind
[root@master ~]# systemctl start nfs && systemctl enable nfs
验证 NFS 导出列表:
[root@master ~]# showmount -e
创建测试文件:
echo 'content-one' > /shared/vol1/index.html
echo 'content-two' > /shared/vol2/index.html
echo 'content-three' > /shared/vol3/index.html
echo 'content-four' > /shared/vol4/index.html
echo 'content-five' > /shared/vol5/index.html
1.2 创建 PersistentVolume
创建 storage-volumes.yaml 文件,定义多个 PV 资源:
apiVersion: v1
kind: PersistentVolume
metadata:
name: volume-data-1
labels:
type: nfs-storage
spec:
nfs:
path: /shared/vol1
server: 192.168.10.20
accessModes: ["ReadWriteMany","ReadWriteOnce"]
capacity:
storage: 1Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: volume-data-2
labels:
type: nfs-storage
spec:
nfs:
path: /shared/vol2
server: 192.168.10.20
accessModes: ["ReadWriteOnce"]
capacity:
storage: 2Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: volume-data-3
labels:
type: nfs-storage
spec:
nfs:
path: /shared/vol3
server: 192.168.10.20
accessModes: ["ReadWriteMany","ReadWriteOnce"]
capacity:
storage: 2Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: volume-data-4
labels:
type: nfs-storage
spec:
nfs:
path: /shared/vol4
server: 192.168.10.20
accessModes: ["ReadWriteMany","ReadWriteOnce"]
capacity:
storage: 4Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: volume-data-5
labels:
type: nfs-storage
spec:
nfs:
path: /shared/vol5
server: 192.168.10.20
accessModes: ["ReadWriteMany","ReadWriteOnce"]
capacity:
storage: 5Gi
部署 PV 并查看状态:
[root@master ~]# kubectl apply -f storage-volumes.yaml
persistentvolume/volume-data-1 created
persistentvolume/volume-data-2 created
persistentvolume/volume-data-3 created
persistentvolume/volume-data-4 created
persistentvolume/volume-data-5 created
[root@master ~]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
volume-data-1 1Gi RWO,RWX Retain Available 8s
volume-data-2 2Gi RWO Retain Available 8s
volume-data-3 2Gi RWO,RWX Retain Available 8s
volume-data-4 4Gi RWO,RWX Retain Available 8s
volume-data-5 5Gi RWO,RWX Retain Available 8s
1.3 定义 PersistentVolumeClaim
PVC 会根据访问模式和存储大小自动匹配最合适的 PV。以下示例定义了多路读写访问模式的 PVC:
storage-claim.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: app-storage
spec:
accessModes: ["ReadWriteMany"]
resources:
requests:
storage: 2Gi
---
apiVersion: v1
kind: Pod
metadata:
name: web-server
spec:
containers:
- name: nginx
image: nginx:latest
volumeMounts:
- name: web-content
mountPath: /usr/share/nginx/html
volumes:
- name: web-content
persistentVolumeClaim:
claimName: app-storage
部署并验证绑定状态:
[root@master ~]# kubectl apply -f storage-claim.yaml
persistentvolumeclaim/app-storage created
pod/web-server created
[root@master ~]# kubectl get pv,pvc
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/volume-data-1 1Gi RWO,RWX Retain Available 5m54s
persistentvolume/volume-data-2 2Gi RWO Retain Available 5m54s
persistentvolume/volume-data-3 2Gi RWO,RWX Retain Bound default/app-storage 5m54s
persistentvolume/volume-data-4 4Gi RWO,RWX Retain Available 5m54s
persistentvolume/volume-data-5 5Gi RWO,RWX Retain Available 5m54s
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/app-storage Bound volume-data-3 2Gi RWO,RWX 10s
[root@master ~]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
web-server 1/1 Running 0 58s 10.244.1.3 node01 <none> <none>
验证数据访问:
[root@master ~]# curl 10.244.1.3
content-three
多路读写测试:
可以使用相同的 PVC 创建多个 Pod,验证是否共享同一存储卷:
# 复制配置并修改 Pod 名称
cp storage-claim.yaml test-pod-1.yaml
cp storage-claim.yaml test-pod-2.yaml
# 修改 metadata.name 后执行
kubectl apply -f test-pod-1.yaml
kubectl apply -f test-pod-2.yaml
# 查看 Pod IP 并测试访问
kubectl get pod -o wide
curl [pod-ip]
2. 动态存储方案:StorageClass
Static PV 需要手动创建,对于大规模环境不友好。动态存储通过 StorageClass 实现自动创建 PV。
架构说明:
- PV 是集群级别的全局资源
- PVC 通过命名空间隔离
- StorageClass 是集群级别的全局资源
部署流程:
- 编写 NFS Client Provisioner 的 RBAC 授权
- 编写 Deployment 资源与 ServiceAccount 绑定
- 创建 StorageClass 关联 NFS 后端
- 编写测试 PVC 验证自动创建 PV
- 编写 Pod 使用 PVC 挂载数据
2.1 部署 NFS Client Provisioner
准备 NFS 共享目录:
[root@master ~]# mkdir /dynamic-storage
[root@master ~]# chmod 777 /dynamic-storage
[root@master ~]# vim /etc/exports
/dynamic-storage *(insecure,rw,no_root_squash,sync,no_subtree_check)
[root@master ~]# exportfs -rv
[root@master ~]# echo 'dynamic test content' > /dynamic-storage/index.html
创建 RBAC 资源:
nfs-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: nfs-provisioner
namespace: default
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: nfs-provisioner-clusterrole
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: nfs-provisioner-binding
subjects:
- kind: ServiceAccount
name: nfs-provisioner
namespace: default
roleRef:
kind: ClusterRole
name: nfs-provisioner-clusterrole
apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: nfs-provisioner-leader
namespace: default
rules:
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: nfs-provisioner-leader-binding
subjects:
- kind: ServiceAccount
name: nfs-provisioner
namespace: default
roleRef:
kind: Role
name: nfs-provisioner-leader
apiGroup: rbac.authorization.k8s.io
创建 Provisioner Deployment:
nfs-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nfs-client-provisioner
labels:
app: nfs-provisioner
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: nfs-provisioner
strategy:
type: Recreate
template:
metadata:
labels:
app: nfs-provisioner
spec:
serviceAccountName: nfs-provisioner
containers:
- name: nfs-provisioner
image: registry.cn-hangzhou.aliyuncs.com/syhj/public:nfs-client-provisioner_D20241016
volumeMounts:
- name: nfs-storage
mountPath: /persistentvolumes
env:
- name: PROVISIONER_NAME
value: dynamic-nfs-storage
- name: NFS_SERVER
value: 192.168.10.20
- name: NFS_PATH
value: /dynamic-storage
securityContext:
runAsUser: 0
runAsGroup: 0
volumes:
- name: nfs-storage
nfs:
server: 192.168.10.20
path: /dynamic-storage
创建 StorageClass:
nfs-storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: dynamic-nfs-storage
provisioner: dynamic-nfs-storage
# 支持 PVC 创建后扩容
allowVolumeExpansion: true
parameters:
# true: 删除 PVC 时将卷重命名归档
# false: 真正删除卷数据
archiveOnDelete: "true"
2.2 验证动态存储效果
创建测试 PVC:
test-claim.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: dynamic-test-claim
annotations:
volume.beta.kubernetes.io/storage-class: "dynamic-nfs-storage"
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 10Mi
创建测试 Pod:
test-pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: storage-test-pod
spec:
containers:
- image: ikubernetes/myapp:v1
name: test-container
volumeMounts:
- mountPath: /test-data
name: nfs-storage
volumes:
- name: nfs-storage
persistentVolumeClaim:
claimName: dynamic-test-claim
执行部署:
kubectl apply -f nfs-rbac.yaml
kubectl apply -f nfs-deployment.yaml
kubectl apply -f nfs-storageclass.yaml
kubectl apply -f test-claim.yaml
kubectl apply -f test-pod.yaml
验证结果:
[root@master ~]# kubectl get pods,svc
NAME READY STATUS RESTARTS AGE
pod/nfs-client-provisioner-555df7ccd5-z86nn 1/1 Running 0 105s
pod/web-server 1/1 Running 0 33m
pod/storage-test-pod 1/1 Running 0 20s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 3d16h
[root@master ~]# kubectl exec -it storage-test-pod /bin/sh
/ # cd /test-data
/test-data # ls
/test-data # touch file1 file2
/test-data # ls
file1 file2
/test-data # exit
# 查看 NFS 服务端目录
[root@master ~]# ls /dynamic-storage/
default-dynamic-test-claim-pvc-a1b2c3d4-e5f6-7890-abcd-ef1234567890 index.html
[root@master ~]# ls /dynamic-storage/default-dynamic-test-claim-pvc-*/
file1 file2
常见问题处理:
若 PVC 状态显示 Pending,检查是否为 Kubernetes 版本问题。v1.20+ 版本默认禁用 selfLink:
# 编辑 API Server 配置
[root@k8sm ~]# vim /etc/kubernetes/manifests/kube-apiserver.yaml
# 添加配置项
- --feature-gates=RemoveSelfLink=false
# 重启服务
# 二进制安装:systemctl restart kube-apiserver && systemctl restart kubelet
# kubeadm 安装:systemctl restart kubelet
# 验证
[root@master ~]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
dynamic-test-claim Bound pvc-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx 10Mi RWX dynamic-nfs-storage 13m
3. 数据库持久化:MySQL 存储实战
3.1 创建 MySQL 专用存储卷
配置 NFS 共享目录:
[root@master ~]# mkdir -p /dynamic-storage/mysql
[root@master ~]# chmod 777 -R /dynamic-storage/mysql/
[root@master ~]# vim /etc/exports
/dynamic-storage/mysql 192.168.10.0/24(rw,no_root_squash,sync)
[root@master ~]# exportfs -rv
[root@master ~]# systemctl restart nfs rpcbind
创建 MySQL 专用 PV:
mysql-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: mysql-data-pv
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 1Gi
persistentVolumeReclaimPolicy: Retain
storageClassName: mysql-nfs
nfs:
path: /dynamic-storage/mysql
server: 192.168.10.20
创建 MySQL 专用 PVC:
mysql-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-data-claim
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: mysql-nfs
查看绑定状态:
[root@master ~]# kubectl get pv,pvc
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/mysql-data-pv 1Gi RWO Retain Bound default/mysql-data-claim mysql-nfs 12s
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/mysql-data-claim Bound mysql-data-pv 1Gi RWO mysql-nfs 9s
3.2 部署 MySQL 应用
mysql-deployment.yaml
apiVersion: v1
kind: Service
metadata:
name: mysql-service
spec:
ports:
- port: 3306
selector:
app: mysql-database
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: mysql-deployment
spec:
selector:
matchLabels:
app: mysql-database
template:
metadata:
labels:
app: mysql-database
spec:
containers:
- image: daocloud.io/library/mysql:5.7.5-m15
name: mysql
env:
- name: MYSQL_ROOT_PASSWORD
value: "rootpass123"
ports:
- containerPort: 3306
name: mysql
volumeMounts:
- name: mysql-persistent-storage
mountPath: /var/lib/mysql
volumes:
- name: mysql-persistent-storage
persistentVolumeClaim:
claimName: mysql-data-claim
部署 MySQL:
[root@master ~]# kubectl apply -f mysql-deployment.yaml
service/mysql-service created
deployment.apps/mysql-deployment created
[root@master ~]# kubectl get pods,svc -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/mysql-deployment-6654fcb867-xnfjq 1/1 Running 0 20s 10.244.1.3 node01 <none> <none>
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 3d22h <none>
service/mysql-service ClusterIP 10.96.245.196 <none> 3306/TCP 20s app=mysql-database
3.3 故障模拟与数据验证
创建测试数据:
[root@master ~]# kubectl exec -it mysql-deployment-6654fcb867-xnfjq /bin/bash
root@mysql-deployment-6654fcb867-xnfjq:/# mysql -uroot -prootpass123
..........
mysql> create database test_db;
Query OK, 1 row affected (0.02 sec)
mysql> create table test_db.users(id int, name varchar(50));
Query OK, 0 rows affected (0.05 sec)
mysql> insert into test_db.users values(1, 'admin');
Query OK, 1 row affected (0.01 sec)
mysql> select * from test_db.users;
+------+------+
| id | name |
+------+------+
| 1 | admin |
+------+------+
模拟节点故障:
[root@node01 ~]# poweroff
等待 Pod 重新调度(大约 10 分钟):
[root@master ~]# kubectl get pods,svc -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/mysql-deployment-6654fcb867-5ln28 1/1 Running 0 12m 10.244.2.3 node02 <none> <none>
pod/mysql-deployment-6654fcb867-xnfjq 1/1 Terminating 0 19m 10.244.1.3 node01 <none> <none>
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 3d23h <none>
service/mysql-service ClusterIP 10.96.158.55 <none> 3306/TCP 19m app=mysql-database
验证数据完整性:
[root@master ~]# kubectl exec -it mysql-deployment-6654fcb867-5ln28 /bin/bash
root@mysql-deployment-6654fcb867-5ln28:/# mysql -uroot -prootpass123
......
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| test_db |
| mysql |
| performance_schema |
+--------------------+
4 rows in set (0.03 sec)
mysql> select * from test_db.users;
+------+------+
| id | name |
+------+------+
| 1 | admin |
+------+------+
1 row in set (0.04 sec)
数据完好无损,故障转移成功。
4. AWS EKS 环境下 EFS 存储配置
AWS EKS 支持使用 EFS(Elastic File System)作为 Kubernetes 存储后端,提供跨可用区的共享存储。
部署配置文件:
efs-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: efs-nginx
namespace: default
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: efs-nginx
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
labels:
app: efs-nginx
spec:
containers:
- image: nginx:1.19
imagePullPolicy: IfNotPresent
name: nginx-container
volumeMounts:
- mountPath: /etc/localtime
name: localtime
- name: efs-storage
mountPath: /data
volumes:
- hostPath:
path: /etc/localtime
type: ""
name: localtime
- name: efs-storage
persistentVolumeClaim:
claimName: efs-pvc
dnsPolicy: ClusterFirst
restartPolicy: Always
terminationGracePeriodSeconds: 30
efs-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: efs-pv
spec:
capacity:
storage: 5Gi
volumeMode: Filesystem
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
storageClassName: efs-storage-class
csi:
driver: efs.csi.aws.com
volumeHandle: fs-068d80072c18bc5c9
efs-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: efs-pvc
spec:
accessModes:
- ReadWriteMany
storageClassName: efs-storage-class
resources:
requests:
storage: 5Gi
efs-storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: efs-storage-class
provisioner: efs.csi.aws.com
部署流程:
kubectl apply -f efs-storageclass.yaml
kubectl apply -f efs-pv.yaml
kubectl apply -f efs-pvc.yaml
kubectl apply -f efs-deployment.yaml
验证结果:
[root@master ~]# kubectl get pv,pvc
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS AGE
efs-pv 5Gi RWX Retain Bound default/efs-pvc efs-storage-class 5m
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
efs-pvc Bound efs-pv 5Gi RWX efs-storage-class 3m
[root@master ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
efs-nginx-xxxxxxxxx-xxxx 1/1 Running 0 2m

以上完成了 Kubernetes 持久化存储的完整实践,涵盖静态 NFS、动态 StorageClass、数据库持久化以及云厂商 EFS 方案。