文章目录

01 引言
02 CSI的核心组件和部署架构
- 2.1 CSI Controller
- 2.2 CSI Node
03 CSI存储插件应用实战

01 引言

声明：本文为《Kubernetes权威指南：从Docker到Kubernetes实践全接触（第5版）》的读书笔记

Kubernetes从1.9版本开始引入容器存储接口 Container Storage Interface (CSI)机制，用于在Kubernetes和外部存储系统之间建立一套标准的存储管理接口，通过该接口为容器提供存储服务。

02 CSI的核心组件和部署架构

Kubernetes CSI存储插件的关键组件和推荐的容器化部署架构如下（其中主要包括两类组件：CSI Controller和CSI Node）：
在这里插入图片描述

2.1 CSI Controller

CSI Controller的主要功能是提供存储服务视角对存储资源和存储卷进行管理和操作。在Kubernetes中建议将其部署为单实例Pod，可以使用StatefulSet或 Deployment控制器进行部署，设置副本数量为1，保证一种存储插件只运行一个控制器实例。

在这个Pod内部署两个容器，分别提供以下功能.

与Master ( kube-controller-manager) 通信的辅助sidecar容器：在 sidecar容器内又可以包含external-attacher和external-provisioner两个容器，它们的功能分别如下：

external-attacher：监控VolumeAttachment资源对象的变更，触发针对 CSI端点的ControllerPublish和ControllerUnpublish操作；
externail-provisioner：监控PersistentVolumeClaim资源对象的变更，触发针对CSI端点的CreateVolume和DeleteVolume操作。

另外，社区正在引入具备其他管理功能的sidecar工具，例如：externalsnapshotter，用于管理存储快照，目前为Alpha阶段；external-resizer用于管理存储容量扩容，目前为Beta阶段。

CsI Driver存储驱动容器，由第三方存储提供商提供，需要实现上述接口：这两个容器通过本地Socket (Unix Domain Socket, UDS)，并使用gPRC协议进行通信。sidecar容器通过Socket调用CSI Driver容器的CSI接口，CSI Driver 容器负责具体的存储卷操作。

2.2 CSI Node

CSI Node的主要功能是对主机（Node ）上的Volume进行管理和操作，在 Kubernetes中建议将其部署为DaemonSet，在需要提供存储资源的各个Node上都运行一个Pod。

在这个Pod中部署以下两个容器：

与kubelet通信的辅助sidecar容器node-driver-registrar：主要功能是将存储驱动注册到kubelet中；
CSI Driver存储驱动容器：由第三方存储提供商提供，主要功能是接收 kubelet 的调用，需要实现一系列与 Node 相关的 CSI 接口，例如： NodePublishVolume 接口（用于将 Volume 挂载到容器内的目标路径）、NodeUnpublishVolume接口（用于从容器中卸载Volu加粗样式me），等等。

流程：

node-driver-registrar容器与kubelet通过 Node主机一个hostPath目录下的 unix socket进行通信。
CSI Driver容器与kubelet通过Node主机另一个hostPath目录下的 unix socket 进行通信，同时需要将 kubelet 的工作目录（默认为 /var/lib/kubelet）挂载给CSI Driver容器，用于为Pod进行Volume的管理操作 (包括mount、 umount等）。

03 CSI存储插件应用实战

下面以csi-hostpath插件为例，对如何部署CSI插件、用户如何使用CSI插件提供的存储资源进行详细说明。

3.1 设置Kubernetes服务启动参数

为kube-apiserver、 kube-controller-manager和kubelet服务的启动参数添加如下内容：

--feature-gates=VolumeSnapshotDataSource=true, CSINodeInfo=true, CSIDriverRegistry=true

这3个特性开关是Kubernetes 从1.12版本开始引入的Alpha 版本功能， CSINodelnfo和CSIDriverRegistry需要手工创建其相应的CRD资源对象。

3.2 创建CRD资源对象

需要创建CSINodelnfo和CSIDriverRegistry CRD资源对象。

csidriver.yaml的内容如下：

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition 
metadata:
	name: csidrivers.csi.storage.k8s.io 
	labels:
		addonmanager.kubernetes.io/mode: Reconcile 
spec:
	group: csi.storage.k8s.io
	names:
		kind: CSIDriver
		plural: csidrivers
	scope: Cluster 
	validation: 
		openAPIV3Schema:
			properties:
				spec:
					description: Specification of the CST Driver.
					properties:
						attachRequired:
							description: Indicates this cSI volume driver requires an attach	operation, and that Kubernetes should call attach and wait for any attach operationto complete before proceeding to mount.
							type: boolean
						podInfoOnMountVersion:
							description: Indicates this cSI volume driver requires additional	pod	information (1ike podName, poduID, etc.) during mount operations.
							type: string
	version: v1alpha1

csinodeinfo.yaml的内容如下：

apiVersion: apiextensions.k8s.io/v1betal 
kind: CustomResourceDefinition 
metadata:
	name: csinodeinfos.csi.storage.k8s.io 
	labels:
		addonmanager.kubernetes.io/mode: Reconcile 
spec:
	group: csi.storage.k8s.io 
	names:
		kind: CSINodeInfo
		plural: csinodeinfos
	scope: Cluster 
	validation:
		openAPIV3Schema:
			properties:
				spec:
					description: Specification of CSINodeInfo 
					properties:
						drivers:
							description: List of CSI drivers running on the node and their specs.
							type: array 
							items:
								properties:
									name:
										description: The CSI driver that this obiect refers to.
										type: string 
									nodeID:
										description: The node from the driver point of view.
										type: string 
									topologyKeys:
										description: Tiist of keys supported by the driver.
										items:
											type: string
										type: array
				status:
					description: Status of CSINodeInfo 
					properties:
						drivers:
							description: List of cSI drivers running on the node and their statuses.
							type: array 
							items:
								properties:
									name:
										description: The CSI driver that this obiect refers to.
									type: string 
									available:
										description: Whether the CSI driver is installed.
										type: boolean
									volumePluginMechanism:
										description: Indicates to external components the required mechanism	to use for any in-tree plugins replaced by this driver.
										pattern: in-treelesi
										type: string

	version: v1alpha1

接着使用kubectl create命令完成创建。

3.3 创建csi-hostpath存储插件相关组件

创建csi-hostpath存储插件相关组件，包括csi-hostpath-attacher、 csihostpath-provisioner和csi-hostpathplugin（其中包含csi-node-driver-registrar和 hostpathplugin）。其中为每个组件都配置了相应的RBAC权限控制规则，对于安全访问Kubernetes资源对象非常重要。

csi-hostpath-attacher.yaml的内容如下：

#RBAC 相关配置
---
apiVersion: v1
kind: ServiceAccount
metadata:
	name: csi-attacher
	namespace: default
	
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
	name: external-attacher-runner
rules:
  - apiGroups: [""]
	resources: ["persistentvolumes"]
	verbs: ["get","list","watch","update"] 
  - apiGroups: [""]
	resources: ["nodes"]
	verbs: ["get","list", "watch"]
  - apiGroups: ["csi.storage.k8s.io"] 
    resources: ["csinodeinfos"]
    verbs: ["get","list", "watch"]
  - apiGroups: ["storage.k8s.io"] 
    resources: ["volumeattachments"]
    verbs: ["get", "1ist","watch","update"]
    
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1 
metadata:
	name: csi-attacher-role 
subjects:
	- kind: ServiceAccount
	  name: csi-attacher
	  namespace: default 
roleRef:
	kind: ClusterRole
	name: external-attacher-runner
	apiGroup: rbac.authorization.k8s.io
	
---
# Attacher must be able to work with config map in current namespace 
# if (and only if) leadership election is enalled
kind: Role
apiVersion: rbac.authorization.k8s.io/v1 
metadata:
	# replace with non-default name space name
	namespace: default
	name: external-attacher-cfg 
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get","watch","1ist", "delete", "update", "create"]
  
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1 
metadata:
	name: csi-attacher-role-cfg
	namespace: default 
	subiects:
	  - kind: ServiceAccount
        name: csi-attacher 
        namespace: default
    roleRef:
		kind: Role
		name: external-attacher-cfg
		apiGroup: rbac.authorization.k8s.io

---
# Service 和 StatefulSet 的定义
kind: Service
apiversion: v1 
metadata:
	name: csi-hostpath-attacher 
	labels:
		app: csi-hostpath-attacher 
spec:
	selector:
		app: csi-hostpath-attacher 
	ports:
	  - name: dummy
		port: 12345

---
kind: StatefulSet
apiVersion: apps /v1 
metadata:
	name: csi-hostpath-attacher 
spec:
	serviceName: "csi-hostpath-attacher" 
	replicas: 1 
	selector:
		matchLabels:
			app: csi-hostpath-attacher 
	template:
		metadata:
			labels:
				app: csi-hostpath-attacher 
	    spec:
			serviceAccountName: csi-attacher 
		    containers:
			- name: csi-attacher
			  image: quay.io/k8scsi/csi-attacher:v1.0.1 
			  imagePullPolicy: IfNotPresent 
			  args:
			  	- --v=5
				- --csi-address=$(ADDRESS)
			  env:
			  - name: ADDRESS
                value: /csi/csi.sock 
              volumeMounts:
			  - mountPath: /csi
				name: socket-dir
			volumes:
			  - hostPath:
					path: /var/lib/kubelet/plugins/csi-hostpath 
					type: DirectoryorCreate
				name: socket-dir

csi-hostpath-provisioner.yaml的内容如下：

# RBAC 相关配置
---
apiversion: v1
kind: ServiceAccount 
metadata:
	name: csi-provisioner
	# replace with non-default name space name 
	namespace: default

---
kind: ClusterRole
apiversion: rbac.authorization.k8s.io/v1 
metadata:
	name: external-provisioner-runner 
	rules:
	  - apiGroups: [""]
		resources: ["secrets"]
		verbs: ["get", "list"]
      - apiGroups: [""]
		resources: ["persistentvolumes"]
		verbs: ["get","list","watch","create", "'delete"] 
	  - apiGroups: [""]
		resources: ["persistentvolumeclaims"]
		verbs: ["get","list","watch", "update"] 
	  - apiGroups: ["storage.k8s.io"]
		resources: ["storageclasses"]
		verbs: ["get", "list", "watch"]
	  - apiGroups: [""]
		resources: ["events"]
		verbs: ["1ist","watch", "create", "update","patch"] 
	  - apiGroups: ["snapshot.storage. k8s.io"]
		resources: ["volumesnapshots"]
 		verbs: ["get", "list"]
	  - apiGroups: ["snapshot.storage.k8s.io"]
		resources: ["volumesnapshotcontents"]
		verbs: ["get", "list"]
	  - apiGroups: ["csi.storage.k8s.io"]
		resources: ["csinodein fos"]
		verbs: ["get", "list","watch"]
	  - apiGroups: [""]
		resources: ["nodes"]
		verbs: ["get","list", "watch"]

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1 
metadata:
	name: csi-provisioner-role 
subjeets:
- kind: ServiceAccount
  name: csi-provisioner
  # replace with non-default namespace name
  namespace: default 
roleRef:
	kind: ClusterRole
	name: external-provisioner-runner
	apiGroup: rbac.authorization.k8s.io

---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1 
metadata:
	namespace: default
	name: external-provisioner-cfg 
rules:
	- apiGroups: [""]
	  resources: ["endpoints"]
	  verbs: ["get","watch","list","delete","update","create"]

---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1 
metadata:
	name: csi-provisioner-role-cfg
	namespace: default 
subjects:
  - kind: ServiceAccount
	name: csi-provisioner
	namespace: default
roleRef:
  - kind: Role
    name: external-provisioner-cfg
    apiGroup: rbac.authorization.k8s.io

---
kind: Service
apiVersion: v1 
metadata:
	name: csi-hostpath-provisioner 
	labels:
		app: csi-hostpath-provisioner 
spec:
	seleetor:
		app: csi-hostpath-provisioner 
		ports:
			- name: dummy
			  port: 12345

---
kind: StatefulSet
apiVersion: apps/v1 
metadata:
	name: csi-hostpath-provisioner 
spec:
	serviceName: "csi-hostpath-provisioner" 
	replicas: 1 
	selector:
		matchLabels:
			app: csi-hostpath-provisioner 
    template:
		metadata:
			labels:
				app: esi-hostpath-provisioner 
		spec:
			serviceAccountName: csi-provisioner 
			containers:
		 		- name: csi-provisioner
				  image: quay.io/k8scsi/csi-provisioner:v1.0.1 
				  imagePullPolicy: IfNotPresent 
				  args:
				  	- "--provisioner=csi-hostpath"
					- "--csi-address=$(ADDRESS)"
					- "--connection-timeout=15s"
				  env:
				    - name: ADDRESS
					  value: /csi/csi.sock 
				  volumeMounts:
				  - mountPath: /csi
                    name: socket-dir
			volumes:
			  - hostPath:
					path: /var/lib/kubelet/plugins/csi-hostpath 
					type: DirectoryorCreate
				name: socket-dir

csi-hostpathplugin.yaml的内容如下：

# RBAC 相关配置
apiVersion: v1
kind: ServiceAccount
metadata:
	name: csi-node-sa
	# replace with non-default namespace name
	namespace: default

---
kind: clusterRole
apiVersion: rbac.authorization.k8s.io/v1 
metadata:
	name: driver-registrar-runner 
	rules:
		- apiGroups: [""]
          resources: ["events"]
		  verbs: ["get","list", "watch", "create", "update", "patch"]
		# Kubernetes versions.
        # - apiGroups: [""]
		# resources: ["nodes"]
		# verbs: ["get","update", "patch"〕

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1 
metadata:
	name: csi-driver-registrar-role 
subjeets:
	- kind: ServiceAccount
      name: csi-node-sa
	  # replace with non-default namespace name
	  namespace: default 
roleRef:
	kind: ClusterRole
    name: driver-registrar-runner
    apiGroup: rbac.authorization.k8s.io

# Daemon Set 的定义
---
kind: DaemonSet
apiVersion: apps/v1 
metadata:
	name: csi-hostpathplugin 
spec:
	selector:
		matchLabels:
			app: csi-hostpathplugin 
	template:
		metadata:
			labels:
				app: csi-hostpathplugin 
		spec:
			serviceAccountName: csi-node-sa
			hostNetwork: true 
			containers:
			  - name: driver-registrar
				image: quay.io/k8scsi/csi-node-driver-registrar:v1.0.1 
				imagePullPolicy: IfNotPresent 
				args:
				  - --v=5
				  - --csi-address=/csi/csi.sock
				  - --kubelet-registration-path=/var/lib/kubelet/plugins /csi-hostpath/csi.sock 
				env:
				  - name: KUBE_NODE_NAME 
				  - valueFrom:
						fieldRef:
							apiversion: v1
							fieldPath: spec.nodeName
				volumeMounts:
				  - mountPath: /csi
					name: socket-dir
				  - mountPath: /registration
               		name: registration-dir
			- name: hostpath
			  image: quay.io/k8scsi/hostpathplugin:v1.0.1 
			  imagePullPolicy: IfNotPresent 
			  args:
			    - "--v=5"
				- "--endpoint=$(CSI_ENDPOINT)"
				- "--nodeid=$(KUBE NODE NAME)"
			  env:
			  	- name: CSI_ENDPOINI
				  value: unix:///csi/csi.sock
                - name: KUBE_NODE_NAME 
                  valueFrom:
				  	fieldRef:
						apiVersion: V1
						fieldPath: spec.nodeName
			  securityContext:
				privileged: true 
			  volumeMounts:
				- mountPath: /csi
				  name: socket-dir
				- mountPath: /var/lib/kubelet/pods 
				  mountPropagation: Bidirectional 
				  name: mountpoint-dir
		volumes:
		  - hostPath:
			  path: /var/lib/kubelet/plugins/csi-hostpath 
			  type: DirectoryorCreate
			name: socket-dir
		  - hostPath:
			  path: /var/1ib/kubelet/pods
			  type: DirectoryorCreate
			name: mountpoint-dir 
		  - hostPath:
				path: /var/lib/kubelet/plugins_registry 
				type: Directory
			name: registration-dir

然后使用kubectl create命令去创建csi-hostpath-attacher.yaml、csi-hostpath-provisioner.yaml、csi-hostpathplugin.yaml。

至此就完成了CSI存储插件的部署。

3.4 应用容器使用CSI存储

应用程序如果希望使用CSI存储插件提供的存储服务，则仍然使用Kubernetes动态存储管理机制。首先通过创建 StorageClass和PVC为应用容器准备存储资源，然后容器就可以挂载PVC到容器内的目录下进行使用了。

创建一个StorageClass，provisioner为CSI存储插件的类型，在本例中为csihostpath：

# csi-storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: Storageclass 
metadata:
	name: csi-hostpath-sc
provisioner: csi-hostpath
reclaimPolicy: Delete
volumeBindingMode: Immediate

# 创建
# kubectl create -f csi-storageclass . yaml
storageclass.storage.k8s.io/csi-hostpath-sc created

创建一个PVC，引用刚刚创建的StorageClass，申请存储空间为1GiB：

# csi-pvc.yaml
apiVersion: V1
kind: PersistentVolumeClaim
metadata:
	name: csi-pvc
spec:
	accessModes:
		ReadwriteOnce
resources:
	requests:
		storage: 1Gi
storageClassName: csi-hostpath-sc

# kubectl create -f csi-pve.yaml 
persistentvolumeclaim/csi-pve created

查看PVC和系统自动创建的PV，状态为Bound，说明创建成功：

$ kubectl get pvc

NAME 	STATUS 	VOLUME										CAPACITY 		ACCESS 	MODES			STORAGECLASS AGE
csi-pve Bound 	pve-f8923093-3e25-11e9-a5fa-000c29069202 	1Gi 			RNO 	csi-hostpath-sc				 40s

$ kubectl get pv
NAME										CAPACITY  ACCESS MODES RECLAIM POLICY	STATUS CLAIM   STORAGECLASS  REASON          AGE
pve-£8923093-3e25-11e9-a5fa-000c29069202    1Gi        RWO          Delete       Bound  default /csi-pvc      csi-hostpath-sc 42S

最后，在应用容器的配置中使用该PVC：

# csi-app. yaml
kind: Pod
apiVersion: v1 
metadata:
	mame: my-csi-app 
spec:
	contaiiners:
		name: my-csi-app
		image: busybox
		imagePullPolicy: IfNotPresent 
		command: ["sleep", "1000000"] 
		volumeMounts:
		- mountPath:"/data"
		  name: my-csi-volume
	volumes:
	  - name: my-csi-volume 
	    persistentVolumeClaim:
			claimName: csi-pvc

# kubectl create -f csi-app.yaml
# pod/my-csi-app created

# kubect1 get pods

NAME			READY 		STATUS 		RESTARTS AGE

my-csi-app		1/1 		Running       0      40s

在Pod创建成功之后，应用容器中的/data目录使用的就是CSI存储插件提供的存储。

k8s教程（Volume篇）-CSI存储机制详解