k8s 1.19.2使用ceph作存储后端

这份文档各组件系统

• ceph：15.2.5
• k8s: 1.19.2
• os: centos7 7.8.2003

  在所有k8s node节点上安装ceph-common

  不需要节点有/etc/ceph/目录下的文件

  yum install -y ceph-common

  创建pool

  ceph osd pool create kube
  rbd pool init kube

创建用户

ceph auth get-or-create \
  client.kube mon 'allow r' \
  osd 'allow class-read object_prefix rbd_children, allow rwx pool=kube' \
  -o /etc/ceph/ceph.client.kube.keyring

创建namespace rbd用于部署csi

kubectl create namespace rbd

创建csi configmap

ceph-csi当前仅支持旧版V1协议。

• 查看mon和clusterID

ceph mon dump

• 创建configmap

cat <<EOF > csi-config-map.yaml
---
apiVersion: v1
kind: ConfigMap
data:
  config.json: |-
    [
      {
        "clusterID": "01c1e960-fbce-11ea-a90e-52540079f755",
        "monitors": [
          "192.168.254.103:6789",
          "192.168.254.103:6789",
          "192.168.254.103:6789"
        ]
      }
    ]
metadata:
  name: ceph-csi-config
  namespace: rbd
EOF

创建ksm configmap，新版本需要这个

参看https://github.com/ceph/ceph-csi/issues/834

cat <<EOF>>kms-config.yaml
---
apiVersion: v1
kind: ConfigMap
data:
  config.json: |-
    {
    }
metadata:
  name: ceph-csi-encryption-kms-config
EOF

kubectl -n rbd create -f  kms-config.yaml

创建csi访问ceph的secret

cat <<EOF > csi-rbd-secret.yaml
---
apiVersion: v1
kind: Secret
metadata:
  name: csi-rbd-secret
  namespace: rbd
stringData:
  userID: kube
  userKey: AQDFhmlfVphsGhAAh8f0Ck3gREjilODu4Sz/Zw==
EOF

配置csi插件

创建rbac

• 下载rbac文件

wget  https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-provisioner-rbac.yaml
wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-nodeplugin-rbac.yaml

• 将rbac中的namespace: default替换为namespace: rbd

sed -i -E 's/(^\s+namespace:)(.*)/\1 rbd/g' csi-provisioner-rbac.yaml
sed -i  -E 's/(^\s+namespace:)(.*)/\1 rbd/g' csi-nodeplugin-rbac.yaml

• apply rbac

kubectl -n rbd apply -f csi-provisioner-rbac.yaml
kubectl -n rbd apply -f csi-nodeplugin-rbac.yaml

创建ceph-csi Provisioner和Node插件

• 下载插件文件

wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin-provisioner.yaml

wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin.yaml

• apply plugin

kubectl -n rbd apply -f  csi-rbdplugin-provisioner.yaml 
kubectl -n rbd apply -f   csi-rbdplugin.yaml

创建storageClass

注意增加fsType、imageFormat、imageFeatures
imageFeatures是一定要增加的，否则centos7上不能正常挂载rbd

cat <<EOF > csi-rbd-sc.yaml
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: csi-rbd-sc
provisioner: rbd.csi.ceph.com
parameters:
   clusterID: 01c1e960-fbce-11ea-a90e-52540079f755
   pool: kube
   fsType: xfs
   imageFormat: "2"
   imageFeatures: "layering"
   csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
   csi.storage.k8s.io/provisioner-secret-namespace: rbd
   csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
   csi.storage.k8s.io/node-stage-secret-namespace: rbd
reclaimPolicy: Delete
mountOptions:
   - discard
EOF
kubectl apply -f storageClass.yaml 

以block模式挂载

这种模式是直接将rbd设备以block模式挂载容器中，使用时需要进入容器格式，这种例子中将rbd设备挂载到/dev/xvda
这种模式下，进入容器后是没有权限在/dev/xvda上创建文件系统、执行mount操作

创建pvc

cat <<EOF > raw-block-pvc.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: raw-block-pvc
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Block
  resources:
    requests:
      storage: 1Gi
  storageClassName: csi-rbd-sc
EOF
kubectl -n rbd apply -f raw-block-pvc.yaml

创建pod

cat <<EOF > raw-block-pod.yaml
---
apiVersion: v1
kind: Pod
metadata:
  name: pod-with-raw-block-volume
spec:
  containers:
    - name: fc-container
      image: centos:7
      command: ["/bin/sh", "-c"]
      args: ["tail -f /dev/null"]
      volumeDevices:
        - name: data
          devicePath: /dev/xvda
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: raw-block-pvc
EOF
kubectl -n rbd apply -f raw-block-pod.yaml

以文件系统方式使用

这种模式会先将rbd设备格式化后再挂载给容器

创建pvc

cat <<EOF > pvc.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rbd-pvc
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 1Gi
  storageClassName: csi-rbd-sc
EOF
kubectl -n rbd apply -f pvc.yaml

创建Pod

cat <<EOF > pod.yaml
---
apiVersion: v1
kind: Pod
metadata:
  name: csi-rbd-demo-pod
spec:
  containers:
    - name: web-server
      image: nginx
      volumeMounts:
        - name: mypvc
          mountPath: /var/lib/www/html
  volumes:
    - name: mypvc
      persistentVolumeClaim:
        claimName: rbd-pvc
        readOnly: false
EOF
kubectl -n rbd apply -f pod.yaml

参考 文档

https://docs.ceph.com/en/latest/rbd/rbd-kubernetes/