Scope & Not-in-scope
- document the setup process of PVC/PV upon Ceph Rook cluster, for learning purpose only
- list out some error met during #1
- K8S related concepts & knowledges is not in this blog scope, you can find it here(https://kubernetes.io/)
steps:
1. Create Ceph rook cluster
git clone https://github.com/rook/rook.git
cd /cluster/examples/kubernetes/ceph
kubectl create -f operator.yaml
# verify the rook-ceph-operator, rook-ceph-agent, and rook-discover pods are in the `Running` state before proceeding
kubectl -n rook-ceph-system get pod
#Create a Rook Cluster
kubectl create -f cluster.yaml
#Use kubectl to list pods in the rook namespace. You should be able to see the following pods once they are all running. The number of osd pods will depend on the number of nodes in the cluster and the number of devices and directories configured
kubectl -n rook-ceph get pod
2. Provision Storage - Block
Block storage allows you to mount storage to a single pod, and there are three types of storage exposed by Rook, in our case let try to use block storage for demostration purpose.
- Block: Create block storage to be consumed by a pod
- Object: Create an object store that is accessible inside or outside the Kubernetes cluster
- Shared File System: Create a file system to be shared across multiple pods
of course we can use the provided examples, after run the below command, k8s will help to create CephBlockPool - replicapool and StorageClass - rook-ceph-block
kubectl create -f storageclass.yaml
3. Create a Statefulset in which volumeClaimTemplates been predefined and to apply the block storage into pods going to create
to create yaml file - statefulset-pvc.yaml, it's content like below
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: web
spec:
serviceName: "nginx"
replicas: 2
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.15.4
ports:
- containerPort: 80
name: web
volumeMounts:
- name: web
mountPath: /usr/share/nginx/html
volumeClaimTemplates:
- metadata:
name: web
spec:
storageClassName: rook-ceph-block
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
use statefulset-pvc.yaml to create pods
kubectl create -f statefulset-pvc.yaml
4. Check pods, Persitent Volume Claim, Persistent Volume, and Stroage bind status
# check pod
kubectl get pod
# check persistent volume claim
kubectl get pvc
# check persitent volume
kubectl get pv
Q&A
Rook cluster creation
- after exec the 'kubectl apply -f cluster.yaml', got 'Init:CrashLoopBackOff'
CrashLoopBackOff.png
[Solution]
check the pod status using 'kubectl describe pod rook-ceph-mon-a-788ff56499-87hzx --namespace=rook-ceph'
you will find the error message:
The keyring does not match the existing keyring in /var/lib/rook/mon-a/data/keyring. You may need to delete the contents of dataDirHostPath on the host from a previous deployment.
RootCause-CrashLoopBachOff.pngon all worker node,go to the dataDirHostPath , namely /var/lib/rook, clean this folder with below command, after removal, related pods/container will auto retry and recover
cd /var/lib/rook
rm -rf *
btw, refer to (https://rook.github.io/docs/rook/master/ceph-quickstart.html)