二进制部署K8s集群第27节之helm踩坑部署harbor

一、创建PVC

需要先部署NFS
https://blog.51cto.com/yht1990/2630775《storageClass动态挂载对接NFS存储》

kubectl create ns harbor
cat > harbor-pvc.yaml <<'eof'
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: harbor-pvc
  namespace: harbor
spec:
  accessModes:     
    - ReadWriteOnce
  storageClassName: storage-nfs 
  resources:
    requests:
      storage: 20Gi
eof
kubectl apply -f harbor-pvc.yaml

二、拉取harbor的chart到本地

[root@k8s-master harbor]# helm repo add harbor https://helm.goharbor.io
[root@k8s-master harbor]# helm repo update
[root@k8s-master harbor]# helm search repo harbor
NAME            CHART VERSION   APP VERSION     DESCRIPTION                                       
harbor/harbor   1.5.1           2.1.1           An open source trusted cloud native registry th...
[root@k8s-master harbor]# helm repo ls
NAME    URL                                      
stable  http://mirror.azure.cn/kubernetes/charts/
harbor  https://helm.goharbor.io  
[root@k8s-master harbor]# helm pull harbor/harbor --version 1.5.1

三、Chart参数设置

生产环境size必须调大

[root@k8s-master harbor]# tar xf harbor-1.5.1.tgz
[root@k8s-master harbor]# cd harbor
[root@k8s-master harbor]#  cp values.yaml values.yaml.bak
[root@k8s-master harbor]#  vim values.yaml 
...
 36       core: harbor.od.com
...
101 externalURL: https://harbor.od.com  # 设置访问域名
...
108 externalURL: https://harbor.od.com
185 193   persistentVolumeClaim:
194     registry:
195       # Use the existing PVC which must be created manually before bound,
196       # and specify the "subPath" if the PVC is shared with other components
197       existingClaim: "harbor-pvc"
198       # Specify the "storageClass" used to provision the volume. Or the default
199       # StorageClass will be used(the default).
200       # Set it to "-" to disable dynamic provisioning
201       storageClass: ""
202       subPath: "registry"
203       accessMode: ReadWriteOnce
204       size: 5Gi
205     chartmuseum:
206       existingClaim: "harbor-pvc"
207       storageClass: ""
208       subPath: "chartmuseum"
209       accessMode: ReadWriteOnce
210       size: 5Gi
211     jobservice:
212       existingClaim: "harbor-pvc"
213       storageClass: ""
214       subPath: "jobservice"
215       accessMode: ReadWriteOnce
216       size: 1Gi
217     # If external database is used, the following settings for database will
218     # be ignored
219     database:
220       existingClaim: "harbor-pvc"
221       storageClass: ""
222       subPath: "database"
223       accessMode: ReadWriteOnce
224       size: 1Gi
225     # If external Redis is used, the following settings for Redis will
226     # be ignored
227     redis:
228       existingClaim: "harbor-pvc"
229       storageClass: ""
230       subPath: "redis"
231       accessMode: ReadWriteOnce
232       size: 1Gi
233     trivy:
234       existingClaim: "harbor-pvc"
235       storageClass: ""
236       subPath: "trivy"
237       accessMode: ReadWriteOnce
238       size: 5Gi
 ...
539 clair:
540   enabled: false
 ...
569 trivy:
570   # enabled the flag to enable Trivy scanner
571   enabled: false
  ...
626 notary:
627   enabled: false
 ...

四、踩坑一

redis持久化数据目录权限导致无法登录
redis数据目录，/var/lib/redis，需要设置redis的用户及用户组权限

/root/harbor/templates/redis/statefulset.yaml
      initContainers:
      - name: "change-permission-of-directory"
        securityContext:
          runAsUser: 0
        image: {{ .Values.database.internal.image.repository }}:{{ .Values.database.internal.image.tag }}
        imagePullPolicy: {{ .Values.imagePullPolicy }}
        command: ["/bin/sh"]
        args: ["-c", "chown -R 999:999 /var/lib/redis"]
        volumeMounts:
        - name: data
          mountPath: /var/lib/redis
          subPath: {{ $redis.subPath }}

五、踩坑二

registry组件的镜像存储目录权限导致镜像推送失败
registry的镜像存储目录，需要设置registry用户的用户及用户组，不然镜像推送失败

/root/harbor/templates/registry/registry-dpl.yaml
      initContainers:
      - name: "change-permission-of-directory"
        securityContext:
          runAsUser: 0
        image: {{ .Values.database.internal.image.repository }}:{{ .Values.database.internal.image.tag }}
        imagePullPolicy: {{ .Values.imagePullPolicy }}
        command: ["/bin/sh"]
        args: ["-c", "chown -R 10000:10000 {{ .Values.persistence.imageChartStorage.filesystem.rootdirectory }}"]
        volumeMounts:
        - name: registry-data
          mountPath: {{ .Values.persistence.imageChartStorage.filesystem.rootdirectory }}
          subPath: {{ .Values.persistence.persistentVolumeClaim.registry.subPath }}

六、踩坑三

chartmuseum存储目录权限，导致chart推送失败

/root/harbor/templates/chartmuseum/chartmuseum-dpl.yaml
      initContainers:
      - name: "change-permission-of-directory"
        securityContext:
          runAsUser: 0
        image: {{ .Values.database.internal.image.repository }}:{{ .Values.database.internal.image.tag }}
        imagePullPolicy: {{ .Values.imagePullPolicy }}
        command: ["/bin/sh"]
        args: ["-c", "chown -R 10000:10000 /chart_storage"]
        volumeMounts:
        - name: chartmuseum-data
          mountPath: /chart_storage
          subPath: {{ .Values.persistence.persistentVolumeClaim.chartmuseum.subPath }}

七、安装harbor

cd 
helm install harbor ./harbor -n harbor
helm -n harbor ls
kubectl -n harbor get po

八、配置访问推送

8.1、域名配置

dns服务器或者hosts里配置

ip harbor.od.com

8.2、配置docker daemon

cat /etc/docker/daemon.json
  "insecure-registries": [  
 "harbor.od.com"   
  ],       
systemctl restart docker

8.3 推送chart

使用账户密码登录admin/Harbor12345

docker login harbor.od.com
helm plugin install https://github.com/chartmuseum/helm-push
helm plugin ls
kubectl get secret harbor-harbor-ingress -n harbor -o jsonpath="{.data.ca\.crt}" | base64 -d >harbor.ca.crt
cp harbor.ca.crt /etc/pki/ca-trust/source/anchors
update-ca-trust enable; update-ca-trust extract
helm repo add myharbor https://harbor.od.com/chartrepo/library --ca-file=harbor.ca.crt
helm repo ls
helm push harbor myharbor --ca-file=harbor.ca.crt -u admin -p Harbor12345