gitlabrunner使用k8s executor

相关的文章有：gitlab-runner使用docker executor
两者主要区别在于，一个部署在docker服务器上，一个部署在k8s的pod中。

所需镜像

基础镜像

gitlab-runner:latest
- gitlab-runner打包成的镜像，需要在k8s上长期驻留
gitlab-runner-helper:x86_64-f100a208
- 这个是gitlab-runner用来启动pod的，他的版本由gitlab-runner版本和服务器版本决定
maven:3-jdk-8
- 如果处理的是java代码，那此镜像既可以做为编译的镜像，也可以作为发布时的基础镜像。
busybox:latest
- 默认镜像，如果不指定runner启动哪个镜像，就选择busybox
docker:dind
- 用于编译业务镜像，基于maven:3-jdk-8
kubectl
- 将业务镜像部署到k8s上

第一步：在K8S中部署gitlab-runner

在gitlab上找到url和token

打开你想要配置CICD的项目或项目组
runner配置地址：settings->CI/CD->Runners->expand
找到设置runner需要的url和token，拿小本本记下来

url和token

创建一个可用的runner

找一台安装了docker的机器，使用以下命令生成toml：docker run -it --name runner gitlab/gitlab-runner:v11.10.0 register

register过程

在settings->CI/CD->Runners->expand下找到新增的runner，获得新tokenkg8sKRsXs9kYWsMp5PJ7，记在小本本上。而上一步启动的runner已经不再需要了。

创建k8s上的gitlab-runner的配置文件

注意：这里的token使用我们新获得的token。

kind: ConfigMap
apiVersion: v1
metadata:
  name: gitlab-runner
  namespace: gitlab-runner
data:
  config.toml: |
    concurrent = 4
    check_interval = 0

    [session_server]
      session_timeout = 1800

    [[runners]]
      name = "first test"
      url = "http://${
    
    gitlabIP}"
      token ="......"
      executor = "kubernetes"
      [runners.kubernetes]
        host = ""
        bearer_token_overwrite_allowed = true
        privileged = true
        #这是runner的默认镜像；具体镜像maven:3-jdk-8在.gitlab-ci.yml中配置
        image = "${
    
    registryIP}/gitlab-runner/busybox:latest"  
        namespace = "gitlab-runner"
        namespace_overwrite_allowed = ""
        cpu_limit = "1"
        memory_limit = "2Gi"
        cpu_request = "1"
        memory_request = "2Gi"
        service_cpu_limit = "1"
        service_memory_limit = "2Gi"
        service_cpu_request = "1"
        service_memory_request = "2Gi"
        helper_cpu_limit = "1"
        helper_memory_limit = "2Gi"
        helper_cpu_request = "1"
        helper_memory_request = "2Gi"
        service_account = "gitlab"
        helper_image = "${
    
    registryIP}/gitlab-runner/gitlab-runner-helper:x86_64-f100a208"
        [[runners.kubernetes.volumes.config_map]]
          name = "gitlab-runner-maven"
          mount_path = "/usr/share/maven/configmap/"

创建configmap：gitlab-runner-maven，保存maven的setting文件

kind: ConfigMap
apiVersion: v1
metadata:
  name: gitlab-runner-maven
  namespace: gitlab-runner
data:
  settings.xml: "此处是maven的settings.xml的内容"

创建serviceaccount.yml 实现权限，否则提示无法在namespace中创建pod

apiVersion: v1
kind: ServiceAccount
metadata:
  name: gitlab
  namespace: gitlab-runner
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: gitlab
  namespace: gitlab-runner
rules:
- apiGroups: [""]
  resources: ["pods","pods/exec"]
  verbs: ["get","list","watch","create","update","patch","delete"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: gitlab
  namespace: gitlab-runner
subjects:
- kind: ServiceAccount
  name: gitlab
  apiGroup: ""
roleRef:
  kind: Role
  name: gitlab
  apiGroup: rbac.authorization.k8s.io

通过deployment.yml部署gitlab-runner

注意写入serviceaccountname gitlab
注意导入configmap，即gitlab-runner的/etc/gitlab-runner/config.toml
configmap通过[[runners.kubernetes.volumes.config_map]]，将maven的settings挂载到了maven:3-jdk-8容器
注意config.toml中token和cicd/runner的token并不一样

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: gitlab-runner
  namespace: gitlab-runner
spec:
  replicas: 1
  selector:
    matchLabels:
      name: gitlab-runner
  template:
    metadata:
      labels:
        name: gitlab-runner
    spec:
      serviceAccountName: gitlab
      containers:
      - args:
        - '--debug'
        - run
        image: ${
    
    registryIP}/gitlab-runner/gitlab-runner:latest
        imagePullPolicy: IfNotPresent
        name: gitlab-runner
        volumeMounts:
        - mountPath: /etc/gitlab-runner
          name: config
        - mountPath: /etc/ssl/certs
          name: cacerts
          readOnly: true
        resources:
          limits:
            cpu: "2"
            memory: "4Gi"
          requests:
            cpu: "2"
            memory: "4Gi"
      restartPolicy: Always
      volumes:
      - configMap:
          name: gitlab-runner
        name: config
      - hostPath:
          path: /usr/share/ca-certificates/mozilla
        name: cacerts

第二步：配置.gitlab-ci.yml

选择maven模板
在 MVAEN_CLI_OPTS 中加入"–settings /usr/share/maven/configmap/settings.xml"
简单示例，打JAR包、打Docker镜像、发布到k8s：

variables:
  MAVEN_OPTS: "-Dhttps.protocols=TLSv1.2 -Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=WARN -Dorg.slf4j.simpleLogger.showDateTime=true -Djava.awt.headless=true"
  MAVEN_CLI_OPTS: "--batch-mode --errors --fail-at-end --show-version -DinstallAtEnd=true -DdeployAtEnd=true --settings /usr/share/maven/configmap/settings.xml"

## 编译并打包
compile:
  stage: build
  script:
    - 'mvn $MAVEN_CLI_OPTS package'
  image: ${
    
    registryIP}/gitlab-runner/maven:3-jdk-8
  artifacts:
    name: "jars"
    expire_in: 60 mins
    paths:
      - target/*.jar
    when: on_success

## 拿到jar包，并编译docker镜像，这里使用一些伪脚本
build_docker_image:
  stage: build_docker_image
  image: ${
    
    registryIP}/gitlab-runner/docker:1.12.6
  services:
    - name: ${
    
    registryIP}/gitlab-runner/docker:1.12.6-dind
      command: ["--insecure-registry=${registryIP}"]
  dependencies:
    - compile
  variables:
    GIT_STRATEGY: none
    DOCKER_HOST: "tcp:127.0.0.1:2375"
  script:
    - 'docker build -t mybiz-${branch}:${CI_PIPELINE_ID} .'
    - 'docker login xxxx '
    - 'docker push xxxx '

## 部署到k8s，这里使用一些伪脚本
deploy_k8s:
  stage: deploy_k8s
  image: ${
    
    registryIP}/gitlab-runner/kubectl:latest
  script:
    - 'create deployment.yml'
    - 'kubectl apply -f deployment.yml'
  only:
    refs:
      - tags
    variables:
      - ${
    
    CI_PROJECT_NAMESPACE} =~ /mygroup/
  variables:
    GIT_STRATEGY: none
  dependencies: []

第三步：测试

打开runner配置地址，可以看到刚才创建的runner，以及连接状态，修改为无需指定tag。

如果你的commit信息中包含 [ci skip] 或者 [skip ci] ，不论大小写，将会跳过jobs。
gitlab -> 项目（组）xxxxx -> CI/CD -> pipelines
- 直接create pipeline就可以了
提示"exceeded quota"，需要在config.toml中设定limit、request等参数
提示"pod limits=0"，serviceaccount没设置好

第四步：Auto-Devops

修改gitlab-CI模板
- 找到模板目录，进行相应的修改。
- 建议修改所有的模板，并在通用模板中提示用户改变模板。不同gitlab版本，地址略有差异：
- vi /opt/gitlab/embedded/service/gitlab-rails/vendor/gitlab-ci-yml/Auto-DevOps.gitlab-ci.yml。
vi /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml

第五步：优化

上面只是比较简单的一个流程，后面还会把一些代码质量、单元测试、自动测试、甚至可以把其他的一些第三方平台也集成进来，更可以做成一个管理平台，将ci.yml都管理起来。

gitlab-runner使用kubernetes executor

文章目录