centos 7用kubeasz和ansible进行二进制部署k8s和升级，k8s实现Nginx+Tomcat+NFS实现动静目录分离和数据持久化

1、centos 7用kubeasz和ansible进行二进制部署k8s

kubeasz下载：kubeasz
kubeasz：kubeasz安装方式
网站上定义了kubeasz和kubernetes的版本对应，按照要求对应安装

部署需要3台设备以上，CPU最好双核以上

角色	IP
ansible	192.168.116.130
k8s-master	192.168.116.132
k8s-node	192.168.116.133

1.1设备都初始化配置

cd /etc/yum.repos.d/
yum install -y wget
wget http://mirrors.aliyun.com/repo/Centos-7.repo
wget http://mirrors.aliyun.com/repo/epel-7.repo
mv CentOS-Base.repo CentOS-Base.repo.bak
yum clean all
yum makecache
systemctl stop firewalld
systemctl disable firewalld
sed -i 's/SELINUX=enforcing$/SELINUX=disabled/g' /etc/selinux/config
setenforce 0

1.2 ansible配置：

hostnamectl set-hostname ansible-130

vi /etc/hosts
192.168.116.130 ansible-130
192.168.116.132 k8s-master
192.168.116.133 k8s-node1

设置秘钥连接和复制host文件
ssh-keygen
ssh-copy-id [email protected]
ssh-copy-id [email protected]
scp /etc/hosts [email protected]:/etc/hosts
scp /etc/hosts [email protected]:/etc/hosts

下载脚本，自动下载k8s需要的安装包
yum install -y python git python-pip ansible python-netaddr  -y
curl -C- -fLO --retry 3 https://github.com/easzlab/kubeasz/releases/download/2.2.0/easzup

在脚本定义了下载的安装包版本，可以自己手动修改，但是需要注意kubeasz和kubernetes的版本对应关系
vi easzup

export DOCKER_VER=19.03.5
export KUBEASZ_VER=2.2.0
export K8S_BIN_VER=v1.17.2
export EXT_BIN_VER=0.4.0
export SYS_PKG_VER=0.3.3

chmod +x ./easzup
./easzup -D

配置安装的版本
cd /etc/ansible/
cp example/hosts.multi-node ./hosts

设置master和node的IP和软件配置
vi hosts   修改配置

#设置etcd的IP
[etcd]
192.168.116.132 NODE_NAME=etcd1

# master node(s) #设置master节点的IP
[kube-master]
192.168.116.132

# work node(s)  #设置node节点的IP
[kube-node]
192.168.116.133

# K8S Service CIDR, not overlap with node(host) networking  #k8s集群管理IP段
SERVICE_CIDR="10.0.0.0/16"

# Cluster CIDR (Pod CIDR), not overlap with node(host) networking  #k8s集群容器IP段
CLUSTER_CIDR="20.0.0.0/16"

# NodePort Rang #设置node对外开放端口
NODE_PORT_RANGE="20000-40000"

# Cluster DNS Domain #设置DNS域名
CLUSTER_DNS_DOMAIN="cluster.local."

ansible all -m ping    测试ansible到节点的网络是否正常

开始安装k8s
ansible-playbook 01.prepare.yml 
ansible-playbook 02.etcd.yml 
ansible-playbook 03.docker.yml
ansible-playbook 04.kube-master.yml
ansible-playbook 05.kube-node.yml
ansible-playbook 06.network.yml

1.3在master查看k8s状态

hostnamectl set-hostname k8s-master
kubectl get pod -A
kubectl get nodes

运行容器测试k8s能否正常工作
kubectl run net-test --image=alpine --replicas=1 sleep 360000 
kubectl get pod -A

可以看到pod的的IP和node的IP
kubectl get pod -A -o wide

1.4 在master安装coredns，解析外部网站域名

wget https://storage.googleapis.com/kubernetes-release/release/v1.17.2/kubernetes.tar.gz
cd /root/kubernetes/cluster/addons/dns/coredns/
cp coredns.yaml.base coredns.yml
vi coredns.yml       修改配置

#原本这行  kubernetes __PILLAR__DNS__DOMAIN__ in-addr.arpa ip6.arpa，修改为下面，这里的cluster.local.对应上面ansible的hosts的DNS配置
kubernetes cluster.local. in-addr.arpa ip6.arpa

#修改forward . /etc/resolv.conf，指定外网的DNS的IP地址 
forward . 223.5.5.5

# image: k8s.gcr.io/coredns:1.6.5   修改镜像源
image: coredns/coredns:1.6.7
 
#修改容器使用内存，最好1G以上  memory: __PILLAR__DNS__MEMORY__LIMIT__
memory: 1Gi

#设置DNS容器的IP  clusterIP: __PILLAR__DNS__SERVER__，对应上面ansible的hosts的集群管理service网段的IP
clusterIP: 10.0.0.2

kubectl apply -f coredns.yml

测试容器能否连接外网解析域名

kubectl run net-test1 --image=alpine --replicas=1 sleep 360000 
kubectl get pod -A
kubectl exec -it net-test1-5fcc69db59-s8r2z /bin/sh

2、centos 7用kubeasz升级k8s

按照上面搭建k8s集群后，查看k8s版本为1.17.2
K8S版本下载：K8S源码包
选择这个，查看详细的源码包

选择server，注意CPU型号架构，根据设备的cpu型号下载

master查看版本

kubectl version

master关闭k8s服务

systemctl stop kube-apiserver.service kube-controller-manager.service kubelet.service kube-proxy.service kube-scheduler.service

node关闭k8s服务

systemctl stop kubelet.service kube-proxy.service

ansible配置

mkdir /backup         创建老文件备份目录
cd /etc/ansible/bin/
mv kube-apiserver kube-controller-manager kubectl kubelet kube-proxy kube-scheduler /backup/

cd /tmp   下载新的文件到目录
wget https://storage.googleapis.com/kubernetes-release/release/v1.19.7/kubernetes-server-linux-amd64.tar.gz
tar xf kubernetes-server-linux-amd64.tar.gz
cd kubernetes/server/bin

默认新的文件都有执行权限，没有手动加上就行
cp kube-apiserver kube-controller-manager kube-proxy kubectl kubelet kube-scheduler /etc/ansible/bin/

默认kubeasz会自动安装easzctl，这个专门用来升级版本
easzctl upgrade  开始升级master和node

在master和node查看新的版本是否安装成功

kubectl version

master运行容器测试

kubectl run net-test1 --image=alpine --replicas=1 sleep 360000
kubectl get pod -A -o wide

3、k8s实现Nginx+Tomcat+NFS实现动静目录分离和数据持久化

拓扑图：

实现方式：
将静态文件保留在http://192.168.116.133/目录下，容器为nginx容器
将动态文件保留在http://192.168.116.133/tomcat目录下，容器为tomcat容器
nginx和tomcat容器实际都是共用同一个NFS服务器

3.1 搭建NFS服务器配置

NFS服务器配置

yum install -y nfs-utils
mkdir /data   创建共享目录
mkdir /data/k8s/tomcat -p
mkdir /data/k8s/nginx -p

vi /etc/exports
/data *(rw,no_root_squash)

systemctl start nfs
systemctl enable nfs

showmount -e 192.168.116.130   查看是否共享成功

k8s-master 和全部node节点测试能否连接

yum install  nfs-utils -y
showmount -e 192.168.116.130   查看是否共享成功

3.2 master配置nginx的dockerfile镜像

mkdir /dockerfile/nginx -p
cd /dockerfile/nginx/

下载nginx源码包
wget http://nginx.org/download/nginx-1.18.0.tar.gz

vi nginx.conf   配置nginx的配置文件
user nginx;
daemon off;
worker_processes auto;
error_log /app/nginx/logs/nginx-error.log;
pid /app/nginx/run/nginx.pid;
events {
    
    
    worker_connections 10240;
}
http {
    
    
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
    access_log  /app/nginx/logs/nginx-access.log  main;
    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 4096;
server {
    
    
    listen       80;
    server_name  k8s-nginx;
    location / {
    
    
            root   html;
            index  index.html index.htm;
    }
    location /tomcat {
    
    
            proxy_pass http://k8s-tomcat-1-service:8080/;
    }
  }
}

vi nginx-dockerfile

FROM centos:7.7.1908
RUN useradd -s /sbin/nologin nginx
#nginx
RUN mkdir /app
RUN yum install -y psmisc net-tools wget gcc  pcre-devel zlib-devel make
ADD nginx-1.18.0.tar.gz /app/
RUN cd /app/nginx-1.18.0/ && ./configure --prefix=/app/nginx  --user=nginx\
--user=nginx\
--group=nginx\
--with-http_ssl_module\
--with-http_v2_module\
--with-http_realip_module\
--with-http_stub_status_module\
--with-http_gzip_static_module\
--with-pcre\
--with-stream\
--with-stream_ssl_module\
--with-stream_realip_module && make && make install
RUN mkdir /app/nginx/run
ENV PATH=$PATH:/app/nginx/sbin
RUN chown nginx.nginx -R /app/
COPY nginx.conf /app/nginx/conf/
EXPOSE 80
CMD ["nginx"]

制作nginx镜像
docker build -t test/nginx:v1.0 -f nginx-dockerfile .

3.3 master配置tomcat的dockefile镜像

mkdir /dockerfile/tomcat
cd /dockerfile/tomcat/

下载jdk包
wget https://repo.huaweicloud.com/java/jdk/8u191-b12/jdk-8u191-linux-x64.tar.gz

下载tomcat包
wget https://mirrors.bfsu.edu.cn/apache/tomcat/tomcat-8/v8.5.77/bin/apache-tomcat-8.5.77.tar.gz --no-check-certificate

设置jdk路径
vi tomcat.conf
JAVA_HOME=/usr/local/jdk

设置tomcat主页
vi index.jsp
<%@ page language="java" contentType="text/html; charset=UTF-8"
    pageEncoding="UTF-8"%>
<!DOCTYPE html>
<html>
<head>
    <meta charset="utf-8">
    <title>tomcat-test</title>
</head>
<body>
<h1>tomcat-test</h1>
<%
out.println("test");
%>
<br>
<%=request.getRequestURL()%>
</body>
</html>

配置镜像dockerfile
vi tomcat-dockerfile

FROM centos:7.7.1908
RUN yum install -y wget curl psmisc net-tools
#jdk
COPY jdk-8u191-linux-x64.tar.gz /
ADD jdk-8u191-linux-x64.tar.gz /usr/local/
RUN ln -s /usr/local/jdk1.8.0_191 /usr/local/jdk
ENV JAVA_HOME=/usr/local/jdk
ENV PATH=$PATH:$JAVA_HOME/bin
ENV JRE_HOME=$JAVA_HOME/jre
ENV CLASSPATH=$JAVA_HOME/lib/:$JRE_HOME/lib

#tomcat
COPY apache-tomcat-8.5.77.tar.gz /
ADD apache-tomcat-8.5.77.tar.gz /usr/local/
RUN ln -s /usr/local/apache-tomcat-8.5.77 /usr/local/tomcat
ENV PATH=$PATH:/usr/local/tomcat/bin
COPY tomcat.conf /usr/local/tomcat/conf/
COPY index.jsp /usr/local/tomcat/webapps/ROOT/
EXPOSE 8080
CMD ["/usr/local/tomcat/bin/catalina.sh","run" ]

制作tomcat镜像
docker build -t test/tomcat:v1.0 -f tomcat-dockerfile .

前台运行测试镜像能否使用
docker run --rm -it  -p 8080:8080 test/tomcat:v1.0
curl 192.168.116.132:8080

3.4 master制作K8S的容器文件

将k8s-master制作好的镜像保存传送到node节点，要不然容器运行不了

docker save test/nginx:v1.0 -o /root/k8s-nginx.tar
docker save test/tomcat:v1.0 -o /root/k8s-tomcat.tar

scp /root/k8s-nginx.tar [email protected]:/root/
scp /root/k8s-tomcat.tar [email protected]:/root/

node节点导入镜像到docker
注意：因为是用本地镜像，所以有多个node的话，每个node都要手动导入镜像

docker load -i k8s-nginx.tar
docker load -i k8s-tomcat.tar

在k8s-master制作tomcat.yml容器文件

vi tomcat.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: k8s-tomcat-1
  name: k8s-tomcat-1-deployment
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: k8s-tomcat-1
  template:
    metadata:
      labels:
        app: k8s-tomcat-1
    spec:
      containers:
        - name: k8s-tomcat-1-container
          image: test/tomcat:v1.0
          imagePullPolicy: Never
          ports:
          - containerPort: 8080
            protocol: TCP
            name: k8s-tomcat-1
          volumeMounts:
          - mountPath: /usr/local/tomcat/webapps/ROOT
            name: tomcat-nfs
      volumes:
      - name: tomcat-nfs
        nfs:
          server: 192.168.116.130
          path: /data/k8s/tomcat
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: k8s-tomcat-1
  name: k8s-tomcat-1-service
  namespace: default
spec:
  ports:
  - name: k8s-tomcat-1
    port: 8080
  selector:
    app: k8s-tomcat-1

在k8s-master制作nginx.yml容器文件

vi nginx.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx-deployment
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx-container
          image: test/nginx:v1.0
          imagePullPolicy: Never
          ports:
          - containerPort: 80
            protocol: TCP
            name: httpd
          volumeMounts:
          - mountPath: /app/nginx/html
            name: nginx-nfs
      volumes:
      - name: nginx-nfs
        nfs:
          server: 192.168.116.130
          path: /data/k8s/nginx
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: nginx
  name: k8s-nginx-service
  namespace: default
spec:
  type: NodePort
  ports:
    - name: httpd
      port: 80
      protocol: TCP
      targetPort: 80
      nodePort: 30080
  selector:
    app: nginx
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800

3.5 master运行容器文件

master运行配置文件

kubectl apply -f tomcat.yml
kubectl apply -f nginx.yml
kubectl get pod -A -o wide   查看容器状态和node节点IP
kubectl get service -o wide  查看容器映射端口

在node上查看挂载情况，可以看到容器的挂载目录和NFS的共享目录

df -h   查看挂载情况
ss -tnl 查看端口是否打开

3.6 在NFS服务器和k8s容器查看NFS服务共享状态

在NFS服务器上查看目录默认都是空的

ls /data/k8s
ls /data/k8s/nginx/
ls /data/k8s/tomcat/

在master修改tomcat容器内容，测试tomcat容器的目录是否映射到了NFS服务器

kubectl get pod -A
kubectl exec -it k8s-tomcat-1-deployment-78c649d77b-dstrz /bin/bash

跟NFS服务器一样时空目录
ls /usr/local/tomcat/webapps/ROOT/ 
vi /usr/local/tomcat/webapps/ROOT/index.jsp

<%@ page language="java" contentType="text/html; charset=UTF-8"
    pageEncoding="UTF-8"%>
<!DOCTYPE html>
<html>
<head>
    <meta charset="utf-8">
    <title>tomcat-test</title>
</head>
<body>
<h1>tomcat-test</h1>
<%
out.println("test");
%>
<br>
<%=request.getRequestURL()%>
</body>
</html>

cat /usr/local/tomcat/webapps/ROOT/index.jsp

在master修改nginx容器内容，测试nginx容器的目录是否映射到了NFS服务器

kubectl get pod -A
kubectl exec -it nginx-deployment-64d8d89bd4-ggf75 /bin/bash

可以看到nginx容器和NFS服务器共享的目录内容是一致的
ls /app/nginx/html/
echo nginx >/app/nginx/html/index.html
cat /app/nginx/html/index.html 

在NFS服务器确认2个容器生成的文件是否存在

ls /data/k8s/tomcat/
ls /data/k8s/nginx/

curl http://192.168.116.133:30080/index.html
curl http://192.168.116.133:30080/tomcat/index.jsp

3.7 删除nginx容器和tomcat容器，测试当容器崩溃重建数据是否会丢失

删除容器，可以看到容器已经删除
kubectl delete -f tomcat.yml
kubectl delete -f nginx.yml
kubectl get pod -A

在NFS服务器查看数据是否丢失

ls /data/k8s/tomcat/
ls /data/k8s/nginx/

因为nginx和tomcat容器删除了，所以访问不了
curl http://192.168.116.133:30080/index.html
curl http://192.168.116.133:30080/tomcat/index.jsp

在master重新生成容器，测试数据是否丢失

kubectl apply -f tomcat.yml
kubectl apply -f nginx.yml
kubectl get pod -A

在客户端重新访问，可以看到能访问，证明容器重建数据不会丢失

curl http://192.168.116.133:30080/index.html
curl http://192.168.116.133:30080/tomcat/index.jsp