The k8s-web cluster architecture starts from scratch (1)

As an architecture, k8s itself is a very troublesome thing to build an environment. I have tried many methods, hoping to achieve the fastest and most direct construction. First, I will list my cai (keng) process.

k8s construction process

  1. I try to use rancher to build. rancher is an open source full-stack enterprise-level container management platform that provides k8s deployment. All we have to do is add the host and rancher will help us deploy. However, in this dynasty (cough cough), this deployment is indeed a bit stressful, and it is not entirely because pulling mirrors is slow, so people who have the conditions can try it, but I can only try other methods.
  2. The most troublesome way of course is to install it manually. Manual installation can not only help us better understand the k8s process, but also best ensure that every step is error-free. But it's so complicated. Specifically, I have written an md document, and interested students can refer to it.

  3. The third method was found on the Internet. Kind-hearted students have written the manual installation as ansible playbook, so the installation is very convenient. I really want to thank it here. I have indicated the author below, you can go to him to study and research.

k8s-web build

Attached link: docker builds web cluster: https://my.oschina.net/daba0007/blog/1605365

 kubenetes and web cluster: https://my.oschina.net/daba0007/blog/1602425 

The k8s-web cluster architecture starts from scratch (2):  https://my.oschina.net/daba0007/blog/1606600

First look at the picture of the k8s we planned to build last time

This is a diagram indicating the service relationship

This is a diagram indicating the relationship between services, pods and rc

Let's start to introduce the deployment of k8s-web cluster

 

redis service

Having said so much, our k8s architecture is finally built. According to our previous web cluster, we first build a redis master-slave configuration architecture.

principle

For persistent databases (Mysql, Oracle, and SqlServer), they store data on the hard disk of the machine where we deploy the database. When the processing and access volume of the website is very large, the pressure on our database becomes greater. In order to improve the efficiency of the website and reduce the number of reads and writes of the database, we need to introduce caching technology. The cache is the data backup stored in the memory. When the data has not changed in essence, we do not let the data query go to the database for operation, but go to the memory to fetch the data, which greatly reduces the number of reads and writes of the database, and Reading data from memory is faster than querying the database, which improves efficiency at the same time.

Redis as a cache not only supports simple k/v type data, but also provides storage of data structures such as list, set, zset, hash, etc., and supports data persistence, which can keep the data in memory on disk, restart It can be loaded again for use. And memcache, the maximum limit of its single value is 1GB, and memcached can only save 1MB of data.

Therefore, our web cluster uses redis as a cache service.

Master-slave

The first is to build a redis image of our own. There are two types of redis images, one is redis-master and the other is redis-slave. The replication function of redis is to support data synchronization between multiple databases. One is the master database (master) and the other is the slave database (slave). The master database can perform read and write operations, and automatically synchronize data when a write operation occurs. The slave database is generally read-only and receives data synchronized from the master database. A master database can have multiple slave databases, and a slave database can only have one master database.

Through the replication function of redis, the read-write separation of the database can be well realized, and the load capacity of the server can be improved. The master database is mainly for write operations, while the slave database is responsible for read operations.

Process:

  1. When a slave database starts, it sends a sync command to the master database,
  2. After the master database receives the sync command, it will start to save the snapshot in the background (perform the rdb operation), and cache the commands received during the save
  3. When the snapshot is complete, redis will send the snapshot file and all cached commands to the slave database.
  4. Once received from the database, the snapshot file is loaded and the received cached commands are executed.

redis master

We can use the docker pull command to pull a standard redis image (the last time I found that I pulled the image of daocloud.io in the current direction, fried chicken fast)

docker pull daocloud.io/redis

After that I need to write my own redis.conf and put it in the mirror for use (mainly comment bind to enable remote access and use background mode)

My dockerfile is as follows

FROM daocloud.io/redis

MAINTAINER daba0007

ADD redis-master.conf /etc/redis/redis.conf

WORKDIR /etc/redis/

CMD redis-server redis.conf

Construct a dockerfile and push it to my repository

docker build -t daba0007/redis-master  .
docker push daba0007/redis-master

This will have the redis-master image I want

Then start using k8s, first write a redis-master-rc.yaml

apiVersion: v1                              # api版本
kind: ReplicationController                 # 类型是rc
metadata:                                   # 元数据(就是rc的对象)
  name: redis-master
spec:                                       #
  replicas: 1                               # 表示运行1个pod实例,如果多个pod可做LB和HA
  selector:                                 # 是RC的pod选择器,即监视和管理拥有这些标签的pod实例。当运行的pod实例小于replicas时,RC会根据spec下的template段定义的pod模板来生成一个新的pod实例。
    name: redis-master
  template:                                 # 定义的模板,pod实例不足时会运行
    metadata:
      labels:                              # lables属性指定了该Pod的标签,注意:这里的lables必须匹配RC的 spec:selector
        name: redis-master
    spec:
      containers:
      - name: redis-master
        image: daba0007/redis-master
        ports:
        - containerPort: 6379

Execute the command and publish it to the k8s cluster

[root@k8s-master redis-master]# kubectl create -f redis-master-rc.yaml

replicationcontroller "redis-master" created
[root@k8s-master redis-master]# kubectl get pods
NAME                 READY     STATUS    RESTARTS   AGE
redis-master-25bpz   1/1       Running   0          13s

[root@k8s-master redis-master]# kubectl get rc
NAME           DESIRED   CURRENT   READY     AGE
redis-master   1         1         1         30s

Next we will write the service

apiVersion: v1
kind: Service
metadata:
  name:redis-master
  labels:
    name: redis-master
spec:
  ports:
  - port: 6379                              # service暴露在cluster ip上的端口,即虚拟端口
    targetPort: 6379                        # Pod上的端口
  selector:
    name: redis-master                      # 哪些Label的Pod属于此服务

then start the service

[root@k8s-master redis-master]# kubectl create -f redis-master-svc.yaml
service "redis-master" created

[root@k8s-master redis-master]# kubectl get services
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.68.0.1    <none>        443/TCP   3h
redis-master   ClusterIP   10.68.69.173   <none>        6379/TCP   5s

This successfully created the redis master service

redis-slave

After we have the redis master node, we also want to use the slave node to provide read operations to the master node.

Since the IP address is automatically assigned by the Kubernetes system after the service is created, the virtual IP address of a Service cannot be known in advance in other Pods, so a mechanism is needed to find the service. Kubernetes cleverly uses Linux environment variables to add a set of Service-related environment variables to each Pod container to record the mapping relationship from service names to virtual IP addresses. Taking the redis-master service as an example, the following two records will be added to the environment variables of the container

REDIS_MASTER_SERVICE_HOST=10.68.69.173
REDIS_MASTER_SERVICE_PORT=6379

Then write a dockerfile, the slave node of redis needs to declare the master node

FROM daocloud.io/redis

MAINTAINER daba0007

ADD redis-slave.conf /etc/redis/redis.conf

WORKDIR /etc/redis/

CMD redis-server redis.conf --slaveof ${REDIS_MASTER_SERVICE_HOST} ${REDIS_MASTER_SERVICE_PORT}

Then create our slave dockerfile and push it to my repository

docker build -t daba0007/redis-slave  .
docker push daba0007/redis-slave

Then write slave's redis-slave-rc.yaml

apiVersion: v1
kind: ReplicationController
metadata:
  name: redis-slave
  labels:
    name: redis-slave
spec:
  replicas: 2                                   # 启动两个副本来做从服务
  selector:
    name: redis-slave
  template:
    metadata:
      labels:
        name:redis-slave
    spec:
      containers:
      - name: redis-slave
        image: daba0007/redis-slave
        env:                                    # 获取k8s环境的变量
        - name: GET_HOSTS_FROM
          value: env
        ports:
        - containerPort: 6379

then execute

[root@k8s-master redis-slave]# kubectl create -f redis-slave-rc.yaml
replicationcontroller "redis-slave" created

[root@k8s-master redis-slave]# kubectl get rc
NAME           DESIRED   CURRENT   READY     AGE
redis-master   1         1         1         10h
redis-slave    2         2         1         5s

[root@k8s-master redis-slave]# kubectl get pods
NAME                 READY     STATUS    RESTARTS   AGE
redis-master-25bpz   1/1       Running   1          10h
redis-slave-plrxq    1/1       Running   0          12s
redis-slave-thb9r    1/1       Running   0          12s

Then write redis-slave-svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: redis-slave
  labels:
    name: redis-slave
spec:
  ports:
  - port: 6379
  selector:
    name: redis-slave

then execute

[root@k8s-master redis-slave]# kubectl create -f redis-slave-svc.yaml
service "redis-slave" created

[root@k8s-master redis-slave]# kubectl get services
NAME           TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
kubernetes     ClusterIP   10.68.0.1      <none>        443/TCP    14h
redis-master   ClusterIP   10.68.69.173   <none>        6379/TCP   10h
redis-slave    ClusterIP   10.68.174.55   <none>        6379/TCP   5s

Seeing that it is all running, it means that the startup is successful and the operation is smooth.

test

Enter the master node, execute the write value a=1 on the master node, and find that the value of a is equal to 1 after reading from the database, indicating that the synchronization has been successful, and the redis master-slave cluster configuration is successful.

[root@k8s-master redis]# kubectl exec -ti redis-master-25bpz /bin/bash
root@redis-master-25bpz:/etc/redis# redis-cli
127.0.0.1:6379> set a 1
OK
127.0.0.1:6379> exit
root@redis-master-25bpz:/etc/redis# exit
exit
[root@k8s-master redis]# kubectl exec -ti redis-slave-plrxq /bin/bash
root@redis-slave-plrxq:/etc/redis# redis-cli
127.0.0.1:6379> get a
"1"
127.0.0.1:6379> exit

mysql service

The next step is to deploy the k8s database mysql.

principle

MySQL is a relational database management system that is open source and supports large databases and multiple languages. In our web cluster, it mainly plays the role of data storage. It will store the service's data into its database. When our service calls data for the first time, it will go to mysql to find data and write to redis at the same time. After that, if the data cannot be found on redis, it will go to mysql to find it.

Master-slave

The master-slave replication of mysql refers to copying the data of a certain host of Mysql to other hosts (slaves) and re-executing it. During replication, one server acts as the master, and one or more other servers act as slaves. When a slave connects to the master, it informs the master of the location of the last successful update the slave read in the log. The slave receives any updates that have occurred since then, then blocks and waits for the master to notify of new updates.

MySQL master-slave replication can be implemented in the following ways.

  1. master

Modify the configuration file on the master host, such as usually modifying my.cnf.

[mysqld] 
server-id=1 
log-bin

Create a synchronization account on mysql and authorize it.

As follows, create a user named test and a password of 123456:

create user 'test'@'%' identified by '123456';

As follows, authorize the repl user to allow synchronization:

grant replication slave on *.* to 'test'@'%' identified by '123456';

2.  slave

Similarly, modify the configuration file on the slave host.

[mysqld]
server-id=2 
log-bin

Then configure as follows, where xxxx is the ip address of the master host.

change master to master_host='x.x.x.x',master_user='test',master_password='123456';

mysql-master

There are two files Dockerfile,  docker-entrypoint.sh in the docker official website image file https://hub.docker.com/_/mysql/, the version 5.7 is used here. Download it and add the following to the dockerfile (modify the above sentence at the same time):

RUN sed -i '/\[mysqld\]/a server-id=1\nlog-bin' /etc/mysql/mysql.conf.d/mysqld.cnf

and add in docker-entrypoint.sh

echo "CREATE USER '$MYSQL_REPLICATION_USER'@'%' IDENTIFIED BY '$MYSQL_REPLICATION_PASSWORD' ;" | "${mysql[@]}"
echo "GRANT REPLICATION SLAVE ON *.* TO '$MYSQL_REPLICATION_USER'@'%' IDENTIFIED BY '$MYSQL_REPLICATION_PASSWORD' ;" | "${mysql[@]}"
echo 'FLUSH PRIVILEGES ;' | "${mysql[@]}"
echo

Two environment variables MYSQL_REPLICATION_USER and MYSQL_REPLICATION_PASSWORD are added above, which are used as the account and password for master-slave replication.

then create the image and upload it

docker build -t daba0007/mysql-master .
docker push daba0007/mysql-master

This will generate a new mysql-master image to serve

Then create mysql-master-rc.yaml, here we have to mount a database folder to facilitate data migration in the future

apiVersion: v1
kind: ReplicationController
metadata:
  name: mysql-master
  labels:
    name: mysql-master
spec:
  replicas: 1
  selector:
    name: mysql-master
  template:
    metadata:
      labels:
        name: mysql-master
    spec:
      containers:
      - name: mysql-master
        image: daba0007/mysql-master
        volumeMounts:                                   # 挂载在容器上的数据卷
        - name: mysql-database                          # 标签
          mountPath: /etc/mysql/database                # 容器上数据卷的路径
          readOnly: false                               # 可读写
        env:
        - name: MYSQL_ROOT_PASSWORD                     # 数据库root密码
          value: "123456"
        - name: MYSQL_REPLICATION_USER                  # 提供给从数据库同步的账号
          value: "test"
        - name: MYSQL_REPLICATION_PASSWORD              # 提供给从数据库同步的账号的密码
          value: "123456"
        ports:
        - containerPort: 3306
      volumes:                                           # 挂载在主机上的数据卷
      - name: mysql-database                             # 对应标签
        hostPath:
          path: /root/k8s/mysql/mysql-master/database    # 主机上数据卷的路径

then execute

[root@k8s-master mysql-master]# kubectl create -f mysql-master-rc.yaml
replicationcontroller "mysql-master" created

[root@k8s-master mysql-master]# kubectl get rc
NAME           DESIRED   CURRENT   READY     AGE
mysql-master   1         1         1         12s
redis-master   1         1         1         1d
redis-slave    2         2         2         1d

[root@k8s-master mysql-master]# kubectl get pods
NAME                 READY     STATUS    RESTARTS   AGE
mysql-master-whtwd   1/1       Running   0          8s
redis-master-25bpz   1/1       Running   3          2d
redis-slave-plrxq    1/1       Running   2          1d
redis-slave-thb9r    1/1       Running   2          1d

Then write the mysql service mysql-master-svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: mysql-master
  labels:
    name: mysql-master
spec:
  ports:
    - port: 3306      
      targetPort: 3306 
  selector:
    name: mysql-master

then create

[root@k8s-master mysql-master]# kubectl create -f mysql-master-svc.yaml
service "mysql-master" created

NAME           TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
kubernetes     ClusterIP   10.68.0.1      <none>        443/TCP    2d
mysql-master   ClusterIP   10.68.83.79    <none>        3306/TCP   5s
redis-master   ClusterIP   10.68.69.173   <none>        6379/TCP   2d
redis-slave    ClusterIP   10.68.174.55   <none>        6379/TCP   1d

At this point, we have realized the creation of the mysql main service

mysql-slave

mysql-slave is slightly different from the creation of master

Add the following to the dockerfile (and modify the above sentence):

RUN RAND="$(date +%s | rev | cut -c 1-2)$(echo ${RANDOM})" && sed -i '/\[mysqld\]/a server-id='$RAND'\nlog-bin' /etc/mysql/mysql.conf.d/mysqld.cnf

Here the server-id is a random number.

and add in docker-entrypoint.sh

echo "STOP SLAVE;" | "${mysql[@]}"
echo "CHANGE MASTER TO master_host='$MYSQL_MASTER_SERVICE_HOST', master_user='$MYSQL_REPLICATION_USER', master_password='$MYSQL_REPLICATION_PASSWORD' ;" | "${mysql[@]}"
echo "START SLAVE;" | "${mysql[@]}"

In the slave configuration above, the master_host item uses $MYSQL_MASTER_SERVICE_HOST, and this environment variable (enviromnent variable) is generated by k8s.

After the k8s service is created, it will automatically assign a cluster ip. This cluster ip is dynamic, and we cannot use it directly or hardcode it. In order to make the service visible to the container, k8s generates a set of environment variables, which are used to record The mapping relationship between service name and cluster ip address, so that these variables can be used in the container to use the service. (Similarly, links are provided in Docker.)

Example: If the name of service is foo, the generated environment variables are as follows:

FOO_SERVICE_HOST
FOO_SERVICE_PORT

For more information, please refer to the official k8s information: http://kubernetes.io/docs/user-guide/container-environment/

Then create an image and upload it

docker build -t daba0007/mysql-slave .
docker push daba0007/mysql-slave

Then write mysql-slave-rc.yaml

apiVersion: v1
kind: ReplicationController
metadata:
  name: mysql-slave
  labels:
    name: mysql-slave
spec:
  replicas: 2
  selector:
    name: mysql-slave
  template:
    metadata:
      labels:
        name: mysql-slave
    spec:
      containers:
      - name: mysql-slave
        image: daba0007/mysql-slave
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: "123456"
        - name: MYSQL_REPLICATION_USER
          value: "test"
        - name: MYSQL_REPLICATION_PASSWORD
          value: "123456"
        ports:
        - containerPort: 3306

then create

[root@k8s-master mysql-slave]# kubectl create -f mysql-slave-rc.yaml
replicationcontroller "mysql-slave" created

[root@k8s-master mysql-slave]# kubectl get rc
NAME           DESIRED   CURRENT   READY     AGE
mysql-master   1         1         1         22m
mysql-slave    2         2         1         7s
redis-master   1         1         1         1d
redis-slave    2         2         2         1d

[root@k8s-master mysql-slave]# kubectl get pods
NAME                 READY     STATUS    RESTARTS   AGE
mysql-master-whtwd   1/1       Running   0          1m
mysql-slave-6x8bx    1/1       Running   0          11s
mysql-slave-n58vk    1/1       Running   0          11s
redis-master-25bpz   1/1       Running   3          2d
redis-slave-plrxq    1/1       Running   2          1d
redis-slave-thb9r    1/1       Running   2          1d

After that write mysql-slave-svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: mysql-slave
  labels:
    name: mysql-slave
spec:
  ports:
    - port: 3306      
      targetPort: 3306 
  selector:
    name: mysql-slave

Then create the mysql-slave service

[root@k8s-master mysql-slave]# kubectl create -f mysql-slave-svc.yaml
service "mysql-slave" created

[root@k8s-master mysql-slave]# kubectl get svc
NAME           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
kubernetes     ClusterIP   10.68.0.1       <none>        443/TCP    2d
mysql-master   ClusterIP   10.68.83.79     <none>        3306/TCP   1m
mysql-slave    ClusterIP   10.68.208.186   <none>        3306/TCP   9s
redis-master   ClusterIP   10.68.69.173    <none>        6379/TCP   2d
redis-slave    ClusterIP   10.68.174.55    <none>        6379/TCP   1d

In this way, the mysql-slave service is successfully created to test whether the following data is synchronized to the main database

test

How to prove that we have successfully set up a mysql master-slave connection?

First log in to our main database, import our database, and view the database, you will find that the data has been imported

[root@k8s-master mysql-slave]# kubectl exec -ti mysql-master-whtwd  /bin/bash
root@mysql-master-whtwd:/# cd /etc/mysql/database/
root@mysql-master-whtwd:/etc/mysql/database#  mysql -uroot -p -e "create database form;"      # 创建form数据库
Enter password:
root@mysql-master-whtwd:/etc/mysql/database# mysql -uroot -p form < form.sql                  # 导入我们之前的数据库
Enter password:
root@mysql-master-whtwd:/etc/mysql/database# mysql -uroot -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 12
Server version: 5.7.20-log MySQL Community Server (GPL)

Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show master status;
+-------------------------------+----------+--------------+------------------+-------------------+
| File                          | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+-------------------------------+----------+--------------+------------------+-------------------+
| mysql-master-whtwd-bin.000003 |    18218 |              |                  |                   |
+-------------------------------+----------+--------------+------------------+-------------------+
1 row in set (0.01 sec)

mysql> use form;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables;
+----------------------------+
| Tables_in_form             |
+----------------------------+
| auth_group                 |
| auth_group_permissions     |
| auth_permission            |
| auth_user                  |
| auth_user_groups           |
| auth_user_user_permissions |
| django_admin_log           |
| django_content_type        |
| django_migrations          |
| django_session             |
| form_code_data             |
| form_user                  |
| form_user_data             |
+----------------------------+
13 rows in set (0.00 sec)

Then check our slave database and find that the slave library has been connected to the master library and written to the form database, indicating that our mysql master-slave service cluster has been successfully built.

[root@k8s-master mysql-slave]# kubectl exec -ti mysql-slave-6x8bx -- mysql -uroot -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 5
Server version: 5.7.20-log MySQL Community Server (GPL)

Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show slave status\G;
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 10.68.83.79
                  Master_User: test
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-master-whtwd-bin.000003
          Read_Master_Log_Pos: 18218
               Relay_Log_File: mysql-slave-6x8bx-relay-bin.000005
                Relay_Log_Pos: 18457
        Relay_Master_Log_File: mysql-master-whtwd-bin.000003
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 18218
              Relay_Log_Space: 3015257
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 1
                  Master_UUID: 379cac8b-f626-11e7-8b34-c2a2572e0349
             Master_Info_File: /var/lib/mysql/master.info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
           Master_Retry_Count: 86400
                  Master_Bind:
      Last_IO_Error_Timestamp:
     Last_SQL_Error_Timestamp:
               Master_SSL_Crl:
           Master_SSL_Crlpath:
           Retrieved_Gtid_Set:
            Executed_Gtid_Set:
                Auto_Position: 0
         Replicate_Rewrite_DB:
                 Channel_Name:
           Master_TLS_Version:
1 row in set (0.00 sec)

ERROR:
No query specified

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| form               |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
5 rows in set (0.01 sec)

 

In the next issue we will continue to study web services and nginx services

 

Some common error checking commands

get log

kubectl logs -f [pods]                          # [pods]写入你pods的名字

delete

kubectl delete svc redis-master
kubectl delete rc redis-master

Thanks to the following bloggers for their ideas

  1. https://www.kubernetes.org.cn/3265.html
  2. http://blog.csdn.net/test103/article/details/55663562
  3. https://my.oschina.net/FrankXin/blog/875414
  4. https://www.jianshu.com/p/509b65e9a4f5

 

 

All my codes are on https://github.com/daba0007/k8s-web, you are welcome to download and study. Please indicate the author when reprinting 

Guess you like

Origin http://43.154.161.224:23101/article/api/json?id=324429933&siteId=291194637