恢复之前的环境：
[root@foundation38 compose]# docker network ls 查看网络状态
NETWORK ID NAME DRIVER SCOPE
818f7445019d bridge bridge local
f4ff71176a19 host host local
b12b1dd0542e none null local
[root@foundation38 compose]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1827c740a059 haproxy "/docker-entrypoin..." 30 minutes ago Up 21 minutes 0.0.0.0:8080->80/tcp compose_haproxy_1
dd054ad399f4 rhel7:v1 "/usr/sbin/httpd -..." 35 minutes ago Up 15 minutes 80/tcp compose_apache_1
e21cf912a341 nginx "nginx -g 'daemon ..." 56 minutes ago Up 21 minutes 80/tcp compose_nginx_1
[root@foundation38 compose]# pwd
/tmp/docker/compose
[root@foundation38 compose]# docker-compose stop 停止compose
Stopping compose_haproxy_1 ... done
Stopping compose_apache_1 ... done
Stopping compose_nginx_1 ... done
[root@foundation38 compose]# docker container prune 删除停止的容器
WARNING! This will remove all stopped containers.
Are you sure you want to continue? [y/N] y
Deleted Containers:
1827c740a059e85ac4b47b832f7f4f3eaa5dc28fab2def309efca4b06b5fcf65
dd054ad399f4f4627cc11b8c4cd7710bd56206d1b467364cbfb89709cefe3f5b
e21cf912a341413409fd9d8ff5b52c12deb9a9c82920d62ef56f292e266dff37
Total reclaimed space: 5.013 kB
[root@foundation38 compose]# systemctl restart docker 重启docker服务
在三台红帽7.3虚拟机操作进行swarm集群：
[root@test1 ~]# ls
docker-engine-17.03.1.ce-1.el7.centos.x86_64.rpm
docker-engine-selinux-17.03.1.ce-1.el7.centos.noarch.rpm
[root@test1 ~]# yum install * -y 安装docker服务
[root@test1 ~]# systemctl start docker 开启服务
[root@test1 ~]# yum install -y bash-* 安装一些docker相关工具在test1主节点:
[root@localhost ~]# docker swarm init 初始化
Swarm initialized: current node (xrtfwmppzsb653287axdbrvrr) is now a manager.
To add a worker to this swarm, run the following command:
docker swarm join \
--token SWMTKN-1-1hw1jlhwzr6fiojlxuoovje9bxiuln1mci7b5gx1kuqujvnguj-2euasgizomoyk4ta77hamnijo \
172.25.38.11:2377
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
在test2和test3从节点加入主节点：
在test1查看节点状态：
[root@localhost ~]# docker swarm node ls
Usage: docker swarm COMMAND
Manage Swarm
Options:
--help Print usage
Commands:
init Initialize a swarm
join Join a swarm as a node and/or manager
join-token Manage join tokens
leave Leave the swarm
unlock Unlock swarm
unlock-key Manage the unlock key
update Update the swarm
Run 'docker swarm COMMAND --help' for more information on a command.
[root@test1 ~]# docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
ea79ei9dlqliz06cvt73ebxfm test2 Ready Active
lrv5sm0wnb9y5087mle7ghn52 * test1 Ready Active Leader
yh8mwhfppccxs1w9jk9qn9hnd test3 Ready Active 
在test1和test2和test3均做好解析保证可以ping通域名：
[root@test1 ~]# vim /etc/hosts
[root@test1 ~]# cat /etc/hosts | tail -n 1
172.25.38.250 westos.org
[root@test1 ~]# ping westos.org
PING westos.org (172.25.38.250) 56(84) bytes of data.
64 bytes from westos.org (172.25.38.250): icmp_seq=1 ttl=64 time=0.362 ms
^C
--- westos.org ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.362/0.362/0.362/0.000 ms
在真机将证书传递到三台虚拟机：
[root@foundation38 certs.d]# cd ..
[root@foundation38 docker]# pwd
/etc/docker
[root@foundation38 docker]# ls
certs.d daemon.json key.json
[root@foundation38 docker]# scp -r certs.d/ [email protected]:/etc/docker/
root@172.25.38.11's password:
ca.crt 100% 2098 2.1KB/s 00:00
[root@foundation38 docker]# scp -r certs.d/ [email protected]:/etc/docker/
[email protected]'s password:
ca.crt 100% 2098 2.1KB/s 00:00
[root@foundation38 docker]# scp -r certs.d/ [email protected]:/etc/docker/
root@172.25.38.13's password:
ca.crt 100% 2098 2.1KB/s 00:00
[root@foundation38 docker]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[root@foundation38 docker]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
在真机重新生成证书：
[root@foundation38 docker]# pwd
/etc/docker
[root@foundation38 docker]# docker run -d \
> --restart=always \
> --name registry \
> -v `pwd`/certs:/certs \
> -e REGISTRY_HTTP_ADDR=0.0.0.0:443 \
> -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
> -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
> -p 443:443 \
> registry:2
dff160d34d7a0c78e732e758cb03d852c1abc0ed3c3c4e073ecb253a26d87853
[root@foundation38 kiosk]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4f4e43ad5ab1 registry:2 "/entrypoint.sh /e..." 29 minutes ago Up 2 minutes 0.0.0.0:443->443/tcp, 5000/tcp registry
[root@foundation38 docker]# docker load -i nginx.tar
d8bd0657b25f: Loading layer 130.9 MB/130.9 MB
a582cd499e0f: Loading layer 1.024 kB/1.024 kB
f5a1ef78853b: Loading layer 1.024 kB/1.024 kB
605083c53cc5: Loading layer 1.024 kB/1.024 kB
c95d1af3960d: Loading layer 67.97 MB/67.97 MB
0a410ba58143: Loading layer 3.584 kB/3.584 kB
fb6fc13654c9: Loading layer 1.024 kB/1.024 kB
36b0adefb2f3: Loading layer 1.024 kB/1.024 kB
[root@foundation38 docker]# docker tag nginx westos.org/nginx
[root@foundation38 docker]# docker push westos.org/nginx
The push refers to a repository [westos.org/nginx]
5f70bf18a086: Pushed
3f3324023e75: Pushed
f0d7d68f89e5: Pushed
917c0fc99b35: Pushed
latest: digest: sha256:e59bbe13ca0c309644d9bca972c1b660c170fbdee7ff81eda9ff966b86f14e63 size: 1978
[root@foundation38 docker]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
rhel7 v1 1796ce7f20c8 24 hours ago 169 MB
rhel7 v4 5644778574b6 29 hours ago 140 MB
rhel7 v3 7065648c9208 2 days ago 178 MB
rhel7 v2 0f3f4a1b0b2e 3 days ago 154 MB
ubuntu latest 735f80812f90 3 weeks ago 83.5 MB
hello-world latest 2cb0d9787c4d 6 weeks ago 1.85 kB
registry 2 b2b03e9146e1 6 weeks ago 33.3 MB
game2048 latest 19299002fdbe 20 months ago 55.5 MB
haproxy latest fbd1f55f79b3 2 years ago 139 MB
nginx latest af4b3d7d5401 2 years ago 190 MB
westos.org/nginx latest af4b3d7d5401 2 years ago 190 MB
registry latest bca04f698ba8 2 years ago 423 MB
stress latest db646a8f4087 4 years ago 282 MB
westos.org/rhel7 latest 0a3eb3fde7fd 4 years ago 140 MB
rhel7 latest 0a3eb3fde7fd 4 years ago 140 MB
[root@test1 ~]# docker service create --name nginx --publish 80:80 --replicas 3 westos.org/nginx
[root@test1 ~]# docker service ls 查看看到3/3就可以代表节点部署成功
ID NAME MODE REPLICAS IMAGE
9bzx7d8r7ok9 nginx replicated 3/3 westos.org/nginx
[root@test1 ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9cbac3f88b65 westos.org/nginx:latest "nginx -g 'daemon ..." 3 minutes ago Up 3 minutes 80/tcp, 443/tcp nginx.2.r4etdxuvlfn4q6oag13xd6qfv
[root@test1 ~]# docker service scale nginx=6 添加6个nginx服务
nginx scaled to 6
[root@test1 ~]# docker service ps nginx 由于重复了进行了多次，但是可以看到test1,test2,和test3分别出来了两次，实现了负载均衡
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
k2wn13zjycq3 nginx.1 westos.org/nginx:latest test2 Running Running 6 seconds ago
dv70fh9tfmfi \_ nginx.1 westos.org/nginx test2 Shutdown Shutdown 7 seconds ago
qenmgc0j1cx0 \_ nginx.1 westos.org/nginx test3 Shutdown Failed 4 minutes ago "task: non-zero exit (137)"
wom3e64dull5 \_ nginx.1 westos.org/nginx test2 Shutdown Rejected 9 minutes ago "No such image: westos.org/ngi…"
o0v8ufq3w5hy \_ nginx.1 westos.org/nginx test1 Shutdown Rejected 9 minutes ago "No such image: westos.org/ngi…"
ix9fmookzzc7 nginx.2 westos.org/nginx:latest test2 Running Running 9 seconds ago
r4etdxuvlfn4 \_ nginx.2 westos.org/nginx test1 Shutdown Shutdown 11 seconds ago
amhphgcj17eg \_ nginx.2 westos.org/nginx test1 Shutdown Failed 4 minutes ago "task: non-zero exit (137)"
imn4k8nrtalk \_ nginx.2 westos.org/nginx test2 Shutdown Rejected 9 minutes ago "No such image: westos.org/ngi…"
s2xnc3rx51qb \_ nginx.2 westos.org/nginx test1 Shutdown Rejected 9 minutes ago "No such image: westos.org/ngi…"
yju0kp3e9k77 nginx.3 westos.org/nginx:latest test3 Running Running 4 seconds ago
k2ndl012dgum \_ nginx.3 westos.org/nginx test3 Shutdown Shutdown 5 seconds ago
uz9alk3zfy0i \_ nginx.3 westos.org/nginx test3 Shutdown Failed 4 minutes ago "task: non-zero exit (137)"
9ai0s12k6hml \_ nginx.3 westos.org/nginx test2 Shutdown Rejected 9 minutes ago "No such image: westos.org/ngi…"
omgm1pomv9lk \_ nginx.3 westos.org/nginx test2 Shutdown Rejected 9 minutes ago "No such image: westos.org/ngi…"
i0l2nybq46vo nginx.4 westos.org/nginx:latest test3 Running Running 12 seconds ago
q09m0ddd5i67 nginx.5 westos.org/nginx:latest test1 Running Running 11 seconds ago
ldx0pw5f3az5 nginx.6 westos.org/nginx:latest test1 Running Running 11 seconds ago 
在三个节点分别查看docker进程来验证负载均衡效果：
在网页调用三个节点的IP可以分别看到nginx的测试页：
进行配置负载均衡：
[root@foundation38 docker]# docker load -i visualizer.tar
[root@foundation38 docker]# docker images
dockersamples/visualizer latest 17e55a9b2354 11 months ago 148 MB
[root@foundation38 docker]# docker tag dockersamples/visualizer westos.org/visualizer
[root@foundation38 docker]# docker push westos.org/visualizer
在server1主节点：
[root@test1 ~]# docker service create \
> --name=viz \
> --publish=8080:8080/tcp \
> --constraint=node.role==manager \
> --mount=type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock \
> westos.org/visualizer
wel4zkrwri5fw04wbvg413lns
[root@test1 ~]# docker service ls 需要等待加载比较慢，直到出现1/1即可
ID NAME MODE REPLICAS IMAGE
9bzx7d8r7ok9 nginx replicated 6/6 westos.org/nginx:latest
wel4zkrwri5f viz replicated 0/1 westos.org/visualizer:latest
[root@test1 ~]# docker service ls
ID NAME MODE REPLICAS IMAGE
9bzx7d8r7ok9 nginx replicated 6/6 westos.org/nginx:latest
wel4zkrwri5f viz replicated 0/1 westos.org/visualizer:latest
[root@test1 ~]# docker service ls
ID NAME MODE REPLICAS IMAGE
9bzx7d8r7ok9 nginx replicated 6/6 westos.org/nginx:latest
wel4zkrwri5f viz replicated 1/1 westos.org/visualizer:latest
在三个节点分别部署测试页：
[root@test1 ~]# docker service scale nginx=3
nginx scaled to 3
[root@test1 ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
333b6417451c westos.org/visualizer@sha256:ea603c00f25b18c9a3aa49703c47bd5ca6c105637fe6f4887b606043c66ab528 "npm start" 2 minutes ago Up 2 minutes (healthy) 8080/tcp viz.1.lajtc604ju2nxhpd8z5s5ulbf
ef3505414e00 westos.org/nginx@sha256:e59bbe13ca0c309644d9bca972c1b660c170fbdee7ff81eda9ff966b86f14e63 "nginx -g 'daemon ..." 18 minutes ago Up 18 minutes 80/tcp, 443/tcp nginx.6.ldx0pw5f3az5ff3wijfu8vzxo
[root@test1 ~]# echo server1 > index.html
[root@test1 ~]# docker container cp index.html nginx.6.ldx0pw5f3az5ff3wijfu8vzxo:/usr/share/nginx/html
[root@test2 ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
91f92c29116d westos.org/nginx@sha256:e59bbe13ca0c309644d9bca972c1b660c170fbdee7ff81eda9ff966b86f14e63 "nginx -g 'daemon ..." 19 minutes ago Up 19 minutes 80/tcp, 443/tcp nginx.2.ix9fmookzzc7d0xf6md745gu2
[root@test2 ~]# docker container cp index.html nginx.2.ix9fmookzzc7d0xf6md745gu2:/usr/share/nginx/html
lstat /root/index.html: no such file or directory
[root@test2 ~]# echo server2 > index.html
[root@test2 ~]# docker container cp index.html nginx.2.ix9fmookzzc7d0xf6md745gu2:/usr/share/nginx/html
[root@test3 ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
852aa8d24e17 westos.org/nginx@sha256:e59bbe13ca0c309644d9bca972c1b660c170fbdee7ff81eda9ff966b86f14e63 "nginx -g 'daemon ..." 20 minutes ago Up 20 minutes 80/tcp, 443/tcp nginx.3.yju0kp3e9k77v5xap249xhiql
[root@test3 ~]# echo server3 > index.html
[root@test3 ~]# docker container cp index.html nginx.3.yju0kp3e9k77v5xap249xhiql:/usr/share/nginx/html
在真机测试可以负载均衡：
[root@foundation38 docker]# for i in {1..10}; do curl 172.25.38.11;done
server2
server1
server3
server2
server1
server3
server2
server1
server3
server2
网页访问可以看到nginx的负载均衡：
将test3的docker关闭：
[root@test3 ~]# systemctl stop docker
[root@test3 ~]# 在真机测试可以看到nginx的测试页：
[root@foundation38 docker]# for i in {1..10}; do curl 172.25.38.11;done
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
在网页可以看到其他节点在进行负载均衡：
部署滚动更新：
在真机push镜像：
[root@foundation38 docker]# docker tag rhel7:v1 westos.org/rhel7:v1
[root@foundation38 docker]# docker push westos.org/rhel7:v1
The push refers to a repository [westos.org/rhel7]
b3fdd1378427: Pushed
10050c3b729d: Pushed
18af9eb19b5f: Pushed
v1: digest: sha256:1808ce0610070cd4d9becb1696be942c700388a6db03e0cc57b3d9728cab2584 size: 948
在server1进行滚动更新配置：
[root@test1 ~]# docker service scale nginx=30
nginx scaled to 30
[root@test1 ~]# docker service update --image westos.org/rhel7:v1 --update-parallelism 3 --update-delay 10s nginx
nginx 每隔10s三个节点会进行滚动更新
在网页测试三个节点每隔10s会进行滚动更新：
在真机使用docker ps也可以进行查看只是数据太多效果不直观！！
在真机推一个2048小游戏到仓库：
[root@foundation38 docker]# docker tag game2048 westos.org/game2048
[root@foundation38 docker]# docker push westos.org/game2048
The push refers to a repository [westos.org/game2048]
88fca8ae768a: Pushed
6d7504772167: Pushed
192e9fad2abc: Pushed
36e9226e74f8: Pushed
011b303988d2: Pushed
latest: digest: sha256:31a46998f55ed03da6d62197f3a91b0a100c0abdd7380d88142ea44bce325001 size: 1364
在server1进行滚动更新配置：
[root@test1 ~]# docker service update --image westos.org/game2048 --update-parallelism 3 --update-delay 10s nginx
nginx 每隔10s三个节点会进行滚动更新
在网页测试输入真机IP可以看到2048小游戏界面：
在网页同样可以看到2048小游戏的滚动更新：
