Docker build clusters using Flink

---

Foreword

Recently contacted about the docker, want to take a cluster Flink and play, online search, I found a lot are from Dockerfile start building, and then I looked at the official website, found that the use of Docker be built tutorial. I refer to the official website of the way, Linux环境to build, the way my record stepped pit.
There are two main ways to build, respectively, 使用docker命令进行构建and 使用docker-compose进行构建, prior to the operation, I have installed a docker, 安装docker并且给docker换国内源tutorials here will not repeat, you can search online

---

Method 1: Use the command to build docker

• Create Network

docker network create app-tier --bridge

• Creating jobmanager container

docker run -t -d --name jmr \
--network app-tier \
-e JOB_MANAGER_RPC_ADDRESS=jmr \
-p 8081:8081  \
flink:1.9.2-scala_2.12 jobmanager

• Creating taskmanager container

docker run -t -d --name tmr \
--network app-tier \
-e JOB_MANAGER_RPC_ADDRESS=jmr   \
flink:1.9.2-scala_2.12 taskmanager

Second way: docker-compose constructed using

Using this mode you must first install docker-compose
I used the following command to install, reference links

su root
curl -L https://get.daocloud.io/docker/compose/releases/download/1.25.4/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

After installation, we can look at the version docker to verify whether the installation was successful

docker-compose --version

Then we can create a folder named in a docker-compose.ymldocument, for convenience, I would like to create in the current folder

touch docker-compose.yml

Then edit this file, writes the following exit and save! ! ! Note that the format can not be wrong

version: "2.1"
services:
  jobmanager:
    image: flink:1.9.2-scala_2.12
    expose:
      - "6123"
    ports:
      - "8081:8081"
    command: jobmanager
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager

  taskmanager:
    image: flink:1.9.2-scala_2.12
    expose:
      - "6121"
      - "6122"
    depends_on:
      - jobmanager
    command: taskmanager
    links:
      - "jobmanager:jobmanager"
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager

Finally, the following command will be able to build up

docker-compose up -d

---

How to view the cluster and log Flink

View cluster via the web

Official mirror exposed the 8081 port, and we are in the mode I and mode B. We will map a machine local port 8081 to the 8081 port container, so we can see the cluster via a browser, because I was on a virtual machine In progress. So the browser by 虚拟机IP:端口accessing the way, if docker is mounted directly on the machine, not installed on the virtual machine, you can directly localhost:8081access

View Log command

#tmr 是镜像命令，可以替换成其他的镜像名
docker logs tmr

---

FAQ

In a way the official website is written so

Running a JobManager or a TaskManager
You can run a JobManager (master).

$ docker run --name flink_jobmanager -d -t flink jobmanager

You can also run a TaskManager (worker). Notice that workers need to register with the JobManager directly or via ZooKeeper so the master starts to send them tasks to execute.

$ docker run --name flink_taskmanager -d -t flink taskmanager

And there are pits, a single container may be constructed, the plurality of constructed on the error
mode II docker-compose.ymlis copied from the official website over

1. 方式一中为什么要创建网络
   答：创建网络是为了让容器能过通过容器名或者容器ID发现对方，与对方进行通信。不指定创建的网络的话，taskmanager可能会出现以下的错误

ERROR org.apache.flink.runtime.taskexecutor.TaskManagerRunner       - TaskManager initialization failed.
java.net.UnknownHostException: jmr: Name or service not known
        at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)

2. 方式一中为什么要指定环境变量JOB_MANAGER_RPC_ADDRESS
   答：不指定环境变量的话，jobmanager（即jmr）不会报错，但是taskmanager（即tmr）会报以下错误

2020-02-27 03:35:12,029 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Could not resolve ResourceManager address akka.tcp://flink@f59c179f3470:6123/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@f59c179f3470:6123/user/resourcemanager..
2020-02-27 03:35:20,782 ERROR org.apache.flink.runtime.taskexecutor.TaskExecutor            - Fatal error occurred in TaskExecutor akka.tcp://[email protected]:38671/user/taskmanager_0.
org.apache.flink.runtime.taskexecutor.exceptions.RegistrationTimeoutException: Could not register at the ResourceManager within the specified maximum registration duration 300000 ms. This indicates a problem with this instance. Terminating now.
        at org.apache.flink.runtime.taskexecutor.TaskExecutor.registrationTimeout(TaskExecutor.java:1111)
        at org.apache.flink.runtime.taskexecutor.TaskExecutor.lambda$startRegistrationTimeout$8(TaskExecutor.java:1097)
        at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:397)

taskmanager在30000ms内无法在jobmanager处注册，抛出异常
出现这个错误的原因是docker启动flink容器的时候，会执行镜像中官方定制的docker-entrypoint.sh脚本文件。
在这个脚本文件中指定了，如果启动命令中没有指定环境变量JOB_MANAGER_RPC_ADDRESS，那么就会使用本机的hostname作为这个环境变量的值，并将其写入到配置文件中。在docker中这个值就是Container ID，所以在日志中可以看到类似的信息

INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.address, f59c179f3470

所以，如果我们没有指定这个环境变量的值，那么taskmanager就会向自身的6123端口发起注册，这是错误的，因为taskmanger没有接受注册的能力，并且taskmanger应该是向jobmanger注册才对。因此我们需要指定JOB_MANAGER_RPC_ADDRESS的值为jobmanger的容器名或者容器ID或者容器的IP

3. 如何往这个集群中增加或者删除容器
   答：删除容器，我们可以简单地通过rm命令来实现

docker rm -f tmr

可以使用docker命令增加容器，修改下容器名字就可以了

docker run -t -d --name tmr1 \
--network app-tier \
-e JOB_MANAGER_RPC_ADDRESS=jmr   \
flink:1.9.2-scala_2.12 taskmanager

You can also modify docker-compose.yamlto add a file, copy it taskmangerconfiguration, modify a container under the name on it (I have not tried, it should be possible), command start, -dparameters docker docker-compose and refer to later in Taiwan in the form of start, so it will not print out a bunch of log, can docker logs [容器名]view the log command

docker-compose up -d [容器名]

4. flink: What 1.9.2-scala_2.12 that?
   A: The format of this string is Image name:tag, tagin 1.9.2 refers to flinkthe version scala_2.12is the version of this release is to use flink scala 2.12built. If you want to use other mirrors, can be used docker search [容器名,比如flink]to search, and then use the docker pull [容器名]command to pull the mirror