Apache SeaTunnel k8s cluster mode Zeta engine deployment guide

The open source China community team made its first live broadcast, telling the story behind the open source China community in the name of sharing."

SeaTunnel provides a method to run the Zeta engine (cluster-mode), which allows Kubernetes to run the Zeta engine locally to achieve more efficient application deployment and management. In this article, we will explore more about SeaTunnel k8s running zeta engine (cluster-mode mode) and learn how to better take advantage of the Zeta engine.

1. Upload SeaTunnel to the server. I have decompressed and executed install-plugin.sh before. For convenience, I directly used the seatunnel after executing the install-plugin.sh script for demonstration.

The lib directory after executing install-plugin contains the following:

tar -zxvf apache-seatunnel-2.3.3-bin.tar.gz
sh apache-seatunnel-2.3.3/bin/install-plugin.sh
tar -czvf  apache-seatunnel-2.3.3-bin.tar.gz  apache-seatunnel-2.3.3

2. Build the seatunnel image. Create a Dockerfile in the same folder where seatunnel is installed. The configuration is as follows, you can choose the version yourself:

FROM openjdk:8
ENV SEATUNNEL_HOME="/opt/seatunnel"
ENV SEATUNNEL_VERSION="2.3.3"
COPY /apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz /opt/apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
WORKDIR /opt
RUN tar -xzvf apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
RUN mv apache-seatunnel-${SEATUNNEL_VERSION} seatunnel
RUN rm -f /opt/apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
WORKDIR /opt/seatunnel

Excuting an order

docker build -t seatunnel:2.3.3 -f Dockerfile.

3. View image

docker images

Mirrored as shown below

4. Load the image into k8s. Minikube is used here for demonstration.

minikube image load seatunnel:2.3.3

Please refer to the above: Teach you step by step how to master the skills of running Zeta engine local mode on SeaTunnel k8s

5. Create configmap as follows

kubectl create configmap hazelcast-client  --from-file= config/hazelcast-client.yaml
kubectl create configmap hazelcast  --from-file=config/hazelcast.yaml
kubectl create configmap seatunnelmap  --from-file=config/seatunnel.yaml

6. Use Reloader to automatically restart pods after updating configmap

wget https://raw.githubusercontent.com/stakater/Reloader/master/deployments/kubernetes/reloader.yaml

kubectl apply -f reloader.yaml

7. Create seatunnel-cluster.yml as follows

apiVersion: v1
kind: Service
metadata:
  name: seatunnel
spec:
  selector:
    app: seatunnel
  ports:
  - port: 5801
    name: seatunnel
  clusterIP: None
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: seatunnel
  annotations:
    configmap.reloader.stakater.com/reload: "hazelcast,hazelcast-client,seatunnelmap"
spec:
  serviceName: "seatunnel"
  replicas: 3
  selector:
    matchLabels:
      app: seatunnel
  template:
    metadata:
      labels:
        app: seatunnel
    spec:
      containers:
        - name: seatunnel
          image: seatunnel:2.3.3
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 5801
              name: client
          command: ["/bin/sh","-c","/opt/seatunnel/bin/seatunnel-cluster.sh -DJvmOption=-Xms2G -Xmx2G"]
          resources:
            limits:
              cpu: "1"
              memory: 4G
            requests:
              cpu: "1"
              memory: 2G
          volumeMounts:
            - mountPath: "/opt/seatunnel/config/hazelcast.yaml"
              name: hazelcast
              subPath: hazelcast.yaml
            - mountPath: "/opt/seatunnel/config/hazelcast-client.yaml"
              name: hazelcast-client
              subPath: hazelcast-client.yaml
            - mountPath: "/opt/seatunnel/config/seatunnel.yaml"
              name: seatunnelmap
              subPath: seatunnel.yaml
      volumes:
        - name: hazelcast
          configMap:
            name: hazelcast
        - name: hazelcast-client
          configMap:
            name: hazelcast-client
        - name: seatunnelmap
          configMap:
            name: seatunnelmap

8. implement

 kubectl apply -f seatunnel-cluster.yml

9. Modify the configuration in configmap

kubectl edit cm hazelcast

修改集群地址 
这里采用的是headless service访问模式
一般pod之间访问其格式为 <pod-name>.<service-name>.<namespace>.svc.cluster.local

Example as below

• seatunnel-0.seatunnel.default.svc.cluster.local
• seatunnel-1.seatunnel.default.svc.cluster.local
• seatunnel-2.seatunnel.default.svc.cluster.local

Friendly reminder: Don’t use tabs, use spaces. Otherwise, an error will be reported

kubectl edit cm hazelcast-client

kubectl edit cm seatunnelmap

Change here to your own hdfs address.

10. You can see the following

11. After all nodes have been updated and are in the running state, you can enter the container to check whether the path has been modified.

kubectl exec -it seatunnel-0 /bin/bash
cat config/hazelcast.yaml

12. View logs within the container

tail -200f logs/seatunnel-engine-server.log

We found that the cluster is running normally.

13. Run tasks

We can open a new connection and log in to another pod node to perform tasks to test the cluster:

kubectl exec -it seatunnel-1 /bin/bash

bin/seatunnel.sh --config config/v2.streaming.conf.template

We found that tasks have also started running in other pods

Original link: https://blog.csdn.net/weixin_41854429/article/details/132836402

This article is published by Beluga Open Source Technology !

The Google Python Foundation team was laid off. Google confirmed the layoffs, and the teams involved in Flutter, Dart and Python rushed to the GitHub hot list - How can open source programming languages ​​and frameworks be so cute? Xshell 8 opens beta test: supports RDP protocol and can remotely connect to Windows 10/11. When passengers connect to high-speed rail WiFi , the "35-year-old curse" of Chinese coders pops up when they connect to high-speed rail WiFi. MySQL's first long-term support version 8.4 GA AI search tool Perplexica : Completely open source and free, an open source alternative to Perplexity. Huawei executives evaluate the value of open source Hongmeng: It still has its own operating system despite continued suppression by foreign countries. German automotive software company Elektrobit open sourced an automotive operating system solution based on Ubuntu.

{{o.name}}

{{m.name}}