Teach you step by step how to use SeaTunnel k8s to run Zeta engine local mode

[Live broadcast preview] Will large models replace programmers? "

Reprinted from Xiao Xiami 0.0

Introduction: With the popularity and development of Kubernetes, more and more enterprises and teams are beginning to use Kubernetes to manage and deploy applications. However, the default way Kubernetes works may not be the best choice, especially when more efficient and reliable application deployment and management capabilities are needed. Therefore, SeaTunnel provides a way to run the Zeta engine (local-mode mode), which allows Kubernetes to run the Zeta engine locally to achieve more efficient application deployment and management. In this article, we will explore more about SeaTunnel k8s running zeta engine (local-mode mode) and learn how to better take advantage of the Zeta engine.

run

1. Upload SeaTunnel to the server. I have decompressed and executed install-plugin.sh before. For convenience, I directly used the seatunnel after executing the install-plugin.sh script for demonstration.

The lib directory after executing install-plugin contains the following

tar -zxvf apache-seatunnel-2.3.3-bin.tar.gz
sh apache-seatunnel-2.3.3/bin/install-plugin.sh
tar -czvf  apache-seatunnel-2.3.3-bin.tar.gz  apache-seatunnel-2.3.3

2. Build the SeaTunnel image. Create a Dockerfile in the same folder where seatunnel is installed. The configuration is as follows, you can choose the version yourself

FROM openjdk:8

ENV SEATUNNEL_VERSION="2.3.3"
COPY /apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz /opt/apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
WORKDIR /opt
RUN tar -xzvf apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
RUN mv apache-seatunnel-${SEATUNNEL_VERSION} seatunnel
RUN rm -f /opt/apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
WORKDIR /opt/seatunnel

Excuting an order

docker build -t seatunnel:2.3.3 -f Dockerfile.

3. View image

docker images

Mirrored as shown below

4. Load the image into k8s. Minikube is used here for demonstration.

minikube image load seatunnel:2.3.3

minikube image ls 

View image

6. Create seatunnel.streaming.conf as shown below

env {
  execution.parallelism = 1
  job.mode = "STREAMING"
  checkpoint.interval = 2000
}

source {
    FakeSource {
      result_table_name = "fake"
      row.num = 160000
      schema = {
        fields {
          name = "string"
          age = "int"
        }
      }
    }
}

transform {

}

sink {
  Console {}
}

7. Create configmap

kubectl create cm seatunnel-config \
--from-file=seatunnel.streaming.conf=seatunnel.streaming.conf

8. Create seatunnel.yaml as follows example

apiVersion: v1
kind: Pod
metadata:
  name: seatunneltest
spec:
  containers:
  - name: seatunnel
    image: seatunnel:2.3.3
    command: ["/bin/sh","-c","/opt/seatunnel/bin/seatunnel.sh --config /data/seatunnel.streaming.conf -e local"]
    volumeMounts:
      - name: seatunnel-config
        mountPath: /data/seatunnel.streaming.conf
        subPath: seatunnel.streaming.conf
  volumes:
        - name: seatunnel-config
          configMap:
            name: seatunnel-config
            items:
            - key: seatunnel.streaming.conf
              path: seatunnel.streaming.conf
~                                                  

implement

kubectl apply -f seatunnel.yaml

View the pod results as follows

Copyright statement: This article is an original article by the blogger and follows the CC 4.0 BY-SA copyright agreement. Please attach the original source link and this statement when reprinting.

Original link: https://blog.csdn.net/weixin_41854429/article/details/132619265

This article is published by Beluga Open Source Technology !

Linus took matters into his own hands to prevent kernel developers from replacing tabs with spaces. His father is one of the few leaders who can write code, his second son is the director of the open source technology department, and his youngest son is a core contributor to open source. Huawei: It took 1 year to convert 5,000 commonly used mobile applications Comprehensive migration to Hongmeng Java is the language most prone to third-party vulnerabilities. Wang Chenglu, the father of Hongmeng: open source Hongmeng is the only architectural innovation in the field of basic software in China. Ma Huateng and Zhou Hongyi shake hands to "remove grudges." Former Microsoft developer: Windows 11 performance is "ridiculously bad " " Although what Laoxiangji is open source is not the code, the reasons behind it are very heartwarming. Meta Llama 3 is officially released. Google announces a large-scale restructuring

{{o.name}}

{{m.name}}