【DataSophon】集成Dinky1.0.1

企业开发 2024-11-04 18:40:37 阅读次数: 0

目录

一、概述

1.1什么是Dinky？

2.1 特性

2.2 核心特性

二、集成安装Dinky

2.1 下载Dinky

2.2 准备服务配置模板

2.3 准备配置文件service_ddl.json

2.4 DDP安装Dinky

一、概述

1.1什么是Dinky？

实时即未来，Dinky 为 Apache Flink 而生，让 Flink SQL 纵享丝滑。

Dinky 是一个开箱即用、易扩展，以 Apache Flink 为基础，连接 OLAP 和数据湖等众多框架的一站式实时计算平台，致力于流批一体和湖仓一体的探索与实践。 致力于简化Flink任务开发，提升Flink任务运维能力，降低Flink入门成本，提供一站式的Flink任务开发、运维、监控、报警、调度、数据管理等功能。

最后，Dinky 的发展皆归功于 Apache Flink 等其他优秀的开源项目的指导与成果。

官网：Dinky

2.1 特性

  • 沉浸式 FlinkSQL 数据开发：自动提示补全、语法高亮、语句美化、在线调试、语法校验、执行计划、Catalog支持、血缘分析等
  • Flink SQL语法增强，如 CDC任务，jar任务，实时打印表数据，实时数据预览，全局变量增强，语句合并、整库同步等
  • 适配 FlinkSQL 多种执行模式：Local、Standalone、Yarn/Kubernetes Session、Yarn Per-Job、Yarn/Kubernetes Application
  • 增强 Flink 生态拓展：Connector、FlinkCDC、Table Store 等
  • 支持 FlinkCDC 整库实时入仓入湖、多库输出、自动建表、模式演变
  • 支持 Flink Java / Scala / Python UDF 开发与自动提交
  • 支持 SQL 作业开发：ClickHouse、Doris、Hive、Mysql、Oracle、Phoenix、PostgreSql、Presto、SqlServer、StarRocks 等
  • 支持实时在线调试预览 Table、 ChangeLog、统计图和 UDF
  • 支持 Flink Catalog、Dinky内置Catalog增强，数据源元数据在线查询及管理
  • 支持自动托管的 SavePoint/CheckPoint 恢复及触发机制：最近一次、最早一次、指定一次等
  • 支持实时任务运维：作业信息、集群信息、作业快照、异常信息、历史版本、报警记录等
  • 支持作为多版本 FlinkSQL Server 以及 OpenApi 的能力
  • 支持实时作业报警及报警组：钉钉、微信企业号、飞书、邮箱等
  • 支持多种资源管理：集群实例、集群配置、数据源、报警组、报警实例、文档、系统配置等
  • 支持企业级管理功能：多租户、用户、角色、命名空间等
  • 更多隐藏功能等待小伙伴们探索

2.2 核心特性

二、集成安装Dinky

2.1 下载Dinky

mv  dinky-release-1.19-1.1.0.tar.gz  /opt/datasophon/DDP/packages
cd /opt/datasophon/DDP/packages
md5sum dinky-release-1.19-1.1.0.tar.gz

echo eb822c24a7bd5d46da5065c780bc0975 > dinky-release-1.19-1.1.0.tar.gz.md5

2.2 准备服务配置模板

cd  /opt/datasophon/datasophon-worker/conf/templates
cat dinky.ftl 
spring:
  datasource:
    url: ${databaseUrl}
    username: ${username}
    password: ${password}
    driver-class-name: com.mysql.cj.jdbc.Driver


# 分发ftl(如果在当前节点安装dinky的话就不用分发)
scp dinky.ftl ddp02:/opt/datasophon/datasophon-worker/conf/templates
scp dinky.ftl ddp03:/opt/datasophon/datasophon-worker/conf/templates

# 重启所有work节点
sh  /opt/datasophon/datasophon-worker/bin/datasophon-worker.sh restart worker

2.3 准备配置文件service_ddl.json

cd /opt/datasophon/DDP/packages/datasophon-manager-1.2.1/conf/meta/DDP-1.2.1
mkdir DINKY && cd DINKY

service_ddl.json

{
  "name": "DINKY",
  "label": "Dinky",
  "description": "流处理极速开发框架,流批一体&湖仓一体的云原生平台,一站式计算平台",
  "version": "1.0.1",
  "sortNum": 19,
  "dependencies":[],
  "packageName": "dinky-release-1.16-1.0.1.tar.gz",
  "decompressPackageName": "dinky-release-1.16-1.0.1",
  "roles": [
    {
      "name": "Dinky",
      "label": "Dinky",
      "roleType": "master",
      "cardinality": "1",
      "logFile": "logs/dinky.log",
      "jmxPort": 10087,
      "startRunner": {
        "timeout": "60",
        "program": "auto.sh",
        "args": [
          "startWithJmx",
          "1.16"
        ]
      },
      "stopRunner": {
        "timeout": "600",
        "program": "auto.sh",
        "args": [
          "stop"
        ]
      },
      "statusRunner": {
        "timeout": "60",
        "program": "auto.sh",
        "args": [
          "status"
        ]
      },
      "restartRunner": {
        "timeout": "60",
        "program": "auto.sh",
        "args": [
          "restart",
          "1.16"
        ]
      },
      "externalLink": {
        "name": "Dinky Ui",
        "label": "Dinky Ui",
        "url": "http://${host}:${serverPort}"
      }
    }
  ],
  "configWriter": {
    "generators": [
      {
        "filename": "application-mysql.yml",
        "configFormat": "custom",
        "outputDirectory": "config",
        "templateName": "dinky.ftl",
        "includeParams": [
          "databaseUrl",
          "username",
          "password",
          "serverPort"
        ]
      }
    ]
  },
  "parameters": [
    {
      "name": "databaseUrl",
      "label": "Dinky数据库地址",
      "description": "",
      "configType": "map",
      "required": true,
      "type": "input",
      "value": "",
      "configurableInWizard": true,
      "hidden": false,
      "defaultValue": "jdbc:mysql://${apiHost}:3306/dinky?useUnicode=true&characterEncoding=UTF-8&autoReconnect=true&useSSL=false&zeroDateTimeBehavior=convertToNull&serverTimezone=Asia/Shanghai&allowPublicKeyRetrieval=true"
    },
    {
      "name": "username",
      "label": "Dinky数据库用户名",
      "description": "",
      "configType": "map",
      "required": true,
      "type": "input",
      "value": "",
      "configurableInWizard": true,
      "hidden": false,
      "defaultValue": "root"
    },
    {
      "name": "password",
      "label": "Dinky数据库密码",
      "description": "",
      "configType": "map",
      "required": true,
      "type": "input",
      "value": "",
      "configurableInWizard": true,
      "hidden": false,
      "defaultValue": "123456"
    },
    {
      "name": "serverPort",
      "label": "Dinky服务端口",
      "description": "",
      "configType": "map",
      "required": true,
      "type": "input",
      "value": "",
      "configurableInWizard": true,
      "hidden": false,
      "defaultValue": "8888"
    }

  ]
}

重启datasophon-manager的api

sh /opt/datasophon/DDP/packages/datasophon-manager-1.2.1/bin/datasophon-api.sh restart api

创建数据库并且运行sql

mysql -u root -p -e "create database dinky";
mysql -u root -p -D dinky < /opt/datasophon/DDP/packages/dinky-mysql.sql;

2.4 DDP安装Dinky

找到Dinky 选择安装

配置元数据库连接信息

安装完成

grafana导入监控模板

dinky.json

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": {
          "type": "grafana",
          "uid": "-- Grafana --"
        },
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "target": {
          "limit": 100,
          "matchAny": false,
          "tags": [],
          "type": "dashboard"
        },
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 0,
  "id": 33,
  "links": [],
  "liveNow": false,
  "panels": [
    {
      "datasource": {
        "type": "prometheus",
        "uid": "hj6gjW44z"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [
            {
              "options": {
                "0": {
                  "text": "下线"
                },
                "1": {
                  "text": "正常"
                }
              },
              "type": "value"
            },
            {
              "options": {
                "match": "null",
                "result": {
                  "text": "下线"
                }
              },
              "type": "special"
            }
          ],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "#d44a3a",
                "value": null
              },
              {
                "color": "#e24d42",
                "value": 0
              },
              {
                "color": "#299c46",
                "value": 1
              }
            ]
          },
          "unit": "none"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 4,
        "w": 4,
        "x": 0,
        "y": 0
      },
      "hideTimeOverride": false,
      "id": 8,
      "links": [
        {
          "targetBlank": true,
          "title": "Tomcat dashboard",
          "url": "/d/chanjarster-tomcat-dashboard/tomcat-dashboard?$__url_time_range&$__all_variables"
        }
      ],
      "maxDataPoints": 100,
      "options": {
        "colorMode": "value",
        "graphMode": "none",
        "justifyMode": "auto",
        "orientation": "horizontal",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "text": {
          "valueSize": 38
        },
        "textMode": "auto"
      },
      "pluginVersion": "9.1.6",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "hj6gjW44z"
          },
          "editorMode": "code",
          "expr": "up{job=\"dinky\"}",
          "format": "time_series",
          "instant": true,
          "interval": "",
          "intervalFactor": 1,
          "legendFormat": "",
          "refId": "A"
        }
      ],
      "title": "Dinky状态",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "hj6gjW44z"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [
            {
              "options": {
                "match": "null",
                "result": {
                  "text": "N/A"
                }
              },
              "type": "special"
            }
          ],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              }
            ]
          },
          "unit": "dateTimeAsIso"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 4,
        "w": 8,
        "x": 4,
        "y": 0
      },
      "id": 2,
      "links": [],
      "maxDataPoints": 100,
      "options": {
        "colorMode": "value",
        "graphMode": "none",
        "justifyMode": "auto",
        "orientation": "horizontal",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "text": {
          "valueSize": 38
        },
        "textMode": "auto"
      },
      "pluginVersion": "9.1.6",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "hj6gjW44z"
          },
          "editorMode": "code",
          "expr": "process_start_time_seconds{job=\"dinky\"}*1000",
          "legendFormat": "__auto",
          "range": true,
          "refId": "A"
        }
      ],
      "title": "Dinky启动时间",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "hj6gjW44z"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [
            {
              "options": {
                "match": "null",
                "result": {
                  "text": "N/A"
                }
              },
              "type": "special"
            }
          ],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              }
            ]
          },
          "unit": "s"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 4,
        "w": 4,
        "x": 12,
        "y": 0
      },
      "id": 4,
      "links": [],
      "maxDataPoints": 100,
      "options": {
        "colorMode": "value",
        "graphMode": "none",
        "justifyMode": "auto",
        "orientation": "horizontal",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "text": {
          "valueSize": 38
        },
        "textMode": "auto"
      },
      "pluginVersion": "9.1.6",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "hj6gjW44z"
          },
          "editorMode": "code",
          "expr": "time() - process_start_time_seconds{job=\"dinky\"}",
          "interval": "",
          "legendFormat": "",
          "range": true,
          "refId": "A"
        }
      ],
      "title": "Dinky运行时长",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "hj6gjW44z"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [
            {
              "options": {
                "match": "null",
                "result": {
                  "text": "N/A"
                }
              },
              "type": "special"
            }
          ],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              }
            ]
          },
          "unit": "bytes"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 4,
        "w": 4,
        "x": 16,
        "y": 0
      },
      "id": 6,
      "links": [],
      "maxDataPoints": 100,
      "options": {
        "colorMode": "value",
        "graphMode": "none",
        "justifyMode": "auto",
        "orientation": "horizontal",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "text": {
          "valueSize": 38
        },
        "textMode": "auto"
      },
      "pluginVersion": "9.1.6",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "hj6gjW44z"
          },
          "editorMode": "code",
          "expr": "jvm_memory_bytes_max{job=\"dinky\",area=\"heap\"}",
          "legendFormat": "__auto",
          "range": true,
          "refId": "A"
        }
      ],
      "title": "Dinky堆内存",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "hj6gjW44z"
      },
      "fieldConfig": {
        "defaults": {
          "mappings": [],
          "max": 100,
          "min": 0,
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "%"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 4,
        "w": 4,
        "x": 20,
        "y": 0
      },
      "id": 10,
      "options": {
        "orientation": "auto",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showThresholdLabels": false,
        "showThresholdMarkers": true
      },
      "pluginVersion": "9.1.6",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "hj6gjW44z"
          },
          "editorMode": "code",
          "expr": "jvm_memory_bytes_used{area=\"heap\",job=\"dinky\"}*100/jvm_memory_bytes_max{area=\"heap\",job=\"dinky\"}",
          "legendFormat": "__auto",
          "range": true,
          "refId": "A"
        }
      ],
      "title": "Dinky堆内存使用率",
      "type": "gauge"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "hj6gjW44z"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 10,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "never",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "links": [],
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "bytes"
        },
        "overrides": [
          {
            "matcher": {
              "id": "byName",
              "options": "Usage %"
            },
            "properties": [
              {
                "id": "custom.drawStyle",
                "value": "bars"
              },
              {
                "id": "custom.fillOpacity",
                "value": 100
              },
              {
                "id": "color",
                "value": {
                  "fixedColor": "#6d1f62",
                  "mode": "fixed"
                }
              },
              {
                "id": "custom.lineWidth",
                "value": 0
              },
              {
                "id": "unit",
                "value": "percentunit"
              },
              {
                "id": "min",
                "value": 0
              },
              {
                "id": "max",
                "value": 1
              }
            ]
          }
        ]
      },
      "gridPos": {
        "h": 10,
        "w": 12,
        "x": 0,
        "y": 4
      },
      "id": 12,
      "links": [],
      "options": {
        "legend": {
          "calcs": [
            "mean",
            "max"
          ],
          "displayMode": "table",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "mode": "multi",
          "sort": "none"
        }
      },
      "pluginVersion": "9.1.6",
      "repeat": "memarea",
      "repeatDirection": "h",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "hj6gjW44z"
          },
          "editorMode": "code",
          "expr": "jvm_memory_bytes_used{area=\"heap\",job=\"dinky\"}",
          "legendFormat": "已用内存",
          "range": true,
          "refId": "A"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "hj6gjW44z"
          },
          "editorMode": "code",
          "expr": " jvm_memory_bytes_max{area=\"heap\",job=\"dlink\"}",
          "hide": false,
          "legendFormat": "总内存",
          "range": true,
          "refId": "B"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "hj6gjW44z"
          },
          "editorMode": "code",
          "expr": "jvm_memory_bytes_used{area=\"heap\",job=\"dlink\"} / jvm_memory_bytes_max >= 0",
          "hide": false,
          "legendFormat": "使用率",
          "range": true,
          "refId": "C"
        }
      ],
      "title": "Dinky堆内存使用趋势",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "hj6gjW44z"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 10,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "never",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "links": [],
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "s"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 10,
        "w": 12,
        "x": 12,
        "y": 4
      },
      "id": 14,
      "links": [],
      "options": {
        "legend": {
          "calcs": [
            "lastNotNull",
            "max",
            "min"
          ],
          "displayMode": "table",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "mode": "multi",
          "sort": "none"
        }
      },
      "pluginVersion": "9.1.6",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "hj6gjW44z"
          },
          "editorMode": "code",
          "expr": "increase(jvm_gc_collection_seconds_sum{job=\"dinky\"}[$__interval])",
          "format": "time_series",
          "interval": "60s",
          "intervalFactor": 1,
          "legendFormat": "{
   
   {gc}}",
          "metric": "jvm_gc_collection_seconds_sum",
          "range": true,
          "refId": "A",
          "step": 10
        }
      ],
      "title": "Dinky GC时间趋势图",
      "type": "timeseries"
    }
  ],
  "schemaVersion": 37,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": []
  },
  "time": {
    "from": "now-6h",
    "to": "now"
  },
  "timepicker": {},
  "timezone": "",
  "title": "Dinky",
  "uid": "9qU9T1OVk",
  "version": 4,
  "weekStart": ""
}

查看datasophon数据库中t_ddh_cluster_service_dashboard表中是否原就存在dinky 如果不存在添加

grafama正常展示

回到datasophon的dinky服务，刷新即可在总览看到详细监控信息。

登录地址

http://192.168.2.100:8888/#/user/login admin/dinky123!@#

登录后界面

参考文章：

Dinky 让Flink作业纵享丝滑_flink dinky-CSDN博客

Dinky介绍 | Dinky

猜你喜欢

转载自blog.csdn.net/qq_35995514/article/details/143452449
今日推荐
周排行
Sping整合ActiveMQ（五.常见错误分析）
jquery ajax发送请求实例模板
北风设计模式课程---24、迭代模式
[Luogu] 兽径管理
1030 Travel Plan （30 分）(dijkstra算法+dfs+边权)
springboot-shiro中的问题
数据访问安全代理 CASB
RocketMQ与Kafka对比
Rider 2019.3.3 发布，跨平台 .NET IDE
Ubuntu切换root su -
每日归档
更多
2025-03-17(0)
2025-03-16(0)
2025-03-15(0)
2025-03-14(0)
2025-03-13(0)
2025-03-12(0)
2025-03-11(0)
2025-03-10(0)
2025-03-09(0)
2025-03-08(0)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022