dataX mysql驱动5.x升级8.x
升级背景
dolphinscheduler datax 执行报错
提示应使用新的驱动版本，现有驱动是5.x,数据库却是8.x,因此报错
2021-12-14 10:52:08.639 [main] WARN Engine - prioriy set to 0, because NumberFormatException, the value is: null
2021-12-14 10:52:08.640 [main] INFO PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
2021-12-14 10:52:08.640 [main] INFO JobContainer - DataX jobContainer starts job.
2021-12-14 10:52:08.641 [main] INFO JobContainer - Set jobId = 0
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
全部替换驱动包为高版本后，依然报相同的错
[dolphinscheduler@host1 datax]$ find . -name mysql*
./plugin/reader/mysqlreader
./plugin/reader/mysqlreader/libs/mysql-connector-java-8.0.25.jar
./plugin/reader/mysqlreader/mysqlreader-0.0.1-SNAPSHOT.jar
./plugin/reader/drdsreader/libs/mysql-connector-java-8.0.25.jar
./plugin/writer/mysqlwriter
./plugin/writer/mysqlwriter/libs/mysql-connector-java-8.0.25.jar
./plugin/writer/mysqlwriter/mysqlwriter-0.0.1-SNAPSHOT.jar
./plugin/writer/drdswriter/libs/mysql-connector-java-8.0.25.jar
./plugin/writer/adswriter/libs/mysql-connector-java-8.0.25.jar
[dolphinscheduler@host1 datax]$
版本升级
下载datax源码
下载地址：https://github.com/alibaba/DataX.git
导入工程，修改配置
修改pom.xml文件，全文搜索mysql-connector-java关键字，修改版本号为8.0.25
修改java文件，全文替换com.mysql.jdbc.Driver为com.mysql.cj.jdbc.Driver
驱动升级后，修改报错代码AdsInsertProxy.java
导入相应类，并对应进行修改
import com.mysql.cj.jdbc.JdbcPreparedStatement;
import com.mysql.cj.jdbc.exceptions.CommunicationsException;
.asSql()改成.toString();
//sql = ((JDBC4PreparedStatement) statement).asSql();
sql = ((JdbcPreparedStatement) statement).toString();
//retryExceptionClasss.add(com.mysql.jdbc.exceptions.jdbc4.CommunicationsException.class);
retryExceptionClasss.add(CommunicationsException.class);
编译项目
clean install -Prelease assembly:assembly
验证项目
驱动包版本
部署环境验证
版本包大小一个G，待验证
上传两小时，验证一分钟，错误解决
[INFO] 2021-12-14 18:23:33.454 - [taskAppId=TASK-16-4850-48955]:[127] - task dir : /tmp/dolphinscheduler/exec/process/3/16/4850/48955
[INFO] 2021-12-14 18:23:33.454 - [taskAppId=TASK-16-4850-48955]:[141] - datax task params {"targetTable":"t_ds_user_datax","postStatements":[],"jobSpeedRecord":1000,"dtType":"MYSQL","dsType":"MYSQL","jobSpeedByte":0,"dataSource":1,"dataTarget":6,"sql":"select user_name from t_ds_user where id=1 ;","preStatements":["truncate table t_ds_user_datax;"]}
[INFO] 2021-12-14 18:23:33.464 - [taskAppId=TASK-16-4850-48955]:[105] - tenantCode user:lwexecut, task dir:16_4850_48955
[INFO] 2021-12-14 18:23:33.464 - [taskAppId=TASK-16-4850-48955]:[109] - create command file:/tmp/dolphinscheduler/exec/process/3/16/4850/48955/16_4850_48955.command
[INFO] 2021-12-14 18:23:33.464 - [taskAppId=TASK-16-4850-48955]:[128] - command : #!/bin/sh
BASEDIR=$(cd `dirname $0`; pwd)
cd $BASEDIR
source /home/dolphinscheduler/app/dolphinscheduler/conf/env/dolphinscheduler_env.sh
/tmp/dolphinscheduler/exec/process/3/16/4850/48955/16_4850_48955_node.sh
[INFO] 2021-12-14 18:23:33.467 - [taskAppId=TASK-16-4850-48955]:[375] - task run command:
sudo -u lwexecut sh /tmp/dolphinscheduler/exec/process/3/16/4850/48955/16_4850_48955.command
[INFO] 2021-12-14 18:23:33.473 - [taskAppId=TASK-16-4850-48955]:[160] - process start, process id is: 299030
[INFO] 2021-12-14 18:23:33.819 - [taskAppId=TASK-16-4850-48955]:[109] - ->
DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.
2021-12-14 18:23:33.803 [main] INFO VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2021-12-14 18:23:33.808 [main] INFO Engine - the machine info =>
osInfo: Oracle Corporation 1.8 25.151-b12
jvmInfo: Linux amd64 3.10.0-693.el7.x86_64
cpu num: 80
totalPhysicalMemory: -0.00G
freePhysicalMemory: -0.00G
maxFileDescriptorCount: -1
currentOpenFileDescriptorCount: -1
GC Names [PS MarkSweep, PS Scavenge]
MEMORY_NAME | allocation_size | init_size
PS Eden Space | 256.00MB | 256.00MB
Code Cache | 240.00MB | 2.44MB
Compressed Class Space | 1,024.00MB | 0.00MB
PS Survivor Space | 42.50MB | 42.50MB
PS Old Gen | 683.00MB | 683.00MB
Metaspace | -0.00MB | 0.00MB
2021-12-14 18:23:33.819 [main] INFO Engine -
{
"content":[
{
"reader":{
"name":"mysqlreader",
"parameter":{
"connection":[
{
"jdbcUrl":[
"jdbc:mysql://192.168.3.8:3306/dolphinscheduler_db"
],
"querySql":[
"select user_name from t_ds_user where id=1 ;"
]
}
],
"password":"******",
"username":"dolphinscheduler"
}
},
"writer":{
"name":"mysqlwriter",
"parameter":{
"column":[
"`user_name`"
],
"connection":[
{
"jdbcUrl":"jdbc:mysql://192.168.3.8:3306/dolphinscheduler_db",
"table":[
"t_ds_user_datax"
]
}
],
"password":"******",
[INFO] 2021-12-14 18:23:44.378 - [taskAppId=TASK-16-4850-48955]:[109] - -> "preSql":[
"truncate table t_ds_user_datax;"
],
"username":"dolphinscheduler"
}
}
}
],
"setting":{
"errorLimit":{
"percentage":0,
"record":0
},
"speed":{
"channel":1,
"record":1000
}
}
}
2021-12-14 18:23:33.829 [main] WARN Engine - prioriy set to 0, because NumberFormatException, the value is: null
2021-12-14 18:23:33.830 [main] INFO PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
2021-12-14 18:23:33.831 [main] INFO JobContainer - DataX jobContainer starts job.
2021-12-14 18:23:33.832 [main] INFO JobContainer - Set jobId = 0
2021-12-14 18:23:34.154 [job-0] INFO OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:mysql://192.168.3.8:3306/dolphinscheduler_db?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true.
2021-12-14 18:23:34.276 [job-0] INFO OriginalConfPretreatmentUtil - table:[t_ds_user_datax] all columns:[
id,user_name,user_password,user_type,email,phone,tenant_id,create_time,update_time,queue
].
2021-12-14 18:23:34.289 [job-0] INFO OriginalConfPretreatmentUtil - Write data [
INSERT INTO %s (`user_name`) VALUES(?)
], which jdbcUrl like:[jdbc:mysql://192.168.3.8:3306/dolphinscheduler_db?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&rewriteBatchedStatements=true&tinyInt1isBit=false]
2021-12-14 18:23:34.289 [job-0] INFO JobContainer - jobContainer starts to do prepare ...
2021-12-14 18:23:34.289 [job-0] INFO JobContainer - DataX Reader.Job [mysqlreader] do prepare work .
2021-12-14 18:23:34.289 [job-0] INFO JobContainer - DataX Writer.Job [mysqlwriter] do prepare work .
2021-12-14 18:23:34.299 [job-0] INFO CommonRdbmsWriter$Job - Begin to execute preSqls:[truncate table t_ds_user_datax;]. context info:jdbc:mysql://192.168.3.8:3306/dolphinscheduler_db?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&rewriteBatchedStatements=true&tinyInt1isBit=false.
2021-12-14 18:23:34.307 [job-0] INFO JobContainer - jobContainer starts to do split ...
2021-12-14 18:23:34.307 [job-0] INFO JobContainer - Job set Max-Record-Speed to 1000 records.
2021-12-14 18:23:34.309 [job-0] INFO JobContainer - DataX Reader.Job [mysqlreader] splits to [1] tasks.
2021-12-14 18:23:34.309 [job-0] INFO JobContainer - DataX Writer.Job [mysqlwriter] splits to [1] tasks.
2021-12-14 18:23:34.319 [job-0] INFO JobContainer - jobContainer starts to do schedule ...
2021-12-14 18:23:34.363 [job-0] INFO JobContainer - Scheduler starts [1] taskGroups.
2021-12-14 18:23:34.365 [job-0] INFO JobContainer - Running by standalone Mode.
2021-12-14 18:23:34.368 [taskGroup-0] INFO TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
2021-12-14 18:23:34.370 [taskGroup-0] INFO Channel - Channel set byte_speed_limit to -1, No bps activated.
2021-12-14 18:23:34.370 [taskGroup-0] INFO Channel - Channel set record_speed_limit to 1000.
2021-12-14 18:23:34.375 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2021-12-14 18:23:34.378 [0-0-0-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select user_name from t_ds_user where id=1 ;
] jdbcUrl:[jdbc:mysql://192.168.3.8:3306/dolphinscheduler_db?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2021-12-14 18:23:34.404 [0-0-0-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select user_name from t_ds_user where id=1 ;
] jdbcUrl:[jdbc:mysql://192.168.3.8:3306/dolphinscheduler_db?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2021-12-14 18:23:34.677 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[303]ms
2021-12-14 18:23:34.677 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] completed it's tasks.
2021-12-14 18:23:44.378 [job-0] INFO StandAloneJobContainerCommunicator - Total 1 records, 5 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 100.00%
[INFO] 2021-12-14 18:23:44.394 - [taskAppId=TASK-16-4850-48955]:[109] - -> 2021-12-14 18:23:44.378 [job-0] INFO AbstractScheduler - Scheduler accomplished all tasks.
2021-12-14 18:23:44.378 [job-0] INFO JobContainer - DataX Writer.Job [mysqlwriter] do post work.
2021-12-14 18:23:44.378 [job-0] INFO JobContainer - DataX Reader.Job [mysqlreader] do post work.
2021-12-14 18:23:44.378 [job-0] INFO JobContainer - DataX jobId [0] completed successfully.
2021-12-14 18:23:44.379 [job-0] INFO HookInvoker - No hook invoked, because base dir not exists or is a file: /home/dolphinscheduler/app/datax/hook
2021-12-14 18:23:44.380 [job-0] INFO JobContainer -
[total cpu info] =>
averageCpu | maxDeltaCpu | minDeltaCpu
-1.00% | -1.00% | -1.00%
[total gc info] =>
NAME | totalGCCount | maxDeltaGCCount | minDeltaGCCount | totalGCTime | maxDeltaGCTime | minDeltaGCTime
PS MarkSweep | 1 | 1 | 1 | 0.025s | 0.025s | 0.025s
PS Scavenge | 1 | 1 | 1 | 0.015s | 0.015s | 0.015s
2021-12-14 18:23:44.380 [job-0] INFO JobContainer - PerfTrace not enable!
2021-12-14 18:23:44.380 [job-0] INFO StandAloneJobContainerCommunicator - Total 1 records, 5 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 100.00%
2021-12-14 18:23:44.380 [job-0] INFO JobContainer -
任务启动时刻 : 2021-12-14 18:23:33
任务结束时刻 : 2021-12-14 18:23:44
任务总计耗时 : 10s
任务平均流量 : 0B/s
记录写入速度 : 0rec/s
读出记录总数 : 1
读写失败总数 : 0
[INFO] 2021-12-14 18:23:44.394 - [taskAppId=TASK-16-4850-48955]:[170] - process has exited, work dir:/tmp/dolphinscheduler/exec/process/3/16/4850/48955, pid:299030 ,exitStatusCode:0
[INFO] 2021-12-14 18:23:44.400 - [taskAppId=TASK-16-4850-48955]:[277] - process id is 299030