Compile datax

Ali Baba datax open official data synchronization of a tool, can be used to synchronize data between many sources, and easy to use, efficient.

datax official has to provide compiled versions can be downloaded directly, but which contains a BUG.

I recently encountered a problem that, when MongoDB to synchronize data from traditional structured databases (such as MySQL), the situation can not be aligned field will appear.

for example. For example in the presence of such a MongoDB Collection:

When synchronized to mysql later, would this be:

Obviously, it is only in order to map to the target terminal, and by field names do not match.

To solve the problem: recompile datax

Download datax directly from the source in github, I wanted to just compile it mongodbreader plug-ins, but has been an error, then think should be the entire source code to compile job.

After datax cd into the directory, the official compiler command:

mvn -U clean package assembly:assembly -Dmaven.test.skip=true

Compiled directly is to be wrong, it will depend on several newspaper can not find, because maven pom.xml file in the dependent source inside setting using the older version, there will be problems, so they need a modification.

To modify where there are three:

odpsreader / pom.xml and odpswriter / pom.xml

<dependency>
    <groupId>com.aliyun.odps</groupId>
    <artifactId>odps-sdk-core</artifactId>
    <version>0.20.7-public</version>
</dependency>

otsstreamreader/pom.xml

<dependency>
    <groupId>com.aliyun.openservices</groupId>
    <artifactId>tablestore-streamclient</artifactId>
    <version>1.0.0</version>
</dependency>

After modification, and then execute the command packaging:

mvn -U clean package assembly:assembly -Dmaven.test.skip=true

The final saw BUILD SUCCESS to indicate the success of the package. The resulting object files in a datax/target/directory

reference:

[. 1]. MongoDB read data written hdfs, loss of field
[2]. ODPs dependencies do not cause the compiler
[. 3]. Otsstreamreader given widget compiler dependent pom
[. 4]. DataX UserGuide