Hadoop has a number of built-in mechanisms that can facilitate ingress and egress operations, to name a few:
- Embedded NameNode HTTP server
- WebHDFS and Hadoop interfaces
- Hbase built-in API, be specifically the org.apache.hadoop.hbase.mapreduce.TableInputFormat and org.apache.hadoop.hbase.mapreduce.TableOutputFormat
- Hadoop fs utility
- Oozie workflow can also be leveraged to move data in/out of HDFS periodically
- Log collector such as Flume, scribe, Chukwa...
- Sqoop to import/export data from/to HDFS and relational databases
The first three are low-level utilities while the rest are high-level. These low-level mechanisms don't provide complete systems to manage the entire ingress and egress process while the hig-level ones do. For ingress here I mean move data into HDFS from external filesystem, database, hbase etc.; for egress, I mean move data from external system into Hadoop HDFS.
But in this blog post, what I most interested in is a third-party script which could be used to move data in/out of HDFS from external filesystem. It's called hdfs-file-splurper which is hosted on GitHub. It's distributed under Apache license, so you can get a copy of it source code via this link.
With HDFS file slurper open source project, you can copy files of any format in/out of HDFS, but with log collector such as Flume, Chukwa or Scribe, you can only copy text file format. The HDFS file slurper was developed by Alex Holmes, who is also the author of Hadoop in Practice. It's an awesome book by the way. Here is brief demonstration of this useful project.
- Case 1, you want to move files in any format from external filesystem to HDFS, say, from the localhost directory /tmp/slurper/in to the HDFS directory /incoming/:
shell$ cat conf/examples/basic.conf DATASOURCE_NAME = test SRC_DIR = file:/tmp/slurper/in WORK_DIR = file:/tmp/slurper/work COMPLETE_DIR = file:/tmp/slurper/complete ERROR_DIR = file:/tmp/slurper/error DEST_STAGING_DIR = hdfs:/incoming/stage DEST_DIR = hdfs:/incoming
Run the Slurper in foreground mode in a console:shell$ bin/slurper.sh \ --config-file /path/to/slurper/conf/examples/basic.conf
In another console create an empty file and watch the Slurper do its stuff:shell$ echo "blocks of text" > /tmp/slurper/in/test.txt
- Case 2, you want to move files of any format from HDFS to local filesystem.
shell$ cat conf/exmaples/basic.conf SRC_DIR=hdfs:/tmp/slurper/in WORK_DIR=hdfs:/tmp/slurper/work COMPLETE_DIR=hdfs:/tmp/slurper/complete ERROR_DIR=hdfs:/tmp/slurper/error DEST_STAGING_DIR=file:/tmp/slurper/stage DEST_DIR=file:/tmp/slurper/dest
Run Hadoop job to write output into the source directory, then see the Slurper do its job.
Important features of HDFS file slurper:
- After a successful file copy you can either remove the source file, or have it moved into another directory.
- Destination files can be compressed as part of the write codec with any compression codec which extends org.apache.hadoop.io.compress.CompressionCodec.
- Capability to write "done" file after completion of copy
- Verify destination file post-copy with CRC32 checksum comparison with source
- Ignores hidden files (filenames that start with ".")
- Customizable destination via a script which can be called for every source file. Or alternatively let the utility know a single destination directory
- Customizable pre-processing of file prior to transfer via script and all files are copied into that location.
- A daemon mode which is compatible with inittab respawn
- Multi-threaded data transfer
For more information of this project, please visit its homepage on GitHub and get a copy of its source code.