经常看到有朋友通过hadoop命令运行job的时候加上"-libjars"参数导入第三方jar文件,今天一位同事问我他写的job使用这个参数为什么不生效,我简单看了看源码,得知其实是"GenericOptionsParser"这个类对conf做了修改,当在job中执行如下命令的时候,在conf中设置了"tmpjars"属性。
new GenericOptionsParser(conf, args);
if (line.hasOption("libjars")) { conf.set("tmpjars", validateFiles(line.getOptionValue("libjars"), conf)); //setting libjars in client classpath URL[] libjars = getLibJars(conf); if(libjars!=null && libjars.length>0) { conf.setClassLoader(new URLClassLoader(libjars, conf.getClassLoader())); Thread.currentThread().setContextClassLoader( new URLClassLoader(libjars, Thread.currentThread().getContextClassLoader())); } }
另外我们可以通过ToolRunner实现自动解析libjars参数, ToolRunner.run()将使用GenericOptionsParser解析libjars参数,如下:
public class LzoSort extends Configured implements Tool { @Override public int run(String[] args) throws Exception { getConf().set("io.compression.codecs", "org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,com.hadoop.compression.lzo.LzopCodec"); getConf().set("io.compression.codec.lzo.class", "com.hadoop.compression.lzo.LzoCodec"); Job job = new Job(getConf(), "lzo sort"); // ............... return job.waitForCompletion(true) ? 0 : 1; } public static void main(String[] args) throws Exception { int res = ToolRunner.run(new Configuration(), new LzoSort(), args); System.exit(res); } }
-- end --