关于window下ecplise的hadoop使用copytolocalfile异常的问题

关于window下ecplise的hadoop使用copytolocalfile异常的问题

先贴异常

Exception in thread "main" java.lang.UnsatisfiedLinkError: 
org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileWithMode0
(Ljava/lang/String;JJJI)Ljava/io/FileDescriptor;

这种情况说明用错copyToLocalFile的方法
方法有
1.copyToLocalFile(Path src, Path dst)
2.copyToLocalFile(boolean delSrc, Path src, Path dst)
3.copyToLocalFile(boolean delSrc, Path src, Path dst,
boolean useRawLocalFileSystem)

只有第三个才可以

public void copyToLocalFile(boolean delSrc,
                   Path src,
                   Path dst,
                   boolean useRawLocalFileSystem)
                     throws IOException
The src file is under FS, and the dst is on the local disk. Copy it from FS control to the local dst name. delSrc indicates if the src will be removed or not. useRawLocalFileSystem indicates whether to use RawLocalFileSystem as local file system or not. RawLocalFileSystem is non crc file system.So, It will not create any crc files at local.
Parameters:
delSrc - whether to delete the src
src - path
dst - path
useRawLocalFileSystem - whether to use RawLocalFileSystem as local file system or not.
Throws:
IOException - - if any IO error

useRawLocalFileSystem指的是本地系统,描述最后提到了crc,说明他要确定是哪个系统才判断需不需要创建crc文件校验。
因为在linux是要的,所以在linux本地进行重复创建文件的话,要先识别是否存在该文件的crc,有则删除,这样才能覆盖。
所以最后的解决方式是:

fs.copyToLocalFile(false, hdfs, local,true);

猜你喜欢

转载自blog.csdn.net/weixin_41487978/article/details/105277328