本文是我学习Hadoop搭建过程中的各种笔记,内容来自于各种公开的教程,起点非常低,从Linux基础开始,直至在PC上搭建Hadoop成功,是真正的从零开始。
感谢过程中帮助我的各位认识的和不认识的老师。
40、Ssh免登陆配置:
上面集群搭建好,也已经测试成功,但是在启停集群服务时需要输入密码,如果是几十台的就要一直输入密码,密码多次输入会导致服务不能正常启动或停止
借助ssh的免登陆:ssh:安全的一个shell,用它进行免登陆,在此启停服务,操作此台机子,跨机子操作另一台服务器都不用密码了
配置ssh: ssh-keygen –t rsa
[root@hadoop01 hadoop-2.7.1]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): (按回车)
Enter passphrase (empty for no passphrase): (按回车)
Enter same passphrase again: (按回车)
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is: (按回车)
ac:b6:58:3c:95:ec:4c:42:3a:6d:16:90:dd:b2:11:0a root@hadoop01
The key's randomart image is: (按回车)
+--[ RSA 2048]----+
| E .o.o |
| ..o+ . |
| . o+ |
| +.+ . |
| o = S |
| = B |
| * o |
| + o |
| . . |
+-----------------+
rsa指加密的类型。 按回车、回车、回车,生成一种图时可以了
此时多了两个目录id_rsa和id_rsa.pub: ll ~/ .ssh/
相当于生成公钥,私钥的对id_rsa和id_rsa.pub known-hosts:记载着访问过这台主机的一些主机名】
[root@hadoop01 hadoop-2.7.1]# ll ~/.ssh/
total 12
-rw-------. 1 root root 1675 Apr 26 10:24 id_rsa
-rw-r--r--. 1 root root 395 Apr 26 10:24 id_rsa.pub
-rw-r--r--. 1 root root 1218 Apr 26 08:41 known_hosts
在hadoop02上没有任何文件的:
[root@hadoop02 local]# ll ~/.ssh/
total 0
设置免密码.ssh的三种方式:
选用第三种方式:
拷贝前,此时需要输密码 root: ssh hadoop01
[root@hadoop01 hadoop-2.7.1]# ssh hadoop01
root@hadoop01's password:
Last login: Thu Apr 26 10:37:47 2018 from hadoop01
退出去: exit
[root@hadoop01 ~]# exit
logout
Connection to hadoop01 closed.
把公钥拷贝到当前的机子上:ssh-copy-id hadoop01需要输密码
[root@hadoop01 hadoop-2.7.1]# ssh-copy-id hadoop01
root@hadoop01's password:
Now try logging into the machine, with "ssh 'hadoop01'", and check in:
.ssh/authorized_keys
to make sure we haven't added extra keys that you weren't expecting.
看有没有这个文件 ll ~/ .ssh/
[root@hadoop01 ~]# ll ~/.ssh/
total 16
-rw-------. 1 root root 395 Apr 26 10:41 authorized_keys
-rw-------. 1 root root 1675 Apr 26 10:24 id_rsa
-rw-r--r--. 1 root root 395 Apr 26 10:24 id_rsa.pub
-rw-r--r--. 1 root root 1218 Apr 26 08:41 known_hosts
刚才执行的命令,产生了authorized_keys这个文件,相当于把公钥复制的这个authorized_keys钥匙的文件里面
再次访问hadoop01,此时就不需要密码了 ssh hadoop01
[root@hadoop01 ~]# ssh hadoop01
Last login: Thu Apr 26 10:59:58 2018 from hadoop01
-bash: my: command not found 问
在hadoop01远程登录hadoop02: ssh hadoop02
此时不能过来,需要输入密码
[root@hadoop01 ~]# ssh hadoop02
root@hadoop02's password:
Last login: Thu Apr 26 09:13:22 2018 from 192.168.216.1
-bash: my: command not found
退出去:exit
[root@hadoop02 ~]# exit
logout
Connection to hadoop02 closed.
在hadoop02上查看: ll ~/ .ssh/
[root@hadoop02 local]# ll ~/.ssh/
total 0
在hadoop01执行:ssh-copy-id hadoop02
[root@hadoop01 ~]# ssh-copy-id hadoop02
root@hadoop02's password:
Now try logging into the machine, with "ssh 'hadoop02'", and check in:
.ssh/authorized_keys
to make sure we haven't added extra keys that you weren't expecting.
在hadoop02上查看:ll ~/ .ssh/
[root@hadoop02 local]# ll ~/.ssh/
total 4
-rw-------. 1 root root 395 Apr 26 11:10 authorized_keys
此时就多了authorized_keys文件
在hadoop01上,此时就不需要密码了,证明ssh做好 ssh hadoop02
[root@hadoop01 ~]# ssh hadoop02
Last login: Thu Apr 26 11:03:33 2018 from hadoop01
-bash: my: command not found
退出去exit
同样的方法:
ssh hadoop03 输密码
exit 退出
ssh-copy-id hadoop03 输密码
ssh hadoop03 此时就到hadoop03的机子上了
exit 退出
查看hadoop01、02、03的jps,在hadoop01上操作停止,此时hadoop01,02,03都在停止,并且因为设置了ssh,所以不需要输入密码 : stop-all . sh
[root@hadoop01 ~]# stop-all.sh
This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh
18/04/26 11:22:08 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Stopping namenodes on [hadoop01]
hadoop01: stopping namenode
hadoop02: stopping datanode
hadoop03: stopping datanode
hadoop01: stopping datanode
Stopping secondary namenodes [hadoop01]
hadoop01: stopping secondarynamenode
18/04/26 11:22:29 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
stopping yarn daemons
stopping resourcemanager
hadoop02: no nodemanager to stop
hadoop03: no nodemanager to stop
hadoop01: stopping nodemanager
hadoop01: nodemanager did not stop gracefully after 5 seconds: killing with kill -9
no proxyserver to stop
启动: start-all . sh
[root@hadoop01 ~]# start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
18/04/26 11:24:46 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [hadoop01]
hadoop01: starting namenode, logging to /usr/local/hadoop-2.7.1/logs/hadoop-root-namenode-hadoop01.out
hadoop02: starting datanode, logging to /usr/local/hadoop-2.7.1/logs/hadoop-root-datanode-hadoop02.out
hadoop03: starting datanode, logging to /usr/local/hadoop-2.7.1/logs/hadoop-root-datanode-hadoop03.out
hadoop01: starting datanode, logging to /usr/local/hadoop-2.7.1/logs/hadoop-root-datanode-hadoop01.out
Starting secondary namenodes [hadoop01]
hadoop01: starting secondarynamenode, logging to /usr/local/hadoop-2.7.1/logs/hadoop-root-secondarynamenode-hadoop01.out
18/04/26 11:25:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop-2.7.1/logs/yarn-root-resourcemanager-hadoop01.out
hadoop03: starting nodemanager, logging to /usr/local/hadoop-2.7.1/logs/yarn-root-nodemanager-hadoop03.out
hadoop02: starting nodemanager, logging to /usr/local/hadoop-2.7.1/logs/yarn-root-nodemanager-hadoop02.out
hadoop01: starting nodemanager, logging to /usr/local/hadoop-2.7.1/logs/yarn-root-nodemanager-hadoop01.out
查看hadoop01、02、03服务是否正常 : jps
hadoop01正常:
[root@hadoop01 ~]# jps
16608 NameNode
16737 DataNode
17154 NodeManager
17477 Jps
16872 SecondaryNameNode
17050 ResourceManager
hadoop02正常:
[root@hadoop02 local]# jps
14014 NodeManager
14143 Jps
13919 DataNode
hadoop03正常:
[root@hadoop03 local]# jps
13971 NodeManager
13876 DataNode
14100 Jps
41、hdfs的常用shell:
什么把集群已经搭建好了,下面怎么用它
shell相当于hdfs shell的一个脚本,相当于在xshell里面,用shell的方式操做文件系统。
FS Shell
调用文件系统(FS)Shell命令应使用 bin/hadoop fs<args>的形式。使所有的FS shell命令使用URI路径作为参数。URI格式是 scheme://authority/path。对HDFS文件系统,scheme是hdfs,对本地文件系统,scheme是file。其中scheme和authority参数都是可选的,如果未加指定,就会使用配置中指定的默认scheme。一个HDFS文件或目录,比如 /parent/child 可以表示成 hdfs:namenode:namenodeport/parent/child,或者/parent/child (假设你配置文件中的默认值是 namenode:namenodeport)。大多数FS Shell命令的行为和对应的Unix Shell 命令类似,不同之处会在下面介绍各命令使用详情时指出。
出错信息时会输出到stderr,其他信息输出到stdout
1、hdfs 的 shell
hdfs dfs –ls /
【/ 查看目录 ; -……,-……是怎么知道的】
hdfs dfs – 出来的是-后面可以跟的
[root@hadoop01 ~]# hdfs dfs -
Usage: hadoop fs [generic options]
[-appendToFile <localsrc> ... <dst>]
appendToFile:追加文件到某个目标地去 ;localsrc:本地文件(Linux文件系统); dst:hdfs文件系统
[-cat [-ignoreCrc] <src> ...] cat:读取文件内容
[-checksum <src> ...] checksum:检验校验核
[-chgrp [-R] GROUP PATH...]
chgrp:改变文件或目录的组;-R:递归改;GROUP:组名;PATH:路径
[-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...]
chmod:改权限,同Linux系统
[-chown [-R] [OWNER][:[GROUP]] PATH...]
chown:改拥有者,也可以加上组,同Linux
[-copyFromLocal [-f] [-p] [-l] <localsrc> ... <dst>]
copyFromlocal:把本地的文件copy到hdfs文件系统
[-copyToLocal [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
copyToLocal:把hdfs文件系统copy到本地
[-count [-q] [-h] <path> ...]
[-cp [-f] [-p | -p[topax]] <src> ... <dst>]
[-createSnapshot <snapshotDir> [<snapshotName>]]
[-deleteSnapshot <snapshotDir> <snapshotName>]
[-df [-h] [<path> ...]] 查看文件大小
[-du [-s] [-h] <path> ...]
[-expunge]
[-find <path> ... <expression> ...]
[-get [-p] [-ignoreCrc] [-crc] <src> ... <localdst>] 下载
[-getfacl [-R] <path>]
[-getfattr [-R] {-n name | -d} [-e en] <path>]
[-getmerge [-nl] <src> <localdst>]
[-help [cmd ...]]
[-ls [-d] [-h] [-R] [<path> ...]]
[-mkdir [-p] <path> ...]
[-moveFromLocal <localsrc> ... <dst>]
[-moveToLocal <src> <localdst>]
[-mv <src> ... <dst>]
[-put [-f] [-p] [-l] <localsrc> ... <dst>]
[-renameSnapshot <snapshotDir> <oldName> <newName>]
[-rm [-f] [-r|-R] [-skipTrash] <src> ...]
[-rmdir [--ignore-fail-on-non-empty] <dir> ...]
[-setfacl [-R] [{-b|-k} {-m|-x <acl_spec>} <path>]|[--set <acl_spec> <path>]]
[-setfattr {-n name [-v value] | -x name} <path>]
[-setrep [-R] [-w] <rep> <path> ...]
[-stat [format] <path> ...]
[-tail [-f] <file>]
[-test -[defsz] <path>]
[-text [-ignoreCrc] <src> ...]
[-touchz <path> ...]
[-truncate [-w] <length> <path> ...]
[-usage [cmd ...]]
通常有两种方式:
hdfs dfs -(东西)/(+路径)
hadoop fs -(东西)/(+路径)
查看目录,查看文件系统有什么:hdfs dfs –ls / 或 hadoop fs –ls /
[root@hadoop01 ~]# hdfs dfs -ls /
18/04/26 11:46:21 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 3 items
-rw-r--r-- 3 root supergroup 1366 2018-04-26 09:30 /README.txt
drwxr-xr-x - root supergroup 0 2018-04-26 09:58 /out
drwx------ - root supergroup 0 2018-04-26 09:57 /tmp
[root@hadoop01 ~]# hadoop fs -ls /
18/04/26 11:47:41 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 3 items
-rw-r--r-- 3 root supergroup 1366 2018-04-26 09:30 /README.txt
drwxr-xr-x - root supergroup 0 2018-04-26 09:58 /out
drwx------ - root supergroup 0 2018-04-26 09:57 /tmp
递归查看:hdfs dfs –lsr /
[root@hadoop01 ~]# hdfs dfs -lsr /
lsr: DEPRECATED: Please use 'ls -R' instead.
18/04/26 11:50:46 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
-rw-r--r-- 3 root supergroup 1366 2018-04-26 09:30 /README.txt
drwxr-xr-x - root supergroup 0 2018-04-26 09:58 /out
drwxr-xr-x - root supergroup 0 2018-04-26 09:58 /out/00
-rw-r--r-- 3 root supergroup 0 2018-04-26 09:58 /out/00/_SUCCESS
-rw-r--r-- 3 root supergroup 1306 2018-04-26 09:58 /out/00/part-r-00000
drwx------ - root supergroup 0 2018-04-26 09:57 /tmp
drwx------ - root supergroup 0 2018-04-26 09:57 /tmp/hadoop-yarn
drwx------ - root supergroup 0 2018-04-26 09:58 /tmp/hadoop-yarn/staging
drwxr-xr-x - root supergroup 0 2018-04-26 09:58 /tmp/hadoop-yarn/staging/history
drwxrwxrwt - root supergroup 0 2018-04-26 09:58 /tmp/hadoop-yarn/staging/history/done_intermediate
drwxrwx--- - root supergroup 0 2018-04-26 09:58 /tmp/hadoop-yarn/staging/history/done_intermediate/root
-rwxrwx--- 3 root supergroup 33646 2018-04-26 09:58 /tmp/hadoop-yarn/staging/history/done_intermediate/root/job_1524706682045_0001-1524707867807-root-word+count-1524707938102-1-1-SUCCEEDED-default-1524707894739.jhist
-rwxrwx--- 3 root supergroup 349 2018-04-26 09:58 /tmp/hadoop-yarn/staging/history/done_intermediate/root/job_1524706682045_0001.summary
-rwxrwx--- 3 root supergroup 113579 2018-04-26 09:58 /tmp/hadoop-yarn/staging/history/done_intermediate/root/job_1524706682045_0001_conf.xml
drwx------ - root supergroup 0 2018-04-26 09:57 /tmp/hadoop-yarn/staging/root
drwx------ - root supergroup 0 2018-04-26 09:58 /tmp/hadoop-yarn/staging/root/.staging
创建test目录:hdfs dfs –makdir /test (一定要跟 根/)
[root@hadoop01 ~]# hdfs dfs -mkdir /test
18/04/26 11:54:46 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
查看创建的test目录是否储存:hdfs dfs -ls
[root@hadoop01 ~]# hdfs dfs -ls /
18/04/26 11:55:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 4 items
-rw-r--r-- 3 root supergroup 1366 2018-04-26 09:30 /README.txt
drwxr-xr-x - root supergroup 0 2018-04-26 09:58 /out
drwxr-xr-x - root supergroup 0 2018-04-26 11:54 /test
drwx------ - root supergroup 0 2018-04-26 09:57 /tmp
在test目录下,递归创建文件01,02,加-p:hdfs dfs –mkdir –p /test/01/02
[root@hadoop01 ~]# hdfs dfs -mkdir -p /test/01/02
18/04/26 12:00:57 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
查看在test下面创建的01 02:
[root@hadoop01 ~]# hdfs dfs -lsr /test/
lsr: DEPRECATED: Please use 'ls -R' instead.
18/04/26 12:01:30 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
drwxr-xr-x - root supergroup 0 2018-04-26 12:00 /test/01
drwxr-xr-x - root supergroup 0 2018-04-26 12:00 /test/01/02
在test目录下,创建一个空的:-touchz
[root@hadoop01 ~]# hdfs dfs -touchz /test/te.txt
18/04/26 12:05:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
查看:
[root@hadoop01 ~]# hdfs dfs -lsr /test
lsr: DEPRECATED: Please use 'ls -R' instead.
18/04/26 12:05:57 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
drwxr-xr-x - root supergroup 0 2018-04-26 12:00 /test/01
drwxr-xr-x - root supergroup 0 2018-04-26 12:00 /test/01/02
-rw-r--r-- 3 root supergroup 0 2018-04-26 12:05 /test/te.txt
复制:-cp 将test下的te.txt复制到01:(加空格复制多个)
[root@hadoop01 ~]# hdfs dfs -cp /test/te.txt /test/01
18/04/26 12:08:56 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/04/26 12:08:57 WARN hdfs.DFSClient: DFSInputStream has been closed already
查看 复制是否到01下:
[root@hadoop01 ~]# hdfs dfs -ls /test/01
18/04/26 12:09:36 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
drwxr-xr-x - root supergroup 0 2018-04-26 12:00 /test/01/01
-rw-r--r-- 3 root supergroup 0 2018-04-26 12:08 /test/01/te.txt
查看 test是否还有te.txt:
[root@hadoop01 ~]# hdfs dfs -lsr /test
lsr: DEPRECATED: Please use 'ls -R' instead.
18/04/26 12:14:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
drwxr-xr-x - root supergroup 0 2018-04-26 12:08 /test/01
drwxr-xr-x - root supergroup 0 2018-04-26 12:00 /test/01/01
-rw-r--r-- 3 root supergroup 0 2018-04-26 12:08 /test/01/te.txt
-rw-r--r-- 3 root supergroup 0 2018-04-26 12:05 /test/te.txt
移动:将test下01下的te.txt,移动到test下01下的02下面
[root@hadoop01 ~]# hdfs dfs -mv /test/01/te.txt /test/01/02
18/04/26 15:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
查看:
[root@hadoop01 ~]# hdfs dfs -lsr /test
lsr: DEPRECATED: Please use 'ls -R' instead.
18/04/26 15:01:17 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
drwxr-xr-x - root supergroup 0 2018-04-26 14:34 /test/01
-rw-r--r-- 3 root supergroup 0 2018-04-26 12:08 /test/01/02
-rw-r--r-- 3 root supergroup 0 2018-04-26 14:34 /test/01/02/te.txt
-rw-r--r-- 3 root supergroup 0 2018-04-26 12:05 /test/te.txt
移动并且改名字:将test下01下的02下的te.txt,移动到test下01,改名字为te
[root@hadoop01 ~]# hdfs dfs -mv /test/01/02/te.txt /test/01/te
18/04/26 15:36:43 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[root@hadoop01 ~]# hdfs dfs -lsr /test
lsr: DEPRECATED: Please use 'ls -R' instead.
18/04/26 15:37:18 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
drwxr-xr-x - root supergroup 0 2018-04-26 14:34 /test/01
-rw-r--r-- 3 root supergroup 0 2018-04-26 12:08 /test/01/02
-rw-r--r-- 3 root supergroup 0 2018-04-26 14:34 /test/01/te
-rw-r--r-- 3 root supergroup 0 2018-04-26 12:05 /test/te.txt
上传:-put 将当前目录的if.sh 上传到/test根目录下的test:
[root@hadoop01 ~]# cd /home/
[root@hadoop01 home]# cd ./shell/
[root@hadoop01 shell]# ll
total 60
……………………
-rwxr--r--. 1 root root 267 Apr 24 15:01 if.sh
……………………
[root@hadoop01 shell]# hdfs dfs -put ./if.sh /test
18/04/26 15:42:01 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[root@hadoop01 shell]# hdfs dfs -ls /test
18/04/26 15:42:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 3 items
drwxr-xr-x - root supergroup 0 2018-04-26 14:34 /test/01
-rw-r--r-- 3 root supergroup 267 2018-04-26 15:42 /test/if.sh
-rw-r--r-- 3 root supergroup 0 2018-04-26 12:05 /test/te.txt
移动上传:-copyFromLocal 移动当前目录下的for.sh 到 hdfs文件系统的test目录下:
[root@hadoop01 shell]# hdfs dfs -copyFromLocal ./for.sh /test
18/04/26 15:44:44 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[root@hadoop01 shell]# hdfs dfs -ls /test
18/04/26 15:44:56 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 4 items
drwxr-xr-x - root supergroup 0 2018-04-26 14:34 /test/01
-rw-r--r-- 3 root supergroup 71 2018-04-26 15:44 /test/for.sh
-rw-r--r-- 3 root supergroup 267 2018-04-26 15:42 /test/if.sh
-rw-r--r-- 3 root supergroup 0 2018-04-26 12:05 /test/te.txt
下载:-get 将hdfs文件系统的test下的if.sh,下载到home下,改名为并if:
[root@hadoop01 shell]# hdfs dfs -get /test/if.sh /home/if
18/04/26 15:46:41 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/04/26 15:46:42 WARN hdfs.DFSClient: DFSInputStream has been closed already
[root@hadoop01 shell]# ll /home/
total 387576
………………
-rw-r--r--. 1 root root 267 Apr 26 15:46 if
………………
下载:-copyToLocal 将hdfs文件系统的test下的for.sh,下载到home目录下:
[root@hadoop01 shell]# hdfs dfs -copyToLocal /test/for.sh /home/
18/04/26 15:49:17 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/04/26 15:49:18 WARN hdfs.DFSClient: DFSInputStream has been closed already
[root@hadoop01 shell]# ll /home/
total 387580
………………
-rw-r--r--. 1 root root 71 Apr 26 15:49 for.sh
………………
查看文件(有多个)查看hdfs正真的文件内容: -text、-tail、-cat
[root@hadoop01 shell]# hdfs dfs -text /test/if.sh
18/04/26 15:51:54 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[root@hadoop01 shell]# hdfs dfs -tail /test/if.sh
18/04/26 15:53:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[root@hadoop01 shell]# hdfs dfs -cat /test/if.sh
18/04/26 15:55:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
都是下面这个结果:
#!/bin/bash
a=10
b=20
c=15
if [ $a -gt $b ]
then
echo "$a > $b"
fi
if [ $a -lt $b ]
then
echo "$a < $b"
else
echo "$a is not sammer $b"
fi
if [ $a -lt $b ]
then
echo "$a < $b"
elif [ $a -gt $b -a $b -lt $c ]
then
echo "$a > $b and $b < $c"
else
echo "$a < $b"
fi
查看每个目录的大小:-du
[root@hadoop01 shell]# hdfs dfs -du /test
18/04/26 15:58:36 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
0 /test/01
71 /test/for.sh
267 /test/if.sh
0 /test/te.txt
查看每个目录大小的总和:-du -s
[root@hadoop01 shell]# hdfs dfs -du -s /test
18/04/26 15:59:16 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
338 /test
删除:hdfs dfs –rmr [-R] /
删除前:
[root@hadoop01 shell]# hdfs dfs -ls /
18/04/27 09:23:10 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 4 items
-rw-r--r-- 3 root supergroup 1366 2018-04-26 09:30 /README.txt
drwxr-xr-x - root supergroup 0 2018-04-26 09:58 /out
drwxr-xr-x - root supergroup 0 2018-04-26 15:44 /test
drwx------ - root supergroup 0 2018-04-26 09:57 /tmp
删除:test
[root@hadoop01 shell]# hdfs dfs -rm -R /test
18/04/27 09:23:52 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/04/27 09:23:53 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
Deleted /test
删除后查看:
[root@hadoop01 shell]# hdfs dfs -ls /
18/04/27 09:24:05 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 3 items
-rw-r--r-- 3 root supergroup 1366 2018-04-26 09:30 /README.txt
drwxr-xr-x - root supergroup 0 2018-04-26 09:58 /out
drwx------ - root supergroup 0 2018-04-26 09:57 /tmp