Hadoop 2.7.5 fs Command

Overview

The File System (FS) shell includes various shell-like commands that directly interact with Hadoop Distributed File System (HDFS) as well as other file systems that Hadoop supports,such as Local FS,HFTP FS,S3 FS,and others.The FS shell is invoked by:

bin/hadoop fs

All  FS  shell commands take path URIs  as arguments. The URI format is scheme://authority/path. For HDFS the scheme is hdfs,and for the Local FS the scheme is file. The scheme and authority are optional. If not specified in the configuration is used. An HDFS file or directory such as /parent/child can be specified as hdfs://namenodehost/parent/child or simply as /parent/child(given the your configuration is set to point to hdfs://namenodehost).

Most of the commands in FS schell behave like corresponding Unix commands. Differences are described with each of the commands. Error information is sent to stderr and the output is sent to stdout. 

If HDFS is being used,hdfs dfs is a synonym.

appendToFile

Usage:hadoop fs -appendToFile ...

Append single src,or multiple srcs from local file system to the destination file system. Also reads input from stdin and appends to destination file system.

hadoop fs -appendToFile localfile /user/hadoop/hadoopfile

hadoop fs -appendToFile localfile1 localfile2 /user/hadoop/hadoopfile

hadoop fs -appendToFile localfile hdfs://nn.example.com/hadoop/hadoopfile

hadoop fs -appendToFile - hdfs://nn.example.com/hadoop/hadoopfile Reads the input from stdin

Exit  code:

Return  0 on success and -1 on error.

cat

Usage:hadoop fs -cat [-ignoreCrc] URI [URI ...]

Copies source paths to stdout.

Options:

The -ignoreCrc option disables checksum verification.

Example:

hadoop fs -cat hdfs://nn1.example.com/file1 hdfs://nn2.example.com/file2

hadoop fs -cat file:///file3 /user/hadoop/file4

Exit Code:

Returns 0 on success and -1 on error.

checksum

Usage:hadoop fs -checksum URI

Returns the checksum information of a file.

Example:

hadoop fs -checksum hdfs://nn1.example.com/file1

hadoop fs -checksum file:///ext/hosts

chgrp

Usage:hadoop fs -chgrp [-R] GROUP URI [URI ...]

Change group association of files. The user must be the owner of files, or else a super-user. Additional information is in the Permission Guide.

Options:

The -R option will make the chage recursively through the directory structure.

chomd

Usage:hadoop fs -chmod [-R] URI [URI...]

Change the permissions of ifles. With -R make the change recursively though the directory structure. The user must be the owner of the file,or else a ssuper-user. Additional information is in the Permission Guide.

Options:

The -R option will make the change recursively through the directory structure.

copyFromLocal

Usage:hadoop fs -copyFromLocal URI

Similar to he fs -put command,except the source is retricted to a local file reference.

Options:

-p: Preserves access and modification times,ownership and the permissions.(assuming the permissions can be propagated across filesystems)

-f: Overwrites the destination if it already exists.

-l: Allow DataNode to lazily persist the file to disk, Forces a rplication factor of 1. This flag will result in reduced durability. Use with care.

-d: Skip creation of temporary file with the suffix ._COPYING_.

copyToLocal

Usage:hadoop fs -copyToLocal [-ignorecrc] [-crc] URI

Similar to get command,except that the destination is restricted to a local file reference.

count

Usage: hadoop fs -count [-q] [-h] [-v]

Count the number of directories,files and bytes under the paths that match the specified file pattern. The output columns with -count are:DIR_COUNT,FILE_COUNT,CONTENT_SIZE,PATHNAME.

The output columns with -count -q are:QUOTA,REMAINING_QUOTA,REMAINING_SPACE_QUOTA,DIR_COUNT,FILE_COUNT,CONTENT_SIZE,PATHNAME.

The -h option shows sizes in human readable format.

The -v option displays a header line.

Example:

hadoop fs -count hdfs://nn1.example.com/file1 hdfs://nn2.example.com/file2

hadoop fs -count -q hdfs://nn1.example.com/file1

hadoop fs -count -q -h hdfs://nn1.example.com/file1

hadoop fs -count -q -h -v hdfs://nn1.example.com/file1

Exit Code:

Returns 0 on success and -1 on error.

cp

Usage:hadoop fs -cp [-f] [-p[topax]] URI [URI...]

Copy files from source to destination. This command allows multiple sources as well in which case the destination must be directory.

'raw.*' namespace extended attributes are preserved if(1) the source and destination filesystems support them(HDFS only),and(2) all source and destination pathnames are in the /.reserved/raw hierarchy. Determination of whether raw.*namespace xattrs are preserved is independent of the -p(preserve) flag.

Options:

The -f option will overwrite the destination if it is already exists.

The -p option will preserve file attributes[topx](timestamps,ownership,permission,ACL,XAttr).If -p is specified with no arg,then preserves timestamps,ownership,permission. If -pa is specified,the preserves permission also because ACL is a super-set of permission. Determination of whether raw namespace extended attributes are preserved is independent of the -p flag.

Example:

hadoop fs -cat -cp /user/hadoop/file1 /user/hadoop/file2

hadoop fs -cat -cp /user/hadoop/file1 /user/hadoop/file2 /user/hadoop/dir

Exit Code:

Returns 0 on success and -1 on error.

createSnapshot

See HDFS Snapshots Guide.

deleteSnapshot

See HDFS Snapshots Guide.

df

Usage: hadoop fs -df [-h] URI [URI...]

Displays free space.

Options:

The -h option will format file size in a "human-readable" fashion(e.g 64.0m instead of 67108864).

Example:

hadoop fs - df /user/hadoop/dir1

du

Usage:hadoop fs -du [-s]  [-h]  URI  [URI...]

Displays  sizes of files and directories contained in the given directory or the length of a file in case it is just a file.

Options:

The  -s option will result in an aggregate summary of file lengths being displayed,rather than the individual files.

The  -h option will format file sizes in a "human-readable"  fashion(e.g 64.0m instead of 67108864)

Example:

hadoop fs -du /user/hadoop/dir1 /user/hadoop/file1 hdfs://nn.example.com/user/hadoop/dir1

Exit  Code:

Returns  0 on success and -1 on error.

expunge

Usage:hadoop fs -expunge

Permanentlty  delete files in checkpoints older than the retention threshold from trash directory,and create new checkpoint.

When  checkpoint is created,recently deleted files in trash are moved under the checkpoint.  Files  in checkpoints older than fs.trash.checkpoint.interval will be permanently deleted on the next invocation of -expunge command.

If  the file system supports the feature,users can configure to create and delete checkpoints periodically by the parameter stored as fs.trash.checkpoint.interval(in core-site.xml).  This  value should be smaller or equal to fs.trash.interval.

Refer  to the HDFS  Architecture  guide for more information about trash feature of HDFS.

find

Usage:hadoop fs -find   ...    ...

Finds  all files that match the specified expression and applies selected actions to them.  If  no path is specified then defaults to the current working directory.  If  no expression is specified then defaults to -print.

The  following primary expressions are recognised:

-name pattern

-iname pattern

Evalusates  as true if the basename of the file matches the pattern using standard file system globbing.  If  -iname is used then the match is acse insenstive.

-print

-print0Alwasys

Evaluates  to true.  Causes  the current pathname to be written to standard output.  If  the -print0 expression is used then an ASCII  NULL  character is appended.

The  following operators are recognised:

expression -a expression

expression and expression

expression expression

Logical  AND  operator for joining two expressions.  Returns  true if both child expressions return true.  Implied  by the juxtaposition of two expressions and so does not need to be explicity  specified.  The  second expression will not be applied if the first fails.

Example:

hadoop fs -find /  -name test -print

Exit  Code:

Returns  0 on success and -1 on error.

get

Usage:hadoop fs -get [-ignorecrc]  [-crc]  [-p]  [-f]  

Copy  files to the local file system.  Files  that fail the CRC  check may be copied eith the -ignorecrc options.  Files  and CRCs  may be copied using the -crc option.

Example:

hadoop fs -get /user/hadoop/file localfile

hadoop fs -get hdfs://nn.example.com/user/hadoop/file localfile

Options:

-p:Preserves  access and modification times,ownership and the permissions.(assuming the permissions can be propagated across filesytems)

-f:Overwrites  the destination if it already exists.

-ignorecrc:Skip  CRC  checks on the file(s)  download.

-crc:Write  CRC  checksums for the files download.

getfacl

Usage:hadoop fs -getfacl [-R] 

Displays  the Access  Control  Lists(ACLs)  of files and directories.  If  a directory has a default ACL,then getfacl also displays the default ACL.

Options:

-R:List  the ACLs  of all files and directories recursively.

path:File  or directory to list.

Example:

hadoop fs -getfacl /file

hadoop fs -getfacl -R  /dir

Exit  Code:

Returns  0 on success and -1 on error.

getmerge

Usage:hadoop fs -getmerge [-nl]  

Takes  a source directory and a destination file as input and concatenates files in src into the destination local file.  Optionally  -nl can be set to enable adding a newline character(LF)  at the end of each file.

Example:

hadoop fs -getmerge -nl /src /opt/output.txt

hadoop fs -getmerge -nl /src/file1.txt /src/file2.txt /output.txt

Exit  Code:

Returns  0 on success and noe-zero on error.

help

Usage:hadoop fs -help

Return  usage output.

ls

Usage:hadoop fs -ls [-d]  [-h] [-R] 

Options:

-d:Directories  are listed as plain files.

-h:Format  file sizes in a human-readable fashion(e.g.  64.0m instead of 67108864).

-R:Recursively  list subdirectories encountered.

For  a file is returns stat on the file with the following format:

permissions number_of_replicas userid groupid filesize modification_date modification_time filename

For  a directory it returns list of its direct children as in Unix.  A directory is listed as:

permissions userid groupid modification_date modification_time dirname

Files within a directory are order by file name by default.

Example:

hadoop fs -ls /user/hadoop/file1

Exit Code:

Returns 0 on success and -1 on error.

mkdir

Usage:hadoop fs -mkdir [-p]

Takes path uri is as argument and creates directories.

Options:

The -p option behavior is much like Unix mkdir -p,creating parent directories along the path.

Example:

hadoop fs -mkdir /user/hadoop/dir1 /user/hadoop/dir2

hadoop fs -mkdir hdfs://nn1.example.com/hadoop/dir hdfs://nn1.example.com/user/hadoop/dir

Exit Code:

Return 0 on success and -1 on error.

moveFromLocal

Usage:hadoop fs -moveFromLocal

Similar  to put command,except that the source localsrc is deleted after it is copied.

mv

Usage:hadoop fs -mv URI  [URI  ...] 

Moves  files from source to destination.  This  command allows multiple sources as well in which case the destination needs to be a directory.  Moving  files across file systems is not permitted.

Example:

hadoop fs -mv /user/hadoop/file1 /user/hadoop/file2

hadoop fs -mv  hdfs://nn.example.com/file1 hdfs://nn.example.com/file2 hdfs://nn.example.com/dir1

Exit  Code:

Returns  0 on success and -1 on error.

put

Usage:hadoop fs -put [-f] [-p] [-l] [-d] [ - | ...]

Copy single src,or multiple srcs from local file system to destination file system. Also reads input from stdin and writes to destination file system if the source is set to "-".

Copying fails if the file already exists,unless the -f flag is given.

Options:

-p:Preserves access and modification times,ownership and permissions.(assuming the permissions can be propagated across filesystems)

-f:Overwrites the destination if it already exists.

-l:Allow DataNode to lazily persist the file to disk,Forces a replication factor of 1. This flag will result in reduced durability. Use with care.

-d:Skip creation of temporary file with the suffix ._COPYING_.

Example:

hadoop fs -put localfile /user/hadoop/hadoopfile

hadoop fs -put -f localfile1 localfile2 /user/hadoop/hadoopdir

hadoop fs -put -d localfile hdfs://nn.example.com/hadoop/hadoopfile

hadoop fs -put - hdfs://nn.example.com/hadoop/hadoopfile Reads the input from stdin.

Exit Code:

Returns 0 on success and -1 on error.

rm

Usage:hadoop fs -rm [-f] [-r] [-R] [-skipTrash] URI [URI ...]

Delete files specified as args.

If trash is enabled,file system instead moves the deleted file to a trash directory(tiven by FileSystem#getTrashRoot).

Currently,the trash feature is disabled by default. User can enable trash by setting a value greater then zero for parameter fs.trash.interval()in core-site.xml.

See expunge about deletion of files trash.

Options:

The -f option will not display a diagnostic message or modify the exit status to reflect an error if the file does not exist.

The -R option deletes the directory and any content under it recursively.

The -r option is equivalent to -R.

The -skipTrash option will bypass trash,if enabled,and delete the specified files immediately. This can be useful when it is necessary to delete files from an over-quota directory.

Example:

hadoop fs -rm hdfs://nn.example.com/file /user/hadoop/emptydir

Exit Code:

Returns 0 on success and -1 on error.

rmdir

Usage:hadoop fs -rmdir [--ignore-fail-on-non-empty] URI [URI ...]

Delete  a directory.

Options:

--ignore-fail-on-non-empty:When  usring wildcards,do not fail if a directory still contais files.

Example:

hadoop fs -rmdir /user/hadoop/emptydir

setfacl

Usage:hadoop fs -setfacl [-R]  [-b | -k -m | -x ]

Sets Access Control Lists(ACLs) of files and directories.

Options:

-b:Remove all but the base ACL entries. The entries for user,group and others are retained for compatibility with permission bits.

-k:Remove the default ACL.

-R:Apply operations to all files and directories recursively.

-m:Modify ACL New entries. Other ACL entries are retained.

--set:Fully replace the ACL,discarding all existing entries. The acl_spec must include entries for user,group,and others for compatibility with permission bits.

acl_spec:Comma Separated list of ACL entries.

path:File or directory to modify.

Example:

hadoop fs -setfacl -m user:hadoop:rw- /file

hadoop fs -setfacl -x user:hadoop /file

hadoop fs -setfacl -b /file

hadoop fs -setfacl -k /dir

hadoop fs -setfacl --set user:rw-,user:hadoop:rw-,group::r--,other::r-- /file

hadoop fs -setfacl -R -m user:hadoop:r-x /dir

hadoop fs -setfacl -m default:user:hadoop:r-x /dir

Exit Code:

Return 0 on success and non-zero on error.

setfattr

Usage:hadoop fs -setfattr -n name [-v value] -x name

Sets and extended attribute name and value for a file or directory.

Options:

-n name:The extended attribute name.

-v vlaue:The extended attribute value. There are three different encoding methods for the value. If the argument is enclosed in double quotes,the the value is the string inside the quotes. If the argument is prefixed with 0x or 0X,then it taken as a hexadecimal number. If the argument begins with 0s or 0S,then it taken a base64 encoding.

-x name:Remove the extended attribute.

path:The file or diretory.

Example:

hadoop fs -setfattr -n user.myAttr -v myValue /file

hadoop fs -setfattr -n user.noValue /file

hadoop fs -setfattr -x user.myAttr /file

Exit Code:

Returns 0 on success and -1 on error.

setrep

Usage:hadoop fs -setrep [-R] [-w]

Changes the replication factor of a file. If path is a directory then the command recursively changes the replication factor of all files under the directory tree rooted at path.

Options:

The -w flag requests that command wait for the replication to complete. This can potentially take a very long time.

The -R flag is accepted for backwards compatibility. It has no effect.

Example:

hadoop fs -setrep -w 3 /user/hadoop/dir1

Exit Code:

Returns 0 on success and -1 on error.

stat

Usage:hadoop fs -stat [format] ...

Print statistics ablout file/directory at in the specified format. Format accepts filesize in blocks(%b),type(%F),group name of owner(%g),name(%n),block size(%o),replication(%r),user name of owner(%u),and modification date(%y,%Y). %y show UTC date as "yyyy-MM-dd HH:mm:ss" and %Y shows milliseconds since January 1,1970 UTC. If the format is not specified,%y is used by default.

Exmaple:

hadoop fs - stat "%F %u:%g %b %y %n" /file

Exit Code:

Returns 0 on success and -1 on error.

tail

Usage:hadoop fs -tail [-f] URI

Displays last kilobyte of the file to stdout.

Options:

The -f option will output appended data as the file grows,as in Unix.

Example:

hadoop fs -tail pathname

Exit Code:

Returns 0 on success and -1 on error.

test

Usage:hadoop fs -test -[defsz] URI

Options:

-d:if the path is a directory,return 0.

-e:if the path exists,return 0.

-f:if the path is a file,return 0.

-s:if the path is not empty,return 0.

-z:if the path is zero length,return 0.

Example:

hadoop fs -test -e filename

text

Usage:hadoop fs -text

Takes  a source file and outputs the file in the text format.  The  allowed formats are zip and TextRecordInputStream.

touchz

Usage:hadoop fs -touchz URI  [URI  ...]

Create  a file of zero length.  An  error is returned if the file exists with non-zero length.

Example:

hadoop fs -touchz pathname

Exit  Code:

Returns  0  on success and -1 on error.

truncate

Usage:hadoop fs -truncate [-w] 

Truncate  all files that match the specified file pattern to the specifiled length.

Options:

Thw  -w flag requests that the command waits for block recovery to complete, if necessary.  Without  -w flag the file may remain unclosed for some time while the recovery is in progress.  During  this time file can not be reopened for append.

Example:

hadoop fs -truncate 55 /user/hadoop/file1 /user/hadoop/file2

hadoop fs -truncate -w 127 hdfs://nn1.example.com/user/hadoop/file1

usage

Usage:hadoop fs -usage command

Return  the help for an individual command.

最后编辑于
©著作权归作者所有,转载或内容合作请联系作者
  • 序言:七十年代末,一起剥皮案震惊了整个滨河市,随后出现的几起案子,更是在滨河造成了极大的恐慌,老刑警刘岩,带你破解...
    沈念sama阅读 211,817评论 6 492
  • 序言:滨河连续发生了三起死亡事件,死亡现场离奇诡异,居然都是意外死亡,警方通过查阅死者的电脑和手机,发现死者居然都...
    沈念sama阅读 90,329评论 3 385
  • 文/潘晓璐 我一进店门,熙熙楼的掌柜王于贵愁眉苦脸地迎上来,“玉大人,你说我怎么就摊上这事。” “怎么了?”我有些...
    开封第一讲书人阅读 157,354评论 0 348
  • 文/不坏的土叔 我叫张陵,是天一观的道长。 经常有香客问我,道长,这世上最难降的妖魔是什么? 我笑而不...
    开封第一讲书人阅读 56,498评论 1 284
  • 正文 为了忘掉前任,我火速办了婚礼,结果婚礼上,老公的妹妹穿的比我还像新娘。我一直安慰自己,他们只是感情好,可当我...
    茶点故事阅读 65,600评论 6 386
  • 文/花漫 我一把揭开白布。 她就那样静静地躺着,像睡着了一般。 火红的嫁衣衬着肌肤如雪。 梳的纹丝不乱的头发上,一...
    开封第一讲书人阅读 49,829评论 1 290
  • 那天,我揣着相机与录音,去河边找鬼。 笑死,一个胖子当着我的面吹牛,可吹牛的内容都是我干的。 我是一名探鬼主播,决...
    沈念sama阅读 38,979评论 3 408
  • 文/苍兰香墨 我猛地睁开眼,长吁一口气:“原来是场噩梦啊……” “哼!你这毒妇竟也来了?” 一声冷哼从身侧响起,我...
    开封第一讲书人阅读 37,722评论 0 266
  • 序言:老挝万荣一对情侣失踪,失踪者是张志新(化名)和其女友刘颖,没想到半个月后,有当地人在树林里发现了一具尸体,经...
    沈念sama阅读 44,189评论 1 303
  • 正文 独居荒郊野岭守林人离奇死亡,尸身上长有42处带血的脓包…… 初始之章·张勋 以下内容为张勋视角 年9月15日...
    茶点故事阅读 36,519评论 2 327
  • 正文 我和宋清朗相恋三年,在试婚纱的时候发现自己被绿了。 大学时的朋友给我发了我未婚夫和他白月光在一起吃饭的照片。...
    茶点故事阅读 38,654评论 1 340
  • 序言:一个原本活蹦乱跳的男人离奇死亡,死状恐怖,灵堂内的尸体忽然破棺而出,到底是诈尸还是另有隐情,我是刑警宁泽,带...
    沈念sama阅读 34,329评论 4 330
  • 正文 年R本政府宣布,位于F岛的核电站,受9级特大地震影响,放射性物质发生泄漏。R本人自食恶果不足惜,却给世界环境...
    茶点故事阅读 39,940评论 3 313
  • 文/蒙蒙 一、第九天 我趴在偏房一处隐蔽的房顶上张望。 院中可真热闹,春花似锦、人声如沸。这庄子的主人今日做“春日...
    开封第一讲书人阅读 30,762评论 0 21
  • 文/苍兰香墨 我抬头看了看天上的太阳。三九已至,却和暖如春,着一层夹袄步出监牢的瞬间,已是汗流浃背。 一阵脚步声响...
    开封第一讲书人阅读 31,993评论 1 266
  • 我被黑心中介骗来泰国打工, 没想到刚下飞机就差点儿被人妖公主榨干…… 1. 我叫王不留,地道东北人。 一个月前我还...
    沈念sama阅读 46,382评论 2 360
  • 正文 我出身青楼,却偏偏与公主长得像,于是被迫代替她去往敌国和亲。 传闻我的和亲对象是个残疾皇子,可洞房花烛夜当晚...
    茶点故事阅读 43,543评论 2 349

推荐阅读更多精彩内容