按字母顺序将文件从本地复制到HDFS-排序

Question

I need to copy files from local file system to HDFS through shell script. Suppose I have two files in my local system

fewInfo.tsv.gz
fewInfo.txt

In the above case, fewInfo.tsv.gz should be copied first(s comes before x) to HDFS and then fewInfo.txt should be copied. Is this possible?

Anyone aware of the internal structure as to how the "put" command works when multiple files are being copied to HDFS?

Hadoop version I am using is Hadoop 2.5.0-cdh5.3.1.

Answer 1

You could loop through the directory in order to find all files, sort the files and then execute the hdfs copy. The advantage would be that you can specify the constraints for the sort (eg by filename, date, order, etc.). There are many options to perform this. One would be to use the find command:

find /some/directory -type f -maxdepth 1 -type f | sort | while IFS= read -r filename; do hdfs dfs -copyFromLocal "$filename" hdfs://target/dir/; done

• -maxdepth 1 argument prevents find from recursively descending into any subdirectories. (If you want such nested directories to get processed, you can omit this.)
• -type -f specifies that only plain files will be processed.
• sort defines that the found files will be sorted. Here you have the possibility to extend by reverse order, sort for modification date, etc.
• while IFS= read -r filename loops thgough the found files. IFS in that loop is to preserve leading and trailing white space. The -r option prevents read from treating backslash as a special character.
• hdfs dfs -copyFromLocal "$filename" hdfs://target/dir/ takes the sorted filenames and copies them from the local directory to the hdfs directory. Alternatively you can also use hadoop -fs put "$filename" hdfs://target/dir/