如何将子目录中的文件复制到HDFS中的单个目录

Question

I have an external table in Impala that is partitioned by two columns so the HDFS directory has two level of directories before you get to the actual data files. The table has become corrupt in the meta store and cannot be queried. I want to copy only the individual (~10k) files into a single directory so I can drop the corrupt table, remove the existing directories and then run the data back into the table with the LOAD DATA INTO table query in Impala. The problem is I cannot find a way to copy just the files so they all end up in a single directory since the LOAD DATA doesn't support subdirectory loading.

The structure looks like:

• myroot
  • mysub1a
    • mysub2a
      • file1.txt
      • file2.txt

There are hundreds of directories at the mysub1 and mysub2 levels

I have been able to get the correct list of just the files with:

hadoop fs -lsr /myroot/ | grep .parq

but I cannot figure out how to pass in the output of this list into

hadoop fs -cp {mylist} /mynewdir/

Answer 1

Wildcards should do the trick:

hdfs dfs -cp /myroot/*/*/*.parq /mynewdir

Note that if you don't need the files at the original locations then a hdfs dfs -mv will be much faster.