如何加入Pig输出文件？

Question

The pig script output a few part files (part-m-00000, part-m-00001, etc) with .pig_header and .pig_schema and I am trying to join them as one output csv. I tried to use the hadoop merge

hadoop fs -getmerge ./output output.csv

but the files are merged with the .pig_schema file as well so it becomes something like

header1,header2,header3
{"fields":[{"name": "header1", "type":...}]}
value1,value2,value3

How do I join them correctly without the .pig_schema included?

Thanks!

Answer 1

使用fileglob： hadoop fs -getmerge ./output/part* output.csv