hadoop 文件系统打开文件并跳过第一行

Question

I'm reading the file in my HDFS using Python language.

Each file has a header and I'm trying to merge the files. However, the header in each file also gets merged.

Is there a way to skip the header from second file?

hadoop = sc._jvm.org.apache.hadoop
conf = hadoop.conf.Configuration()
fs = hadoop.fs.FileSystem.get(conf)

src_dir = "/mnt/test/"
out_stream = fs.create(hadoop.fs.Path(dst_file), overwrite)

files = []
for f in fs.listStatus(hadoop.fs.Path(src_dir)):
  if f.isFile():
    files.append(f.getPath())

for file in files:
  in_stream = fs.open(file)
  hadoop.io.IOUtils.copyBytes(in_stream, out_stream, conf, False)

Currently I have solved the problem with below logic, however would like to know if there is any better and efficient solution? appreciate your help

for idx,file in enumerate(files):
            if debug: 
                print("Appending file {} into {}".format(file, dst_file))

            # remove header from the second file
            if idx>0:
              file_str = ""
              with open('/'+str(file).replace(':',''),'r+') as f:
                for idx,line in enumerate(f):
                  if idx>0:
                    file_str = file_str + line

              with open('/'+str(file).replace(':',''), "w+") as f:
                f.write(file_str)
            in_stream = fs.open(file)   # InputStream object and copy the stream
            try:
                hadoop.io.IOUtils.copyBytes(in_stream, out_stream, conf, False)     # False means don't close out_stream
            finally:
                in_stream.close()

Answer 1

What you are doing now is appending repeatedly to a string. This is a fairly slow process. Why not write directly to the output file as you are reading?

for file_idx, file in enumerate(files):
  with open(...) as out_f, open(...) as in_f:
    for line_num, line in enumerate(in_f):
      if file_idx == 0 or line_num > 0:
        f_out.write(line)

If you can load the file all at once, you can also skip the first line by using readline followed by readlines :

for file_idx, file in enumerate(files):
  with open(...) as out_f, open(...) as in_f:
    if file_idx != 0:
      f_in.readline()
    f_out.writelines(f_in.readlines())