Python3 连接多个文件并忽略头部和尾部记录

Question

I want to concatenate multiple files and skip the header and trailer records in all the files and have the column names(always in the 2nd line of the file) occur only once in the final file while concatenating.

I am able to concatenate,but how do i skip the header,trailer and retain the column names only once?Each file has about 25 million records.

 File1.txt

    H,ABC,file1.txt
    Name,address,zipcode
    Rick,ABC,123
    Tom,XYZ,456
    T,2  -----------------record count

 File2.txt

    H,ABC,file2.txt
    Name,address,zipcode
    Jerry,ABC,123
    T,1


 File3.txt

    H,ABC,file3.txt
    Name,address,zipcode
    John,ABC,123
    Mike,XYZ,456
    T,2

 ***Final Output:***

    Name,address,zipcode
    Rick,ABC,123
    Tom,XYZ,456
    Jerry,ABC,123
    Harry,XYZ,456
    John,ABC,123
    Mike,XYZ,456

Code:

filenames = ['File1.txt', 'File2.txt', 'file3.txt']
with open('output_file', 'w') as outfile:
    for fname in filenames:
        with open(fname) as infile:
            for line in infile:
                outfile.write(line)

Answer 1

Using Python:

Here is a really simple method which uses pandas.read_csv to concatenate your TXT files and output to a single TXT file, using pandas.DataFrame.to_csv .

import pandas as pd
from glob import glob

df = pd.DataFrame()
files = glob('./addr_files/*.txt')

for f in files:
    df = df.append(pd.read_csv(f, skiprows=1, skipfooter=1, engine='python'))

df.to_csv('./addr_files/output.txt', index=False)

Output:

(py35) ~/Desktop/so/addr_files
$ cat output.txt
Name,address,zipcode
Rick,ABC,123
Tom,XYZ,456
Jerry,ABC,123
Harry,XYZ,456
John,ABC,123
Mike,XYZ,456

Using GNU sed :

Here is another option which will stream the output of each file named file*.txt into a new file ( all.txt ), skipping the rows you want to miss; specifically the 1st, 2nd and last.

Given your files are so large, you might want to add a couple printf statements for debugging, so you can see the which file is being processed, as the script loops the files.

#!/usr/bin/env bash

# Print the header to the output file.
sed -n 2p file1.txt > all.txt

# Stream (specific) content of all files to output file.
for f in $( ls file*.txt ); do sed '1d;2d;$d' $f >> all.txt; done

Output:

(base) user@host ~/Desktop/so/concat                                                                             
$ cat all.txt
Name,address,zipcode
Rick,ABC,123
Tom,XYZ,456
Jerry,ABC,123
Harry,XYZ,456
John,ABC,123
Mike,XYZ,456

Answer 2

1)You can modify a little what you had done in the following:

filenames = ['File1.txt', 'File2.txt', 'file3.txt']
with open('output_file', 'w') as outfile:
    outille.write("Name,address,zipcode\n")
    for fname in filenames:
        with open(fname) as infile:
            for line in infile:
                if line.find("Trailer record") < 0 and line.find("Name,address,zipcode") < 0 :
                    outfile.write(line)

2) Alternatively if you are familiar with grep command in unix you can use it. You can use it directly in Python with the sh library and chaining the commands.

Answer 3

The python way

Concatenate iterators so you don't have problems with huge files.

import os
import fnmatch
from itertools import filterfalse
import csv

def get_files(pattern, path):
    """
    Get all files from path that match pattern
    """
    for path, _, filelist in os.walk(path):
        for name in fnmatch.filter(filelist, pattern):
                yield os.path.join(path, name)

def open_files(filenames):
    """
    Open all files that match pattern
    """
    for filename in filenames:
        file = open(filename, newline="")
        yield file
        file.close()

def get_csv(files):
    """
    Return csv reader for files
    """
    for file in files:
        lines = filterfalse(lambda line: line.startswith("H,") or line.startswith("T,"), file)
        reader = csv.DictReader(lines, delimiter=",")
        yield reader

def concatenate(iterators):
    """
    Concatenate iterators into a single sequence
    """
    for it in iterators:
        yield from it

with open('output.txt', 'w', newline="") as output:
    filenames = get_files('*.txt', '.')
    files = open_files(filenames)
    csvs = get_csv(files)
    lines = concatenate(csvs)
    
    fieldnames = ["Name","address","zipcode"]
    writer = csv.DictWriter(output, fieldnames=fieldnames)
    
    writer.writeheader()
    for line in lines:
        writer.writerow(line)