为什么我会错过一些Python迭代？

Question

I have a Parts of Speech (POS) tagged parallel corpus with 25 files in the source language in directory1 and 25 files in the target language in directory2 . Each file contains 1000 lines ie 25000 lines per directory.

Task at hand: I want to remove the POS tags and then write all the text in the source and target language in a single-single text file, say, source.txt & target.txt .

Fortunately, I did this (see code below) but when I run the code - sometimes source.txt or target.txt has 24896 lines or 24871 lines etc but not 25000. After running the code for 2-3 times I get 25000 lines for both files.

Sample POS tagged input: Need\\VBN of\\IN delivery\\NN with\\IN operation\\NN .\\.

This behavior is mysterious to me (non-CS grad). Is there any explanation for this behavior or it is just like that.

Pardon me if it is a dumb question!

outfile1 = open("source.txt",'w')
outfile2 = open("target.txt",'w')

path = '/somePath/'
file_names = []; tempDataSrc = []; tempDataTrg = []

for root, dirs, files in os.walk(path):
    for file in files:
        if file.endswith('.txt'):
            file_names.append(os.path.join(root, file))

file_names = sorted(file_names)

for file in file_names:  
    if ("Src_" in file): # filtering source language files
        infile1 = open(file,'r')
        for line_s in infile1:
            line_s = " ".join(word.split("\\")[0] for word in line_s.split())
            tempDataSrc.append(line_e)

for file in file_names: 
    if ("Trg_" in file): # filtering target language files
        infile2 = open(file,'r')
        for line_t in infile2:
            line_t = " ".join(word.split("\\")[0] for word in line_t.split())
            tempDataTrg.append(line_p)

for line1 in tempDataSrc:
    outfile1.write(line1+'\n')

for line2 in tempDataTrg:
    outfile2.write(line2+'\n')

NOTE: I have conda installation with python 3.6. I am running my code in Spyder IDE; OS: Ubuntu 14.04.5

PS: Any suggestions for writing the code in more pythonic way are also encouraged

Answer 1

I guess the behavior has to do with your environment for running the program )either the IDE or your OS itself) being abruptly killing the process, and it is not finishing to write the output to the files - as you don't close our output files in the code.

You can fix that by simply calling the .close() method on "outfile1" and "outfile2" at the very end of your code.

But, as you asked for input on doing things in a more Pythonic way: since you only write to the output at the end of your script, it makes sense to only "open" then near that part of the code as well. And since we are at it, you might as well use th e with statement to create and write to both files - that will ensure all data produced is flushed to the disk saved even in the case of early termination due to other errors:

with open("source.txt",'w') as outfile1:
    for line1 in tempDataSrc:
        outfile1.write(line1+'\n')

with open("target.txt",'w') as outfile2:
    for line2 in tempDataTrg:
        outfile2.write(line2+'\n')

(The with statement will automatically close the files and flush the data).