Python将制表符分隔的双语txt拆分为两个单独的txt文件（列表），并使用换行符分隔字符串

Question

I have a bi-lingual corpora (EN-JP) from tatoeba and want to split this into two separate files. The strings have to say on the same line respectively.

I need this for training an NMT in nmt-keras and training data has to be stored in separate files for each language. I tried several approaches, but since I'm an absolute beginner with python and coding in general I feel like I'm running in circles.

So far the best I managed was the following:

Source txt:

Go. 行け。
Go. 行きなさい。
Hi. やっほー。
Hi. こんにちは！

Code:

with open('jpns.txt', encoding="utf8") as f:
    columns = zip(*(l.split("\t") for l in f))

list1= list(columns)
print(list1)

[('Go.', 'Go.', 'Hi.', 'Hi.'), ('行け。\n', '行きなさい。\n', 'やっほー。\n', 'こんにちは！')]

Result with my code:

[('Go.', 'Go.', 'Hi.', 'Hi.'), ('行け。\n', '行きなさい。\n', 'やっほー。\n', 'こんにちは！')]

English and Japanese get properly separated (into a Tuple?) but I'm stuck at figuring out how to export only English and how to export only Japanese to an output.en and an output.jp respecitvely.

Expected result:

output.en

Go.
Go.
Hi.
Hi.

output.jp

行け。
行きなさい。
やっほー。
こんにちは！

Each outputted strings should contain an \\n after the string.

Please keep in mind that I'm a total beginner with coding, so I'm not exactly sure what I did after "zip" as I just found this here on stackoverflow. I'd be really gratful for a fully commented suggestion.

Answer 1

The first thing to be aware of is that iterating over a file retains the newlines. That means that in your two columns, the first has no newlines, while the second has newlines already appended to each line (except possibly the last).

Writing the second column is therefore trivial if you've already unpacked the generator columns :

with open('output.jp', 'w') as f:
    f.writelines(list1[-1])

But you still have to append newlines to the first column (and possibly others if you go full-on multilingual). One way would be to append newlines to all the columns but the last. Another would be to strip the columns from the last column and process all of them the same.

You can achieve the result you want with a small loop, and another call to zip :

langs = ('en', 'jp')
for index, (lang, data) in enumerate(zip(langs, columns)):
    with open('output.' + lang, 'w') as f:
        if index < len(langs) - 1:
            data = (line + '\n' for line in data)
        f.writelines(data)

This approach replaces the tuple data with a generator that appends newlines, unless we are at the last column.

There are a couple of ways to insert newlines between each line in the output files. The one I show uses a lazy generator to append to each line individually. This should save a little memory. If you don't care about memory savings, you can output the whole file as a single string:

joiner = '\n' if index < len(langs) - 1 else ''
f.write(joiner.join(data))

You can even write the loop yourself and print to the file:

for line in data:
    print(line, file=f, end='\n' if index < len(args) - 1 else '')

Addendum

Let's also look at the line columns = zip(*(l.split("\\t") for l in f)) in detail, since it is a very common Python idiom for transposing nested lists, and is the key to getting the result you want.

The generator expression l.split("\\t") for l in f is pretty straightforward: it splits each line in the file around tabs, giving you two elements, one in English, and one in Japanese. Adding a * in front of the generator expands it so that each two-element row becomes a separate argument to zip . zip then re-combines the respective elements of each row, so you get a column of the English elements, and a column of the Japanese elements, effectively transposing your original "matrix".

The result is that columns is a generator over the columns. You can convert it to a list , but that is only necessary for viewing. The generator will work fine for the code shown above.