如何使用python从目录模拟排序和唯一文件？

Question

I am trying to sort and unique 30 files of different sizes in one single file.
Each file contains a single line and are separated by newline. That means the file has simple text on each single line.
Here is what I tried to attempt:

lines_seen = set() # holds lines already seen
outfile = open('out.txt', "w")
for line in open('d:\\testing\\*', "r"):
    if line not in lines_seen: # not a duplicate
        outfile.write(line)
        lines_seen.add(line)
outfile.close()

The folder name is testing and it contains 30 different files, which I am trying to combine into file out.txt . The output will be the sorted and unique text, written on each line of the output file.
Well, I thought it would be easy, if I write d:\\\\testing\\\\* and it will read the files from the folder. But I got error:

Traceback (most recent call last):
  File "sort and unique.py", line 3, in <module>
    for line in open('d:\\testing\\*', "r"):
OSError: [Errno 22] Invalid argument: 'd:\\testing\\*'

I would like to know how I can get rid of this error and process my all files efficiently into one single output without any unsuccess.
Please note: RAM is 8 GB and the folder size is about 10 GB.

Answer 1

You just need to loop over all files using os.listdir . Something like this:

lines_seen = set() # holds lines already seen
outfile = open('out.txt', "w")
path = r'd:\testing'
for file in os.listdir(path): #added this line
    current_file = os.path.join(path, file)
    for line in open(current_file, "r"):
        if line not in lines_seen: # not a duplicate
            outfile.write(line)
            lines_seen.add(line)
outfile.close()