Python：将具有相同基名的不同目录中的文件分组

Question

I am struggling with a particularly frustrating task.

I have a set of thousands of files in one directory, say /path/to/file#####.txt . In other directories I have (probably the same number of) files with the same base-names but different suffixes, eg /diff/path/to/file#####.txt.foo .

I am trying to group these files together so that I have a list of lists as

[['/path/to/file#####.txt', '/diff/path/to/file#####.txt.foo', 
  '/another/path/to/file#####.txt.bar'], ...]

It is likely, but not guaranteed that there is a corresponding file in each subforlder, but it's not guaranteed. In other words, '/path/to/file#####.txt' may exist but '/diff/path/to/file#####.txt.foo' might not, so I need to skip that base-name if this occurs.

My purpose here is to create a file list for synchronized data loading.

How can I efficiently do this?

Answer 1

I ended coming up with a solution that's fairly efficient but it does not appear to be the most elegant.

Basically, I first find all file basenames and form a list of lists of all possible sets, eg

groups = [['/path/to/file00000.txt', '/diff/path/to/file00000.txt.foo', 
  '/another/path/to/file00000.txt.bar'],
  ['/path/to/file00001.txt', '/diff/path/to/file00001.txt.foo', 
  '/another/path/to/file00001.txt.bar'], ...]

Then I check for the existence of the file with a given basename in each directory using os.path.exists() , like

del_idx = []
for i in xrange(len(groups)):
    for j in len(groups[i]):
        if not os.path.exists(groups[i][j]):
            del_idx.append(i)
            continue # Because if one doesn't exist, no need to check others

Now that I have a list of indices that are "bad", I just loop through in reverse to delete them.

for i in xrange(len(del_idx)-1,-1,-1):
    groups.pop(del_idx[i])

This works fine in my case where I only have 3-tuples, but if there are a significant number of paths in the tuple, this would probably break down.

For ~260k files the all-groups construction took ~12 sec, the existence check took ~35 sec, and the deletion took ~12 sec. This is fairly reasonable, but, again, this algorithm is O(m*n) for m files and groups of size n, so it's not ideal if group sizes get large.

Answer 2

My proposed solution:

import glob
import os.path as op
from collections import defaultdict
def variable_part(file, base, ext):
    return file[len(base):-len(ext)-1]
def func(dirs):
    base = 'file'
    files = defaultdict(list)
    dirext = []
    for d in dirs:
        local_files = glob.glob(op.join(d, '*'))
        local_ext = '.'.join(local_files[0].split('.')[1:])
        for f in local_files:
            files[variable_part(op.basename(f), base, local_ext)].append(f)
    return list(files.values())

Haven't profiled it, my feeling however is that it's close-to-optimal, each filename is processed once, and after the first directory any access to files should have been amortised already. Some additional optimisation is definitely possible, especially in the handling of strings.

If the variable part are just integers from 0 to M-1, it may be optimal to have a series of M lists X_k of length N, if you have N directories; Each X_k[i] is set to 1 or 0 according to the existence or not of the file file k .xxx in the i-th directory. Only then you produce the final filenames list, removing the need for deletions (which, as you may have noticed, is an expensive operation for a list).

In any case, the minimum complexity for this algorithm is N*M, in no way you can get away from going in each directory and check all the files; those 35 sec may be optimized with a single system call for getting all the directory, and then working in memory, but that does not change the overall complexity, ie how the algorithm scales.

Edit I was kinda curious on this, and I made a test. Indeed, apparently working on the filenames retrieved by glob seems faster than checking each file for existence (at least on my mac HFS+ filesystem, on ssd).

In [0]: def x():
     ...:     return [os.path.exists('test1/file%06d.txt.gz' % i) for i in range(10000)]
     ...:

In [1]: def y():
     ...:     ff = glob.glob('test1/*')
     ...:     res = [False]*10000
     ...:     for s in ff:
     ...:         res[int(s[10:16])] = True
     ...:     return res
     ...:

In [2]: %timeit x()
10 loops, best of 3: 71.2 ms per loop

In [3]: %timeit y()
10 loops, best of 3: 32.6 ms per loop