如何遍历文件并按文件名中的字符串标识文件

Question

In Python: I'm trying to loop through files in a directory, find the files that have a certain string in their file name, open and edit those files. It all seems to be working except being able to select specific files in the directory based on the string:

import re
import datetime as dt
OldValue = input('Enter the value to be replaced: ')
NewValue = input('Enter the replacement value: ')
location = input('Enter path to directory: ')
directory = os.listdir(location)
os.chdir(location)
for root, dirs,files in os.walk('.'):
    for fname in files:
        re.match('PMPM', fname)

for f in os.listdir(location):
    for file in directory:
                open_file = open(file, 'r')
                read_file = open_file.read()
                regex = re.compile(OldValue)
                read_file = regex.sub(NewValue, read_file)
                write_file = open(file, 'w')
                write_file.write(read_file)
                now = dt.datetime.now()
                ago = now-dt.timedelta(minutes=30)
for root, dirs,files in os.walk('.'):
    for fname in files:
        path = os.path.join(root, fname)
st = os.stat(path)
mtime = dt.datetime.fromtimestamp(st.st_mtime)
if mtime > ago:
    print('%s modified %s' % (path,  mtime))

Answer 1

If all you want is a list of the filenames from a given directory containing a given substring then something like this should work:

#!python
import os
dir='.'         # Replace with path to your directory: absolute or relative
pattern = 'foo' # Replace with your target substring
matching_files = [f for f in os.listdir(dir) if pattern in f]

That's all you need for the simplest case. You can then iterate over the list of matching_files

If you want to walk down a directory tree with os.walk() then you have to search the third item from each tuple returned by the generator.

os.walk() recurses down a tree returning a tuple for each subdirectory. Each of these consists of three items: the leading path, the list of subdirectories below that, and the list of filenames (directory entries for anything OTHER than a subdirectory) at that node.

However, there's also a trick! You'll need to prefix each match with that dirpath item at that level. In other words for every match in tuple(os.walk(...))[2] (the list) you need to yield the concatenation of that with the corresponding string from tuple(os.walk(...))[0] to get a full (absolute or relative) path to the matching filename.

One way to get a feel for how this works is to load up your Python interpreter (preferably iPython from the Jupyter project), instantiate a generator with walker = os.walk(dir) (where dir is any valid directory to use as the starting point) and then call this = next(walker) and you can look at this[0] and this[2] before going on to look at the next(walker) .

Let's start with code which returns a list using simple substring matching (as I did with my previous example, but over multiple lines for clarity):

#!python
results = list()
dir = '.'
walker = os.walk(dir)
delimiter = os.path.sep
pattern = '.txt'
for p,_,f in walker:
  matches = ['%s%s%s' % (p, delimiter, x) for x in f if pattern in f]
  results.extend(matches)

In this case I'm using the tuple unpacking of the for loop to give me the path and file list components from each tuple yielded by the os.walk() generator. The matches, at each node in the tree, are extracted in a list comprehension which is also prefixing each match with the path (and using the os.path.sep to make the code portable across different operating system platforms).

Also notice that _ is just a variable name in Python, but it's conventionally used to "throw away" some value. In other words using _ as a variable in Python is a hint to readers and maintainers that this was some unwanted stuff that your code is not interested in using later.

It would be better to write this as a generator function and yield results rather than perform the full traversal (potentially consuming time and memory). With our own generator wrapped around os.walk() we could more easily process each match subject to other conditions (find the first, the first N, wrap in even more filtering, and so on).

Also I'm using simple substring matching (using Python's in operator, which calls the ._ _contains_ _() special method. We can use regular expressions for this ... though I commend being wary of re.match() which only matches patterns at the beginning of each string against which it's matched.

So here's that:

#!python
import os, re
def matchwalk(regex, directory):
    '''Yield path/filenames matching some regular expression
    '''
    sep = os.path.sep
    pattern = re.compile(regex)
    for p,_,f in os.walk(directory):
        for each in f:
            if pattern.search(each):
                yield '{0}{1}{2}'.format(p,sep,each)

This is similar to the previous code example. The differences: I've wrapped it in a function, I'm using yield so the function is a generator (just like os.walk() ). I'm using regular expressions; I prefer to use re.compile() for legibility (there might be some marginal performance benefit as well, but probably not under most Python implementations as the re module will often do it's on interning of regular expression just as Python does interning of many strings). Also I'm using the newer style string formatting function (though I personally prefer the old syntax; this is just for edification).

Answer 2

You might want to have a look at the standard unix style pathname pattern expansoin package, or simply glob .

---

Running through all files with a filename starting with, say 'PMPM' , in a specific directory, say '~/path/to/mydir' , is a simple as:

import os
import glob

pattern = os.path.join(
    os.path.expanduser('~'),
    'path/to/mydir',
    'PMPM*' # mind the * here!
)

for matching_file in glob.glob(pattern):
    with open(matching_file, 'r') as f:
        # do something with the file object
        pass

Or the same in short:

from glob import glob
for mf in glob('home/someuser/path/to/mydir/PMPM*'):
    with open(mf, 'r') as f:
        pass # do something with f