Copy files recursively from source to target based on wildcard using Python

Question

I have a source directory that has several xml files in nested location. I am trying to write a python script to copy files recursively based on a pattern (say *1.2.3.xml) to a target location.

source
├── master.1.2.3.xml
    └── directory
           └── fileA.1.2.3.xml
           ├── fileA.1.2.5.xml
           ├── fileB.1.2.3.xml

Expected outcome:

target
├── master.1.2.3.xml
    └── directory
           └── fileA.1.2.3.xml
           ├── fileB.1.2.3.xml

The following script doesn't do the filtering.

from shutil import copytree
def ignored_files(adir,filenames):
    return [filename for filename in filenames if not filename.endswith('1.2.3.xml')]
copytree(source, target, ignore=ignored_files)

What am I missing here?

Answer 1

What is happening here is that the copytree function will work recursively. First, it will descend into source and give ignored_files() two items for filenames arg - [master.1.2.3.xml, directory]

The ignored_files will return [directory] as it does not match with the pattern and thus copytree will ignore the entire directory itself.

You will have to add an additional check for directories in your condition in ignored_files() something like os.path.isdir() .

Answer 2

Can you please try this?

import glob
import shutil
dest_dir = "path/to/dir"
for file in glob.glob(r'/path/to/dir/*1.2.3.xml'):
    print(file)
    shutil.copy(file, dest_dir)

Answer 3

Try this instead:

def copy_tree_with_wildcards(src: str, dest: str, pattern: str):
src = os.path.abspath(src)
for filename in glob.iglob(src + '/**/' + pattern, recursive=True):
    src_file = os.path.join(src, filename)
    dest_file = dest + filename[len(src):]
    dest_dir = os.path.dirname(dest_file)
    os.makedirs(dest_dir, exist_ok=True)
    if os.path.isfile(src_file):
        shutil.copy(src_file, dest_file)