将文件从一个文件夹复制到另一个在 .txt 文件中具有匹配名称的文件夹

Question

I want to copy files from one big folder to another folder based on matching file names in a .txt file.

My list.txt file contains file names:

S001
S002
S003

and another big folder contains many files for ex. S001, S002, S003, S004, S005 .

I only want to copy the files from this big folder that matches the file names in my list.txt file.

I have tried Bash, Python - not working.

for /f %%f in list.txt do robocopy SourceFolder/ DestinationFolder/ %%f

is not working either.

My logic in Python is not working:

import os
import shutil

def main():

    destination = "DestinationFolder/copy"

    source = "SourceFolder/MyBigData"

    with open(source, "r") as lines:
        filenames_to_copy = set(line.rstrip() for line in lines)

    for filenames in os.walk(destination):
        for filename in filenames:
            if filename in filenames_to_copy:
                shutil.copy(source, destination)

Any answers in Bash, Python or R?

Thanks

Answer 1

I think the issue with your Python code is that with os.walk() your filename will be a list everytime, which will not be found in your filenames_to_copy.

I'd recommend trying with os.listdir() instead as this will return a list of the names of filenames/folders as strings - easier to compare against your filenames_to_copy.

Other note - perhaps you want to do os.listdir() (or os.walk() ) on the source instead of the destination. Currently, you're only copying files from the source to the destination if the file already exists in the destination.

Answer 2

os.walk() will return a tuple of three elements: the name of the current directory inspected, the list of folders in it, and the list of files in it. You are only interested in the latter. So your should iterate with:

for _, _, filenames in os.walk(destination):

As pointed out by JezMonkey, os.listdir() is easier to use as it will list of the files and folders in the requested directory. However, you will lose the recursive search that os.walk() enables. If all your files are in the same folder and not hidden in some folders, you'd rather use os.listdir() .

The second problem I see in you code is that you copy source when I think you want to copy os.path.join(source, filename) .

Can you publish the exact error you have with the Python script so that we can better help you.

UPDATE

You actually don't need to list all the files in the source folder. With os.path.exists you can check that the file exists and copy it if it does.

import os
import shutil

def main():
    destination = "DestinationFolder/copy"
    source = "SourceFolder/MyBigData"

    with open("list.txt", "r") as lines: # adapt the name of the file to open to your exact location.
        filenames_to_copy = set(line.rstrip() for line in lines)

    for filename in filenames_to_copy:
        source_path = os.path.join(source, filename)
        if os.path.exists(source_path):
            print("copying {} to {}".format(source_path, destination))
            shutil.copy(source_path, destination)

Answer 3

You can try with below code -

import glob

big_dir = "~\big_dir"
copy_to = "~\copy_to"
copy_ref = "~\copy_ref.txt"

big_dir_files = [os.path.basename(f) for f in glob.glob(os.path.join(big_dir, '*'))]
print 'big_dir', big_dir_files # Returns all filenames from big directory

with open(copy_ref, "r") as lines:
    filenames_to_copy = set(line.rstrip() for line in lines)
    print filenames_to_copy   # prints filename which you have in .txt file

    for file in filenames_to_copy:
        if file in big_dir_files:   # Matches filename from ref.txt with filename in big dir
            file_to_copy = os.path.join(big_dir, file)
            copy_(file_to_copy, copy_to)

def copy_(source_dir, dest_dir):
    files = glob.iglob(os.path.join(source_dir, '*'))
    for file in files:
        dest = os.path.join(dest_dir, os.path.basename(os.path.dirname(file)))
        if not os.path.exists(dir_name):
            os.mkdir(dest)
        shutil.copy2(file, dest)

Reference:

https://docs.python.org/3/library/glob.html

Answer 4

Thank you @PySaad and @Guillaume for your contributions, although my script is working now: I added:

             if os.path.exists(copy_to):
            shutil.rmtree(copy_to)
            shutil.copytree(file_to_copy, copy_to)

to the script and its working like a charm :)

Thanks a lot for your help!