Copy files to another folder using python

Question

Lets say, i have a source folder with 10k file, i want to copy 1k files to another folder. Tried the below methods, it worked but, is there any way to do it more efficiently ?

sourceFiles = os.listdir("/source/")
destination = "/destination/"

for file in sourceFiles[0 : 1000]:
    shutil.copy(file, destination)

Here what i feel is, i am loading 10k files into a list variable and iteration through each element in the list for 1k times, loading unwanted data into the RAM, which doesn't look good for me. Is there any better way to do the same ?

Answer 1

If you are using Python 3, pathlib.Path.iterdir is a better option:

from pathlib import Path

source = Path('/source')
target = Path('/destination')

counter = 0

for obj in source.iterdir():
    if obj.is_file():
        obj.rename(target / obj.name)
        counter += 1
    if counter > 1000:
        break

It uses a generator and the syntax is cleaner IMHO.

It is also better on memory efficiency. Look:

Python 3.7.5 (default, Dec 15 2019, 17:54:26) 
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from sys import getsizeof
>>> from os import listdir
>>> from pathlib import Path
>>> files = listdir('/usr/bin')
>>> usrbin = Path('/usr/bin')
>>> getsizeof(files)
26744
>>> getsizeof(usrbin.iterdir())
128
>>>