如何从两个文件夹中读取文件并避免在Python中重复

Question

I have the following folders that I read SQL files from and save them as variables:

++folder1
  -1.sql
  -2.sql
  -3.sql
++folder2
  -2.sql

The following code does the job well for a single folder. How I can modify this code to read not just from one folder but from two with a rule that if a file exists in folder2 than don't read the file with the same name from folder1?

folder1 = '../folder1/'
for filename in os.listdir(folder1):
    path = os.path.join(folder1, filename)
    if os.path.isdir(path):
        continue
    with open(folder1 + filename, 'r') as myfile:
        data = myfile.read()
    query_name = filename.replace(".sql", "")
    exec (query_name + " = data")

Answer 1

You can try something like follows:

folders = ['../folder2/','../folder1/']
checked =[]
for folder in folders:
    for filename in os.listdir(folder):
        if filename not in checked:
            checked.append(filename)
            path = os.path.join(folder, filename)
            if os.path.isdir(path):
                continue
            with open(folder + filename, 'r') as myfile:
                data = myfile.read()
            query_name = filename.replace(".sql", "")
            exec (query_name + " = data")

Answer 2

The answer to this is simple: Do two listdir calls, then skip over the files in folder1 that are also in folder2.

One way to do this is with set operations: the set difference a - b means all elements in a that are not also in b , which is exactly what you want.

files1 = set(os.listdir(folder1))
files2 = set(os.listdir(folder2))
files1 -= files2

paths1 = [os.path.join(folder1, file) for file in files1]
paths2 = [os.path.join(folder2, file) for file in files2]
for path in paths1 + paths2:
    if os.path.isdir(path):
        # etc.

---

As a side note, dynamically creating a bunch of variables like this is almost always a very bad idea, and doing it with exec instead of globals or setattr is an even worse idea. It's usually be much better to store everything in, eg, a dict. For example:

queries = {}
for path in paths1 + paths2:
    if os.path.isdir(path):
        continue
    name = os.path.splitext(os.path.basename(path))[0]
    with open(path) as f:
        queries[name] = f.read()