Unable to read file from directory

Question

I have following Directory structure,

F:\TestData

and TestData contains 20 folders with name node1, node2,..., node20 and each node folder contains file with names log.10.X

I need to access each log file from all node folders, For which I have writen code, but it is saying, File not found - log.*

CODE:

directory = "F:\TestData"
p = subprocess.Popen(["find", "./" + directory, "-name", "log.*"], stdout=subprocess.PIPE)
output, err = p.communicate()
foutput = output.split("\n")

Answer 1

Python, unlike POSIX shells, does not automatically do globbing (interpreting * and the like as wildcards related to files in the relevant directory) in strings. It does, however, provide a glob module for that purpose. You can use this to get a list of matching filenames:

import glob

filenames = glob.glob(r'F:\TestData\node*\log.*')

Answer 2

You can just use python to get a list of files in the directory

import os
directory = "F:\TestData\"
file_list = os.listdir(directory)
log_list = filter(lambda x: x.startswith("log"), file_list)

oh, you have to code to iterate sub directory. First os.listdir() in the parent directory ,and iterate the sub directory to get the files

Answer 3

Python's glob module may be an option.

import glob
directory = 'F:\TestData'
logcontents = [open(f,'r').read() for f in glob.glob(directory + '\node*\log.*')]

Answer 4

You also use walk , like this:

import os
directory = "F:\TestData"
for i in os.walk(directory):
    # i like this:
    # ('F:\\TestData', ['node1', 'node2', 'node3'], [])
    # ('F:\\TestData\\node1', [], ['log.1.txt'])
    # ('F:\\TestData\\node2', [], ['log.2.txt'])
    print i
    if i[2] != []:
        # TODO: use the path to finish other
        # If dictory noden have some log file, you should use i[2][n].
        # So, if you only need log.n.txt, you only use i[2][n].
        print os.path.join(i[0], i[2][0])