Reading multiple CSV files from multiple files into pandas DataFrame

Question

I would like to go through different csv files contained in different folders in the same directory. My folders are in my working directory. My folders are named:

folder1, folder2,folder3

each of them have csv's with identical names csv1.csv, csv2.csv .

I tried this code:

import os
import re
import pandas as pd
from pandas.core.frame import DataFrame

rootDir = '.'
for dirName, subdirList, fileList in os.walk(rootDir, topdown=False):
    print('Found directory: %s' % dirName)
    for fname in fileList:
        print('\t%s' % fname)

        if "csv1.csv" == fname:
            var= pd.read_csv(fname)

I can print the name of the csv file in that folder but i get an error: IOError: File csv1.csv does not exist
what could be the problem?

Answer 1

As you can see in comments, you have to join rootDir , dirName and fname .

import os
import re
import pandas as pd
from pandas.core.frame import DataFrame

rootDir = '.'
for dirName, subdirList, fileList in os.walk(rootDir, topdown=False):
    print('Found directory: %s' % dirName)
    for fname in fileList:
        print('\t%s' % fname)
        filepath = os.path.join(rootDir, dirName, fname)
        if "csv1.csv" == fname:
            var = pd.read_csv(filepath)
            print var.head()

os.path.join(path, *paths) :

Join one or more path components intelligently. The return value is the concatenation of path and any members of *paths with exactly one directory separator ( os.sep ) following each non-empty part except the last, meaning that the result will only end in a separator if the last part is empty. If a component is an absolute path, all previous components are thrown away and joining continues from the absolute path component.

On Windows, the drive letter is not reset when an absolute path component (eg, r'\\foo' ) is encountered. If a component contains a drive letter, all previous components are thrown away and the drive letter is reset. Note that since there is a current directory for each drive, os.path.join("c:", "foo") represents a path relative to the current directory on drive C: ( c:foo ), not c:\\foo .