Python throws an error that file doesn't exist when it clearly does

Question

I'm currently trying to read in some text files as Pandas DataFrames, and the program keeps throwing an error that the file doesn't exist.

The code that I've written is:

import os

import pandas as pd


cwd = os.getcwd()

path1 = os.path.join(cwd, 'sub1/sub2/sub3')
path2 = os.path.join(cwd, 'sub1/sub2/sub3')

files = []
for f in list(os.listdir(path1)):
    files.append(pd.read_csv(f))

When I run this code I get a:

FileNotFoundError: [Errno 2] File b'text_file.txt' does not exist: b'text_file.txt'

which is quite confusing because when I run os.listdir(path1) I get a name of all the files with text_file.txt included. Also, I noticed that when I actually move into the directory where the text file is located, the program runs with no problem.

Is there something that I'm fundamentally doing wrong? Thank you.

Answer 1

You have to join the path, as the filename alone will assume that it's relative to your cwd.

files = []
for f in list(os.listdir(path1)):
    fullpath = os.path.join(path1, f)
    files.append(pd.read_csv(fullpath))

Having said that I would recommended using pathlib as it makes file path handling a bit easier. (and pandas supports accepting a Path object)

from pathlib import Path

import pandas as pd

path1 = Path(__file__).parent / 'sub1' / 'sub2' / 'sub3'
files = []
for f in path.glob('*'):
    files.append(pd.read_csv(f))

Answer 2

Well, it is quite easy:

1) You can use the full path, something like /Users/mikhailgenkin/code/text_file.txt . In this case, it does not matter from which location you are trying to read a file (as long as you run python script from your local machine).

2) You can use a relative path, text_file.txt . In this case, you have to run the code (or run your python source file) from the same path. There are other possibilities, for example, if your input file is one level up relative to your source file, you would use ../text_file.txt .