Python: importing module from subdirectory that loads non-python file with relative path

Question

This is my first python project for which I am trying to build professional structure/hierarchy. I have the following directory structure:

project/  
---__init__.py  
---main.py  
---data_lib/  
------__init__.py  
------load.py  
------file.csv 

The code in load.py is:

import pandas as pd
def load_csv():
    print pd.read_csv('file.csv')
if __name__=='__main__':
    load_csv()

The code in main.py is:

from data_lib.load import load_csv
load_csv()

When load.py is run on its own I get a printout of a pandas dataframe of the loaded csv file. When main.py is run I get the error IOError: File file.csv does not exist. I can solve this by adding a path prefix before 'file.csv' defined by:

if __name__=='__main__':
    path_prefix=''
else:
    path_prefix='data_lib/'

Is this the standard solution and programming practice or should I be avoiding this and doing something else instead (especially since the csv file will be one that will need to be regularly updated without any other code files impacted)?

Answer 1

A common idiom in situations like this is to prefix all your file paths with the absolute path to your module:

from os import path

path_prefix = path.dirname(path.abspath(__file__))
csv_path = path.join(path_prefix, 'file.csv')

This way it doesn't matter where you are calling the module from, you will still be able to load resources from within the module.

Adding a path prefix as you have described will only work when you are calling the module from the parent directory. What happens when you call it from somewhere else?