Accessing files and modules in different directories in Python

Question

I am using Python 2.7. I have the following directory structure:

alogos
- resources
  - __init__.py
  - test.py
- lib
- __init__.py
  - utils.py
- common
  - config
    - config.json

My utils.py is the following:

def read_json_data(filename):
    with open(filename) as data:
        json_data=json.load(data)
    return json_data

My test.py has the following:

from lib.utils import read_json_data

running_data = read_json_data('common/config/config.json')
print running_data

when I try to run python test.py from the resources directory, I get the following error:

ImportError: No module named lib.utils

What is the correct way to access files and modules

Answer 1

Your lib.utils module is not present in the current directory (and apparently not anywhere else import checks), and so the import fails.

The Python doc details the module search path:

When a module named spam is imported, the interpreter first searches for a built-in module with that name. If not found, it then searches for a file named spam.py in a list of directories given by the variable sys.path. sys.path is initialized from these locations:

 * the directory containing the input script (or the current directory). * PYTHONPATH (a list of directory names, with the same syntax as the shell variable PATH). * the installation-dependent default. 

After initialization, Python programs can modify sys.path. The directory containing the script being run is placed at the beginning of the search path, ahead of the standard library path. This means that scripts in that directory will be loaded instead of modules of the same name in the library directory. This is an error unless the replacement is intended. See section Standard Modules for more information.

While this is certainly not the only way, what I would do would be to have your lib.utils module as a separate module, stored in a local Pypi server ( Artifactory is one example, but there are others, such as devpi ) where you can install it just like any other module, just from a different index URL from the regular Pypi. That way, any of your scripts can use it just like any other module, and it obviates the need to play assorted path-related games that can add unnecessary complexity.