Python import good practice?

Question

For my current python projects, I currently add this lines of code at the beginning of each scripts :

import os
import sys
current = os.path.abspath(os.path.dirname(__file__))
(__,folder) = os.path.split(current)
while folder != "my_root_folder_name":
    current = os.path.abspath(os.path.join(current, os.path.pardir))
    (__, folder) = os.path.split(current)
sys.path.insert(0, current)

Here my_root_folder_name is the name of the root folder of the project. Having this in every script allows to import any of them from any other of them by writing (wherever the importing script is):

import subfolder1.subfolder2.thescript

Where subfolder1 is in the root folder my_root_folder_name (final path is /my_root_folder_name/subfolder1/subfolder2/thescript.py ).

Is that a good practice ? Do you see any disadvantage of this trick ? What better option do I have ?

EDIT: Let say my project is organised as follow:

• in the main folder, I have subfolders src , data , models , logs , config ...
• in src I have a main script 'my_project.py' and subsubfolders data , models , features .

The reason I use this trick is to make sure I can import a script of a subfolder from one another subfolder. This might lead to circular imports if you don't pay attention, but python '..' syntax for imports does as well. It also makes the imports clearer, since every import is done by writing the entire absolute path to the script.

EDIT 2: This piece of code doesn't import every other scripts !! . It just replaces the first (index 0) sys.path value of the script, which is the actual folder where the file is, by the main folder of the project.

Answer 1

Generally no, this would not be good practice for a few reasons.

1. You should only be importing the packages which are absolutely required for each file. Importing unnecessary packages can cause performance issues as well as having possibly confusing code.
2. You're adding a loop to your code, in every file, for no real required reason. This isn't good practice and depending on how many dependencies you have can slow you down.
3. You're requiring a file to be located in your root folder, and using the os paths to get to it, generally not good performance wise if you can avoid it.

Answer 2

Does the solution need to be written inside the Python code itself?

Setting PYTHONPATH=/path/to/module/root does exactly what you want. For example:

$ find /tmp/py/modules/ -type f
/tmp/py/modules/qwerty/a/b/target.py
/tmp/py/modules/asdf/a/b/source

$ pwd
/tmp/py/modules/asdf

$ python /tmp/py/modules/asdf/a/b/source
Traceback (most recent call last):
  File "/tmp/py/modules/asdf/a/b/source", line 3, in <module>
    import qwerty.a.b.target
ModuleNotFoundError: No module named 'qwerty'

$ export PYTHONPATH=/tmp/py/modules/ 

$ python /tmp/py/modules/asdf/a/b/source
Hello from target

/tmp/py/modules/qwerty/a/b/target.py

#!/usr/bin/env python

print ("Hello from target")

/tmp/py/modules/asdf/a/b/source

#!/usr/bin/env python

import qwerty.a.b.target

The solution is in fact the same as what you're doing, just outside of Python code.

Note that for Python2, you would need to create __init__.py files on each directory in the structure (the files may be empty). Python3's documentation also says they're required but the example above worked just fine without them.

The updated directory contents would be as follows:

./qwerty/__init__.py
./qwerty/a/__init__.py
./qwerty/a/b/__init__.py
./qwerty/a/b/target.py
./asdf/__init__.py
./asdf/a/__init__.py
./asdf/a/b/__init__.py
./asdf/a/b/source

(the files __init__.py are not strictly required under asdf for this simple example, but as you want to be able to import all scripts from all others, that's how you would do in your project)

As an addtional note, if you keep your Python code above, my only suggestion would be to check for reaching the root folder, lest you end up in an infinite loop in the case where the script was moved and my_root_folder_name is not a component of its new location's path. For UNIX, I'd do something like this:

if current == "/":
    raise Exception ("Module folder not found")

Answer 3

I had the same problem some time ago. In my opinion, this is a problem of the Python language that get complicated in a simple action like import statements. I done the same choise to solve the problem, update the PYTHONPATH enviroment variable via sys.path command. I do in a different way to you, i write a recursive folder explorer and put it into __init__.py in the root folder of your project. The code is the following:

import os, sys
from pathlib import Path


def recursive_explorer(path):
    sys.path.append(path)
    subfolders = [subfile for subfile in os.listdir(path) if os.path.isdir(path / subfile)]
    for subfolder in subfolders:
        recursive_explorer(path / subfolder)


recursive_explorer(Path(__file__).parent / 'your_src_folder')

I think my solution is good because is portable and you need to write once.

Answer 4

The benefit of your approach doesn't justify the cost. The benefit is that you can write

import subfolder1.subfolder2.thescript

everywhere instead of some variant of

from ..subfolder2 import thescript

The cost is that everyone reading your code must now figure out the answers to questions like

• What happens if you change the root directory name?
• Could the root directory ever be missing?
• Is there any way the loop can fail to terminate?
• Is there any problem if many modules add the same folder to the head of sys.path?
• Have I thought of all the potential problems with this code? Perhaps I should ask Stack Overflow ;-).

Once Joe Programmer is convinced the code is bug free and has no bad implications, he can copy and paste it without thinking. But is it worth adding 5 lines of not-so-obvious boilerplate in order to avoid a slightly-ugly import syntax Joe already needs to understand? No.

Choose one of these well-understood options:

• Use relative import syntax.
• Use absolute import syntax and always run your code from a script in your project root directory. Python automatically adds the main script's directory to the path.
• Use absolute import syntax and always run your code from a wrapper that sets the path. This way any directory-search tricks are kept in a single file.