Python ModuleNotFoundError, importing between modules in project

Question

This has to be simple, but I swear I've been searching and experimenting for 2 days with no end of errors. The best I've managed to achieve is either the IDE indicates no errors, but fails when running. Or indicates errors, but runs successfully. I'm on Windows 10. Python 3.8.3 conda. I have VS Code, Spyder, and PyCharm--all of which I've uninstalled and reinstalled multiple times hoping one of them would give me useful default behavior.

I think I have a minimally reproducible anomaly that hopefully serves to identify issues, which I'll explain below. As a preface, I'll say that I wish that I could find a complete and direct guide to identifying, declaring, repairing, etc packages and modules. Every source I look at has a different set of suggestions, it seems like. And I'm lost between PYTHONPATH, sys.path, .env workspace files, .vscode json files.

c:\myproject
    __init__.py
    file1.py
    \myenums
        __init__.py
        category.py
        functions.py
        general.py

That's leaving out a lot of files and folders. And I won't belabor a long list of variations of errors I've had. Hopefully this pair of examples is telling.

I have a function in \\myproject\\file1.py and another function in \\myproject\\myenums\\functions.py. Each of those .py files has import references to other .py files. Pretty much all those import statements are red-underlined in VS Code. But depending on how I qualify the import statments--how high up the folder hierarchy I go--one or the other will run but not both. To wit (reduced and simplified indication of actual code):

\myproject\file1.py
    import myenums.category  # red underline for error
    import myenums.functions # red underline for error
    import myenums.general   # red underline for error
    import myenums.results   # red underline for error

    print(myenums.category)  # works
    print(myenums.functions) # works
    print(myenums.general)   # works
    print(myenums.results)   # works

\myproject\myenums\functions.py
    from myenums.category import A, B, C  # red underline, note mutual parent folder
    from myenums.general import X, Y, Z   # red underline, note mutual parent folder
         
    def testfunc(value):
        if value is A.x:
            return X.a
        elif value is A.y:
            return Y.a

    print(testfunc(A.y))    # doesn't work. ModuleNotFoundError: No module named 'myenums'.

I get the working/not working results by Run Without Debugging in either file.

\myproject\file1.py
    import myenums.category  # red underline for error
    import myenums.functions # red underline for error
    import myenums.general   # red underline for error
    import myenums.results   # red underline for error

    print(myenums.category)  # doesn't work, ModuleNotFoundError: No module named 'category' (on import functions)
    print(myenums.functions) 
    print(myenums.general)   
    print(myenums.results)   

\myproject\myenums\functions.py
    from category import A, B, C  # red underline, note parent folder qualification removed
    from general import X, Y, Z   # red underline, note parent folder qualification removed
         
    def testfunc(value):
        if value is A.x:
            return X.a
        elif value is A.y:
            return Y.a

    print(testfunc(A.y))    # Now works!

So depending on whether I overqualify with the containing folder name in functions.py, I can get functions.py or file1.py to run--but never both. And I can never make error indicators go away on the imports.

I can't figure out at all how to diagnose this problem. Could I have too many or the wrong init .py files? Could they be bad files? They look empty in Notepad. And are not .py.bin. When I run print(p) in sys.path, I do see c:\\myproject --but it's a lower case 'c' where all the Anaconda references have upper case 'C'. Hmmm. It's been a long time since I've felt this lost and helpless trying to write some code. I wouldn't have thought that creating references between the files in a single folder would be such a challenge. I have no idea what to try.

Edit: Originally I thought my problem was specific to references between subfolders next to each other in hierarchy, and I thought I would try to fix it by rearranging code to only ever reference to direct subfolders--but this anomaly seems to shoot even that down. Also, I originally posted saying that my anomaly went away. Now I think it's still there, I just had a typo.

Answer 1

Here is my suggestion. Organize your file structure like this for testing purposes:

myproject\
    tests.py
    myproject\
        __init__.py
        file1.py
        myenums\
            __init__.py
            category.py
            functions.py
            general.py

What I did was create a topmost folder for testing purposes that doesn't include a __init__.py file, ie, it is not part of the main module myproject . You can do all of your testing from here.

The general rule of thumb is that, when creating a Python package, you should not run the scripts directly from inside the package.

To illustrate this, let's write a function inside of myproject/myproject/file1.py :

# myproject/myproject/file1.py

def file1_function():
    print("Inside file1.py!")

Now, we have to import it in the topmost __init__.py so that we can access it from outside the function.

# myproject/myproject/__init__.py

from myproject import file1

Established earlier, we are going to run our tests from tests.py , which is outside of the package. In tests.py , we can say:

# myproject/tests.py

import myproject
myproject.file1.file1_function()

Which outputs:

Inside file1.py!

Of course, you could always replace from myproject import file1 with from myproject.file1 import file1_function , which allows us to call file1_function a little more easily:

# myproject/tests.py

import myproject
myproject.file1_function()

Which gives us the same output as before.

What happens when we need to import files/functions/classes/etc. to use elsewhere in the package? Well, we can use absolute imports.

For demonstration purposes, let's say there is an important piece of code in functions.py that file1.py needs to use.

# myproject/myproject/myenums/functions.py

def really_important_function():
    print("Inside myproject/myenums/functions.py!")

Now we have to do the same thing that we did above. The only difference is that we are trying to access it from inside the package. Let's move into the __init__.py that is located in myenums :

# myproject/myproject/myenums/__init__.py

from myproject.myenums.functions import really_important_function

Here we use an absolute import , which is an import that stems from the root of the package ( myproject is the root).

To use this in file1.py , we then have to import it into file1.py :

# myproject/file1.py

from myenums import really_important_function

def use_important_function():
    really_important_function()

Now that we are able to access really_important_function from myproject/myenums/functions.py in myproject/file1.py , we can import it in tests.py the same way we did in the above example:

# myproject/tests.py

import myproject

myproject.file1.use_important_function()

Which outputs:

Inside myproject/myenums/functions.py!

I think it should be worth noting that you don't necessarily need to traverse the package upwards, updating every __init__.py just to test a function that won't be imported directly anyway. If you want to test really_important_function from tests.py , you can do this:

# myproject/tests.py

import myproject

myproject.myenums.functions.really_important_function()

Which gives the correct output.

Conclusion

Another way to look at this is as follows. Let's say you define " function " in functions.py and you want to use it in a function defined in file1.py :

myproject\
    tests.py -> import myproject
             -> myproject.use_function()
    myproject\
        __init__.py -> from myproject.file1 import use_function
        file1.py -> from myproject.myenums import function
                 -> def use_function():
                        function()
        myenums\
            __init__.py -> from myproject.myenums.functions import function
            category.py
            functions.py -> def function():
                                ...
            general.py