python3 cannot find a module that I could import with python2

Question

I'm trying to switch from python2 to python3. In the process I'm also switching from anaconda to miniconda3 as my primary package management tool. There are some other packages that I clone from github. I found that I can no longer import any modules from packages downloaded from github rather than conda. For example, import linetools.utils used to work and no longer works. import linetools.linetools.utils works but this is not enough, since various modules inside the package reference each other. This package is not written by me so changing all occurrences of linetools.utils to linetools.linetools.utils is not desirable.

Here are additional information for you to help solve my problem.

My $PYTHONPATH environment variable is set to /Users/lwymarie/python/ . I also tried putting this same path to my $path variable as well. Didn't solve my problem.

Here are the packages I have under PYTHONPATH. Just showing a few.

flemish.local> ls $PYTHONPATH
sdsspy          Barak           desisim         linetools           desiutil        pymc3           specutils           PypeIt          pyqtgraph       statsmodels
RemoteObserving     fitsio          Ska.Numpy       ginga           redrock

Here is the expected behavior, when I used python2 and anaconda2. I'm using the package linetools as an example.

flemish.local> python
Python 2.7.12 |Anaconda custom (x86_64)| (default, Jul  2 2016, 17:43:17) 
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import linetools.utils

Here's the sys.path when the above worked. Just showing a few of the elements.

>>> import sys
>>> sys.path
['', '/Users/lwymarie/python', '/Users/lwymarie/anaconda/lib/python27.zip', '/Users/lwymarie/anaconda/lib/python2.7', '/Users/lwymarie/anaconda/lib/python2.7/plat-darwin', '/Users/lwymarie/anaconda/lib/python2.7/plat-mac', '/Users/lwymarie/anaconda/lib/python2.7/plat-mac/lib-scriptpackages', '/Users/lwymarie/anaconda/lib/python2.7/lib-tk', '/Users/lwymarie/anaconda/lib/python2.7/lib-old', '/Users/lwymarie/anaconda/lib/python2.7/lib-dynload', '/Users/lwymarie/anaconda/lib/python2.7/site-packages', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/PIL', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/Sphinx-1.3.1-py2.7.egg', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/aeosa', '/Users/lwymarie/python/specutils', '/Users/lwymarie/python/ginga', '/Users/lwymarie/python/statsmodels', '/Users/lwymarie/python/Ska.Numpy', '/Users/lwymarie/python/linetools', '/Users/lwymarie/python/pymc3', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/numpydoc-0.6.0-py2.7.egg', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/nbsphinx-0.2.9-py2.7.egg', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/recommonmark-0.4.0-py2.7.egg', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/CommonMark-0.5.4-py2.7.egg', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/joblib-0.10.2-py2.7.egg', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/Theano-0.8.2-py2.7.egg', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/ginga-2.7.0-py2.7.egg', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/QtPy-1.3.1-py2.7.egg', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/desiutil-1.9.9.dev596-py2.7.egg', '/Users/lwymarie/anaconda/lib/python2.7/site-packages/desisim-0.24.0.dev1308-py2.7.egg', '/Users/lwymarie/python/redrock/py', '/Users/lwymarie/python/PypeIt', '/Users/lwymarie/anaconda/lib/python2.7/site-packages']

Exiting Python, here's the $path variable when the above worked.

flemish.local> echo $path
/Users/lwymarie/anaconda/bin . /usr/local/bin /usr/local/etc /opt/local/bin /opt/local/sbin /usr/sbin /sbin /usr/bin /bin /usr/lang /etc /usr/etc /usr/X11/bin /usr/local/scisoft/bin /usr/local/texlive/2020/bin/x86_64-darwin /Library/Ruby/Gems/2.0.0 ./py/ ./

Here is the unwanted behavior, when I used python3 and miniconda3.

flemish.local> python
Python 3.7.4 (default, Aug 13 2019, 15:17:50) 
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import linetools.utils
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'linetools.utils'

Here's the sys.path when the above error happened. I also tried the sys.path.append("/Users/lwymarie/python/linetools/") trick and it still couldn't find the module.

>>> import sys
>>> sys.path
['', '/Users/lwymarie/python', '/Users/lwymarie/miniconda3/lib/python37.zip', '/Users/lwymarie/miniconda3/lib/python3.7', '/Users/lwymarie/miniconda3/lib/python3.7/lib-dynload', '/Users/lwymarie/miniconda3/lib/python3.7/site-packages']

Exiting Python, here's the $path variable when the above error happened.

flemish.local> echo $path
/Users/lwymarie/miniconda3/bin /Users/lwymarie/miniconda3/bin . /usr/local/bin /usr/local/etc /opt/local/bin /opt/local/sbin /usr/sbin /sbin /usr/bin /bin /usr/lang /etc /usr/etc /usr/X11/bin /usr/local/scisoft/bin /usr/local/texlive/2020/bin/x86_64-darwin /Library/Ruby/Gems/2.0.0 ./py/ ./

Here's an image of the directory tree of the linetools package.

If you successfully help me get python3 and linetools working, your good deed will be rewarded with eternal life and happiness.

Answer 1

There is '/Users/lwymarie/python/linetools' in your python2 sys.path but not in your python3 sys.path. Did you re-setup linetools using python3?

cd ~/python/linetools
python setup.py develop

Answer 2

I think what Steven Lau suggested should work.

But keep in mind that you have multiple index paths. You have /Users/lwymarie/python and /Users/lwymarie/python/linetools (I see this for your py2 setup).