python 如何从不在 sys.path 中的路径加载模块

Question

I was debugging a Python import issue where I can't load jinja2 properly while running my code using a pex file.

Setup a breakpoint in my code, and found myself in a very interesting situation

(Pdb) import jinja2
(Pdb) import jinja2.utils
(Pdb) jinja2.utils
*** AttributeError: 'module' object has no attribute 'utils'
(Pdb) sys.modules.get("jinja2")
<module 'jinja2' from '/home/xxx/.pex/install/Jinja2-2.10.3-py2.py3-none-any.whl.02f611acfe90225e024bd496a640e01844bf5a32/Jinja2-2.10.3-py2.py3-none-any.whl/jinja2/__init__.py'>
(Pdb) sys.modules.get("jinja2.utils")
<module 'jinja2.utils' from '/usr/local/lib/python2.7/dist-packages/jinja2/utils.pyc'>
(Pdb) [i for i in sys.path if "usr/local/lib/python2.7" in i]
[]

Before the breakpoint, jinja2 was already partially loaded because I set up the breakpoint inside jinja2's __init__.py , here is the stacktrace:

-> exec code in run_globals
  /home/xxx/.pex/code/e7fe7e585c258eb7224ec5030b9d53b74cdfb06f/application/app.py(2)<module>()
-> from flask import Flask, abort, request
  /home/xxx/.pex/install/Flask-1.0-py2.py3-none-any.whl.abe8dd5f49600e54576b3360b7622e1cc97e86fc/Flask-1.0-py2.py3-none-any.whl/flask/__init__.py(19)<module>()
-> from jinja2 import Markup, escape
> /home/xxx/.pex/install/Jinja2-2.10.3-py2.py3-none-any.whl.02f611acfe90225e024bd496a640e01844bf5a32/Jinja2-2.10.3-py2.py3-none-any.whl/jinja2/__init__.py(58)<module>()
-> print("aaaaaaaaa")

Questions are:

1. How does python even load the jinja2.utils from /usr/local/lib/python2.7/dist-packages when it is not even not part of the sys.path?
2. Why python found jinja2's __init__.py path correctly but failed to find the utils.py which is under the same directory?

~$ ls /home/xxx/.pex/install/Jinja2-2.10.3-py2.py3-none-any.whl.02f611acfe90225e024bd496a640e01844bf5a32/Jinja2-2.10.3-py2.py3-none-any.whl/jinja2/
asyncfilters.py  bccache.py  compiler.py   debug.py     environment.py  ext.py      _identifier.py  __init__.py  loaders.py  nativetypes.py  optimizer.py  runtime.py  tests.py  visitor.py
asyncsupport.py  _compat.py  constants.py  defaults.py  exceptions.py   filters.py  idtracking.py   lexer.py     meta.py     nodes.py        parser.py     sandbox.py  utils.py

Answer 1

Confirmed it is a pex issue.

Pex loaded jinja2 in the beginning (which also included jinja2.utils ), then it removed the jinaj2 module and the path but failed to remove jinja2.utils .

The issue had been fixed with latest version of pex: https://github.com/pantsbuild/pex/commit/0f9bf622e1565dcdd27c3b79b788571c9608c272