urllib3中的相对导入

Question

I'm studying how python loads modules.

I'm starting from requests that loads urllib3 .

I have found those line :

from .packages.six.moves.http_client import (
  IncompleteRead as httplib_IncompleteRead
)

in file called src/urllib3/exceptions.py

That import is a relative one, so I'm looking for the folder src/urllib3/packages/six/moves/http_client.py or src/urllib3/packages/six/moves/http_client/__init__.py

Those files don't exist.

Thankfully src/urllib3/packages/six.py defines a new module loader, so the import .packages.six.moves.http_client requires http_client core python module instead.

But I don't understand how the src/urllib3/packages/six.py is loaded. I've not found that kind of import.

Is there a way to know which module loads src/urllib3/packages/six.py directly?

I have tried with __name__ global variable as suggested here . Its value is urllib3.packages.six but urllib3.packages doesn't load that file.

Answer 1

The from .packages.six.moves.http_client import ... expression causes .packages.six to be loaded first . Python always loads all packages in a nested package reference to a module.

So .packages.six.moves.http_client causes Python first to look for urllib3.packages , then for urllib3.packages.six , and so on. The import machinery does so by looking for the full name in sys.modules , and if it is not there, triggers a module search and load for each.

The first time this happens, sys.modules['urllib3.packages.six'] doesn't exist yet, the import machinery finds the file urllib3/packages/six.py , imports that, before it'll look for more names.

And, as you discovered, the very act of importing the six.py module file, causes that module to add sys.modules['urllib3.packages.six.moves'] and further references to standard library modules.

Python's import machinery is quite a complex beast; the Python reference documentation covers this comprehensively in The import system ; the specific entries to look for are:

A direct call to __import__() performs only the module search and, if found, the module creation operation. While certain side-effects may occur, such as the importing of parent packages , and the updating of various caches (including sys.modules), only the import statement performs a name binding operation.

and under Regular packages

Importing parent.one will implicitly execute parent/__init__.py and parent/one/__init__.py. Subsequent imports of parent.two or parent.three will execute parent/two/__init__.py and parent/three/__init__.py respectively.

and under The module cache :

The first place checked during import search is sys.modules . This mapping serves as a cache of all modules that have been previously imported, including the intermediate paths . So if foo.bar.baz was previously imported, sys.modules will contain entries for foo , foo.bar , and foo.bar.baz . Each key will have as its value the corresponding module object.

(bold emphasis in quoted sections added by me).

Note that everything in the urllib3/packages directory is a vendorized package; a project that normally would be installed independently, but which the urllib3 project has decided to package up with their own distribution to avoid having to worry about what versions to support. six is such an independent project, you can install it from PyPI .

You can find more information on the six.moves virtual package in the six project documentation . It's purpose is to make it easier for library developers to write code that is compatible both with Python 2 and Python 3, without having to worry about what standard library name to import on either version.

Answer 2

One notable thing about import function in Python is that it can import not only modules themselves but also variables, classes, functions (generally namespaces) from inside them. In the example, you have provided:

from .packages.six.moves.http_client import (
  IncompleteRead as httplib_IncompleteRead
)

import statement refers to moves variable from src/urllib3/packages/six.py module, which is defined at line 316 of that file and being assigned to an instance of class _MovedItems :

moves = _MovedItems(__name__ + ".moves")

and http_client is a property or method of this class instance