How do I import an external module when a local module masks a standard library module imported by the external module?

Question

I have a local module named tokenize.py , masks a standard library module of the same name. I only discovered this when I tried to import an external module (sklearn.linear_model), which in turn does import tokenize and expects to get the standard library module, but gets my local module instead.

This is related to How to access a standard-library module in Python when there is a local module with the same name? , but the setting is different, because applying the above solution would require modifying the external module.

An option would be to rename the local tokenize.py , but I would prefer not to do so as "tokenize" best expresses the module's role.

To illustrate the problem, here is a sketch of the module structure:

\my_module
      \__init__.py
      \tokenize.py
      \use_tokenize.py

In use_tokenize.py, there is the following import:

import sklearn.linear_model

Which results in the following error when invoking python my_module/use_tokenize.py :

Traceback (most recent call last):
  File "use_tokenize.py", line 1, in <module>
    import sklearn.linear_model
  <...>
  File "<EDITED>/lib/python2.7/site-packages/sklearn/externals/joblib/format_stack.py", line 35, in <module>
    generate_tokens = tokenize.tokenize
AttributeError: 'module' object has no attribute 'tokenize'

Is there any way to suppress local modules when importing an external module?

edit: Added python2.7 as a tag due to comments that the solution varies by Python version

Answer 1

The problem is not so much the module name, but that you're running a module like it were a script . When Python runs a script, it adds the script's containing directory as the first element in sys.path , so all module lookups from anywhere will search that directory first.

To avoid this, ask Python to execute it as a module instead:

python -m my_module.use_tokenize

Or, of course, you could just keep executable scripts out of your module hierarchy.

Answer 2

The paths that the interpreter searches for modules in are listed in sys.path . To prevent the third-party module from seeing the local module at import, we remove . from the path. This can be achieved by:

import sys
sys.path = sys.path[1:]
import sklearn.linear_model #using the original example.

However, this will not work if the local tokenize has already been imported, and it will also prevent the local tokenize from being imported, even if we restore the old sys.path as follows:

import sys
old_path = sys.path
sys.path = sys.path[1:]
import sklearn.linear_model #using the original example.
sys.path = old_path

This is because the Python interpreter maintains an internal mapping of imported modules, so that later requests for the same module are fulfilled from this mapping. This mapping is global to the interpreter, so import tokenize returns the same module from any code that runs it - which is exactly the behavior we are trying to alter. To achieve this, we have to alter this mapping. The easiest way to do this is to simply delete the relevant entry from sys.modules .

import sys
old_path = sys.path
sys.path = sys.path[1:]
import sklearn.linear_model #using the original example.
sys.path = old_path
del sys.modules['tokenize'] #get of the mapping to the standard library tokenize
import tokenize #cause our local tokenize to be imported