I am writing a Python script where some of the core functionalities can be done by another existing library. Unfortunately, while that library has more features, it is also slower, so I'd like if the user could at runtime select whether they want to use that library or my own fast and simple implementation. Unfortunately I'm stuck at a point where I don't understand some of the workings of Python's module system.
Suppose that my main program was main.py
, that the (optional) external module is in module_a.py
and that my own fast and simple implementation of module_a
together with the actual program code that uses either my own implementation or the one of module_a
is in the file module_x.py
:
main.py:
import module_x
module_x.test(True)
module_x.test(False)
module_a.py:
class myclass():
def __init__(self):
print("i'm myclass in module_a")
module_x.py:
class myclass():
def __init__(self):
print("i'm myclass in module_x")
def test(enable_a):
if enable_a:
try:
from module_a import myclass
except ImportError:
global myclass
enable_a = False
else:
global myclass
i = myclass()
When I now execute main.py
I get:
$ python3 main.py
i'm myclass in module_a
i'm myclass in module_a
But why is this? If False
is passed to test()
then the import of the module_a
implementation should never happen. Instead it should only see myclass
from the local file. Why doesn't it? How do I make test()
use the local definition of myclass
conditionally?
My solution is supposed to run in Python3 but I see the same effect when I use Python2.7.
An import
statement is permanent within the thread of execution unless it is explicitly undone. Furthermore, once the from ... import
statement is executed in this case, it replaces the variable myclass
in the global scope (at which point the class it was previously referencing defined in the same file is no longer referenced and can in theory be garbage collected)
So what is happening here is whenever you run test(True)
the first time, your myclass
in module_x
is effectively deleted and replaced with the myclass
from module_a
. All subsequent calls to test(False)
then call global myclass
which is effectively a no-op since the global myclass now refers to the one imported from the other class (and besides the global
call is unneeded when not changing the global variable from a local scope as explained here ).
To work around this, I would strongly suggest encapsulating the desired module-switching behavior in a class that is independent of either module you would like to switch. You can then charge that class with holding a reference to both modules and providing the rest of you client code with the correct one. Eg
module_a_wrapper.py
import module_x
import module_a
class ModuleAWrapper(object):
_target_module = module_x # the default
@classmethod
def get_module(cls):
return cls._target_module
def set_module(enable_a):
if enable_a:
ModuleAWrapper._target_module = module_a
else:
ModuleAWrapper._target_module = module_x
def get_module():
return ModuleAWrapper.get_module()
main.py:
from module_a_wrapper import set_module, get_module
set_module(True)
get_module().myclass()
set_module(False)
get_module().myclass()
Running:
python main.py
# Outputs:
i'm myclass in module_a
i'm myclass in module_x
You can read more about the guts of the python import system here
The answer by lemonhead properly explains why this effect happens and gives a valid solution.
The general rule seems to be: wherever and however you import a module, it will always replace any variables of the same name from the global scope.
Funnily, when I use the import foo as bar
construct, then there must neither be a global variable named foo
nor one named bar
!
So while lemonhead's solution worked it adds lots of complexity and will lead to my code being much longer because every time I want to get something from either module I have to prefix that call with the getter function.
This solution allows me to solve the problem with a minimal amount of changed code:
module_x.py:
class myclass_fast():
def __init__(self):
print("i'm myclass in module_x")
def test(enable_a):
if enable_a:
try:
from module_a import myclass
except ImportError:
enable_a = False
myclass = myclass_fast
else:
myclass = myclass_fast
i = myclass()
So the only thing I changed was to rename the class I had in global scope from myclass
to myclass_fast
. This way it will not be overwritten anymore by the import of myclass
from module_a
. Then, on demand, I change the local variable myclass
to either be the imported module or myclass_fast
.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.