Problem Dynamically Importing Modules in Python 3

Question

I have situation where, in my Python 3 project, at runtime certain modules have to be included. I'm using importlib.import_module for this.

---

SECOND UPDATE: I did find a way to do something close to what I wanted. Some additional code may have rendered some of my links here off by a little bit. I will post an answer to show what I did.

---

---

FIRST UPDATE: Via various Python groups, I've been told this is not possible. I have an update section at the bottom.

---

The Setup

The idea is that when the user runs my script, any files called "agents_*.py" (where the * is replaced by some word) are searched for classes and those are imported. So for example, a file called agents_human.py will have a class called HumanAgent . It's that HumanAgent class that will be instantiated which is why I need to import the module (file) so this can happen.

The Basic Problem

The issue is two-fold:

1. I can't seem to get this module to look in multiple directories.
2. When my program is installed, the import module doesn't work at all.

Problem Context

I have a code base that shows what I'm trying to do: https://github.com/jeffnyman/pacumen

The problem is in the load_agent() method; specifically this statement .

To show this, if you clone that repo you can run the following command from the project root:

python3 -m pacumen

• That will work if the agents_human.py file is in the root of the project.
• It will not work if the agents_human.py file is an agents directory within the project.

In my code repo, the agents_human.py file is in the agents directory and thus the import fails. But if that file was moved to the root, you would see that the import works. Yet, as can be seen in the load_agent() function, I have added the current working directory plus the "agents" directory to the path ( lines 119 - 121 )

When I say the import fails, I mean literally that: when the logic to import is called, the ImportError exception is triggered. Again, this is only when the agents_human.py file is in the agents directory.

Problem To Be Solved

Ideally I would like this to work for both approaches: when the file is in the root and when in the agents directory.

Other Problem Context

I was just going to live with having to put the files in the root. But then, even worse, when the program is installed (as opposed to run from the project root), even the part that works above no longer works.

Specifically, from the project root of the cloned project, I do this:

pip3 install .

Then I create some directory (say, test_pacumen ). In there I put an agents_human.py file with the only contents being this:

class HumanAgent:
  pass

(Also required is the layouts directory from my project.)

Then I run:

pacumen

In this context, the logic will never import the agents_human.py file ... even if it's in the root of where the command is being executed from.

What Have I Tried

My understanding from looking at various examples is that I have to import via relative names or providing an explicit anchor. But that seems to be via providing a path that the modules are searched within. The code that I linked to above seems to be doing that.

I get the module names that are found on the path (via this line ) and, in all cases, it does find the relevant files that I put in place. So it seems the pathing is working; it's the importing that's not. And that's where I'm not finding a lot of guidance.

I have also tried just getting the end part of the path for each file path this way:

module_end_dir = os.path.basename(os.path.normpath(module_dir))

Then, with that in place, I have tried this:

agent_module = importlib.import_module(f"{module_end_dir}.{module_name[:-3]}")

That works locally for when agents_human.py is in the agents directory but then does not work if the agents_human.py is in the root. And it still doesn't work at all when the program is installed.

Summary of Problems

1. In the first problem context (the non-installed context), I don't know why the agents_human.py module can't be imported from agents but it can be imported from the root.

2. In the second context (installed), it seems like the agents_human.py module can't be imported even if it is in the root.

I'm hoping what I provided here isn't too confusing.

UPDATE:

I was given the idea to try something like this for my load_agent function:

def load_agent(pacman, not_human):
  module = importlib.import_module("agents.agents_human")
  return getattr(module, pacman)

That does work when the program is run via its project root but doesn't work when the project is installed via pip. Apparently this has to do with entry points to the scripts and how Python can't be made to recognize directories as modules in that context.

Another example was provided that I figured out as such:

def load_agent(pacman, not_human):
  import re
  pysearchre = re.compile('.py$', re.IGNORECASE)
  agent_files = filter(pysearchre.search, os.listdir(os.path.join(os.getcwd(), 'agents')))
  form_module = lambda fp: '.' + os.path.splitext(fp)[0]
  agents = map(form_module, agent_files)

  importlib.import_module('agents')

  for agent in agents:
      if not agent.startswith('__'):
          agent_module = importlib.import_module(agent, package="agents")

  if pacman in dir(agent_module):
      return getattr(agent_module, pacman)

  raise Exception("The agent " + pacman + " is not specified in any agents_*.py file.")

Same situation. That works with the program when run without installing it; it does not when the program is installed.

This was basically meant to serve as my confirmation that what I want to do isn't possible when a Python program is installed via pip. I'm not entirely convinced but I include this update just in case others come across this.

Answer 1

What you're doing won't work because of your entry points in setup.py but also because Python has no concept of your agents directory when you install the program.

You could try adding the agents path to the sys.path:

sys.path.insert(0, os.getcwd() + '/agents')

But I doubt that would work for when someone installs your program. (And you don't need that for when the program is run from the project as you've found.) An installation via pip basically divorces the executing script from the directory context when you run it directly from the project itself.

Another thing you might try is just importing via the file location. In other words, don't treat your agents directory (from an installation) as an actual module. Rather, just treat it as what it is: a directory location. The problem is that this is another area where Python manages to screw things up between versions.

If you're using 3.5 and up, you can do something like this:

import importlib.util
spec = importlib.util.spec_from_file_location("agents.agent_human", "/agents/agents_human.py")
foo = importlib.util.module_from_spec(spec)
spec.loader.exec_module(foo)

If you're using Python 3.3 or 3.4 you can do this:

from importlib.machinery import SourceFileLoader
foo = SourceFileLoader("agents.agent_human", "/agents/agents_human.py").load_module()

But note that even in Python 3.4 this has been deprecated.

You might want to try some variations on these. You'll probably have to add some logic to check through any files in the agents directory, on the assumption that you want to try to find whatever the class is among all files in that directory.

Answer 2

I found a solution that basically works but I had to revise my requirements a bit. Originally I wanted the case to be where someone could install my pacumen project and have certain agent files found either in a project root directory or in an agents directory within the project root.

A problem was that while this would work with a local copy of the project, if the project was installed via pip , the agents directory could not be imported from because it was not recognized.

What I finally did was accept that I will only have files with the pattern agents_*.py in the project root. I was able to get this working with literally just the addition of one line to my load_agents() method as such:

sys.path.insert(0, os.getcwd())

Here is that method with the line in question . That coupled with the code I already had in place allowed this to work just fine.