Is it possible to use static Python modules in the builtin table as submodules of a Python package?

Question

In a static VTK build with Python support, the Python modules are compiled statically. These modules are then linked into the vtkpython binary and added to the builtin table prior to the Python interpreter being initialized. We cannot use shared Python modules because that would end up duplicating static globals from the core libraries into each of the shared Python modules.

I've tried "tricking" the import machinery by adding to sys.modules manually:

Python 2.7.16 (default, Apr 30 2019, 15:54:59) 
[GCC 8.3.1 20190223 (Red Hat 8.3.1-2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> import math
>>> sys.modules['vtkmodules.vtkCommonCorePython'] = math
>>> import vtkmodules.vtkCommonCorePython
>>> import vtkmodules
>>> from vtkmodules import vtkCommonCorePython
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name vtkCommonCorePython

so the import mechanisms cannot be tricked in this way in Python2. There might be some hope for Python3 that such a hack would work:

% bin/vtkpython
Python 3.7.3 (default, May 11 2019, 00:45:16) 
[GCC 8.3.1 20190223 (Red Hat 8.3.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> import math
>>> sys.modules['vtkmodules.vtkCommonCorePython'] = math
>>> from vtkmodules import vtkCommonCorePython
>>> vtkCommonCorePython.sin
<built-in function sin>

Here's the relevant code (full code in use is here , but I've stripped out MPI bits which are disabled in my build and some VTK log-to-terminal code for simplicity here).

#include "vtkPython.h"
#include "vtkPythonCompatibility.h"

#include "vtkPythonInterpreter.h"
#include "vtkpythonmodules.h"
#include <vtksys/SystemTools.hxx>
// SNIP: some headers removed for brevity.

#include <string>

int main(int argc, char **argv)
{
  std::string fullpath;
  std::string error;

  if (vtksys::SystemTools::FindProgramPath(argv[0], fullpath, error))
  {
    vtkPythonInterpreter::SetProgramName(fullpath.c_str());
  }

  // SNIP: MPI setup (associated class also removed above)

  /**
   * This function is generated and exposed in vtkpythonmodules.h.
   * This registers any Python modules for VTK for static builds.
   */
  vtkpythonmodules_load();

  // SNIP: VTK log-to-terminal logic

  return vtkPythonInterpreter::PyMain(argc, argv);
}

The vtkpythonmodules_load() function is generated by the bindings wrapper logic and looks like this for a static build (snipped for brevity):

#include "vtkCommonCorePython.h"
// SNIP: Includes for other Python init functions.
#if PY_VERSION_HEX < 0x03000000         
#define PY_IMPORT(module) PyImport_AppendInittab("vtkmodules." #module, init ## module)            
#else                                     
#define PY_IMPORT(module) PyImport_AppendInittab("vtkmodules." #module, PyInit_ ## module)         
#endif                                

static void vtkpythonmodules_load() {     
  PY_IMPORT(vtkCommonCorePython);
  // SNIP: PY_IMPORT calls for the rest of VTK's modules.
}

The vtkmodules package is otherwise Python code (including wrappers for users to actually import which ensure that dependent modules are also imported). For example, vtkmodules/vtkCommonDataModel.py looks like this:

from __future__ import absolute_import
from . import vtkCommonCore
from . import vtkCommonMath
from . import vtkCommonTransforms
from .vtkCommonDataModelPython import *

to ensure that the C++ classes used by those in vtkCommonDataModelPython are loaded first.

I'd expect that doing builtin modules would work in this case, but it seems not. Is there a way to do this? If not, is it something that just hasn't been necessary before and no one has asked or is it explicitly not supported? Thanks.

Answer 1

It seems that one potential solution is to place all of the builtin modules at the top-level rather than inside of a package. This is suitable for us right now because our compiled module names are implementation details anyways and we have the trampoline .py files to get dependency order correct so that base types are properly registered before derived types. Our fix is described in this MR .