Boost-Python: Load python module with unicode chars in path

Question

I'm working on game project. I use python 2.7.2 for scripting. My application works fine with non unicode path to .exe. But it can't load scripts with unicode path using boost::python::import (import_path.c_str());

I tried this example 5.3. Pure Embedding http://docs.python.org/extending/embedding.html#embedding-python-in-c

It also can't handle unicode path. I linked python as dll. Explain me, please, how to handle such path.

Answer 1

boost::python::import needs a std::string , so chances are that import_path misses some characters.

Do you have to work on multiple platform ? On Windows, you could call GetShortPathName to retreive the 8.3 filename and use that to load your dll.

You can make a quick test :

1. Rename your extension to "JaiDéjàTestéÇaEtJaiDétestéÇa.pyd".
2. At the command line, type dir /x *.pyd to get the short file name (JAIDJT~1.PYD on my computer)
3. Use the short name to load your extension.

+The file name above if French for "I already tested this and I didn't like it". It is a rhyme that takes the edge off working with Unicode ;)

Answer 2

This isn't really an answer that will suit your needs, but maybe it will give you something to go on.

I ran into a very similar problem with Python, in my case my application is a pure Python application. I noticed as well that if my application was installed to a directory with a path string that could not be encoded in MBCS (what Python converts to internally for imports, at least Python prior to 3.2 as far as I understand), the Python interpreter would fail, claiming not module of that name existed.

What I had to do was write an Import Hook to trick it into loading those files anyway.

Here's what I came up with:

import imp, os, sys

class UnicodeImporter(object):
    def find_module(self,fullname,path=None):
        if isinstance(fullname,unicode):
            fullname = fullname.replace(u'.',u'\\')
            exts = (u'.pyc',u'.pyo',u'.py')
        else:
            fullname = fullname.replace('.','\\')
            exts = ('.pyc','.pyo','.py')
        if os.path.exists(fullname) and os.path.isdir(fullname):
            return self
        for ext in exts:
            if os.path.exists(fullname+ext):
                return self

    def load_module(self,fullname):
        if fullname in sys.modules:
            return sys.modules[fullname]
        else:
            sys.modules[fullname] = imp.new_module(fullname)
        if isinstance(fullname,unicode):
            filename = fullname.replace(u'.',u'\\')
            ext = u'.py'
            initfile = u'__init__'
        else:
            filename = fullname.replace('.','\\')
            ext = '.py'
            initfile = '__init__'
        if os.path.exists(filename+ext):
            try:
                with open(filename+ext,'U') as fp:
                    mod = imp.load_source(fullname,filename+ext,fp)
                    sys.modules[fullname] = mod
                    mod.__loader__ = self
                    return mod
            except:
                print 'fail', filename+ext
                raise
        mod = sys.modules[fullname]
        mod.__loader__ = self
        mod.__file__ = os.path.join(os.getcwd(),filename)
        mod.__path__ = [filename]
        #init file
        initfile = os.path.join(filename,initfile+ext)
        if os.path.exists(initfile):
            with open(initfile,'U') as fp:
                code = fp.read()
            exec code in mod.__dict__
        return mod

sys.meta_path = [UnicodeImporter()]

I still run into two issues when using this:

1. Double clicking on the launcher file (a .pyw file) in windows explorer does not work when the application is installed in a trouble directory. I believe this has to do with how Windows file associations passes the arguments to pythonw.exe (my guess is Windows passes the full path string, which includes the non-encodeable character, as the argument to the exe). If I create a batch file and have the batch file call the Python executable with just the file name of my launcher, and ensure it's launched from the same directory, it launches fine. Again, I'm betting this is because now I can use a relative path as the argument for python.exe, and avoid those trouble characters in the path.
2. Packaging my application using py2exe, the resulting exe will not run if placed in one of these trouble paths. I think this has to do with the zipimporter module, which unfortunately is a compiled Python module so I cannot easily modify it (I would have to recompile, etc etc).