Refer to a file within python package

Question

I would like to publish on PyPI a python package which has a method of a class that invokes an executable file.

The package is structured as follows:

/project
    LICENSE
    MANIFEST.in
    README.md
    setup.py
    /package_name
        classA.py
        classB.py
        executableFile
        classC.py

Inside a method of classA.py I call the executable file with the subprocess module as follows:

subprocess.call('./executableFile', shell=True)

If I install the package via pip in a virtualenv and I have a main.py script using the installed package as follows:

from package.classA import classA

obj_instance = classA()
obj_instance.method() #inside this method there is the subprocess.call(...)

The program crashes and tells me that executableFile is not found. I think this happens because the subprocess.call(...) searches the executableFile in the folder where there is the main.py instead of the package folder.

How can I specify the exact path of the executable file?

I've tried to use

os.path.dirname(os.path.abspath('executableFile'))

to get the package folder's name, but it returns me the folder path where there is the main.py

How can I solve?

Thanks in advance.

Answer 1

setuptools comes with a pkg_resources framework for handling cases like this.

There's a lot to read there, and I don't know of any good tutorials, but the basic idea is this:

You specify executableFile as a file resource. setuptools figures out where to stick that at install (or wheel-building) time.

At runtime, you call resource_filename to get the path to executableFile , and you subprocess.call that path.

You probably also want to atexit the cleanup_resources function.

Under the covers, pkg_resources will figure out all the details. With --inplace it will be a file in your source tree; with an exploded install, it will be a file somewhere in your site-packages ; with an egg install, or even as part of a site-packages zipped up by py2exe or the like, it will be a file extracted on demand to a cache directory. But you don't have to worry about that; you just get a path and it works.

---

Meanwhile:

I think this happens because the subprocess.call(...) searches the executableFile in the main.py folder instead of the package folder.

It searches the current working directory. That may be the same as the directory main.py is in, but only if your user ran your program by changing to that directory and running ./main.py or python main.py or similar.

I've tried to use

os.path.dirname(os.path.abspath('executableFile'))

That isn't going to work, because abspath is searching the current working directory too, so it's just a more complicated way of getting the same incorrect path. And then you dirname that, and get back the working directory, which is useless.

If you knew the relative path from package.classA.py to executableFile , you could find it this way:

packagedir = os.path.dirname(os.path.abspath(__file__))
execdir = os.path.join(packagedir, '<that relative path>')
execpath = os.path.join(execdir, 'executableFile')

Or, if you knew the relative path from main.py , you could do the same thing inside main.py and store the path somewhere to be used later (eg, pass it into the classA constructor).

But this isn't an ideal solution. The relative path within your source tree may not be the same as the relative path after installation. If your user installs your package as an egg, or bundles your package up in a zipfile with the rest of some application's library, you're just getting the path to the egg or zip, and there is no executableFile to be found anywhere relative to that, because it only exists inside the archive. And so on.

The point of pkg_resources is to solve all of those issues, and maybe others I didn't think of, and maybe others that don't exist in Python 3.7/pip 10/setuptools 40 but will exist in the Python 3.8/pip 12/setuptools 45 that one of your users is dealing with 18 months from now after you've long forgotten how you got it all working…