Python setup.py: how to include local package dependency?

Question

The question is pretty much self explanatory. I have got a .zip package dependency that I need to include in my python package installation. So how do I include this local dependency in my setup.py ? I have got install_requires but this loads my dependency packages from PyPI which is not what I want in this case.

The most relevant files in my project are structured as follows:

myproject
|- setup.py
|- mypackage
   |- __init__.py
   |- vendor
      |- __init__.py
      |- dependencies
         |- StreamingDataReader.zip

I then tried ...

include_package_data=True,
package_data={
    'StreamingDataReader': ['mypackage/vendor/dependencies/StreamingDataReader.zip'],
}

... with no success. My code can't still locate the package: ImportError: No module named 'StreamingDataReader'

Answer 1

This is an old question, but I have something to say about it, so I'm posting.

You have two separate problems here. The first is that the zip file needs to be included in your distribution. To do this, use package_data as you're intending, but you need to specify, for your package, what data within it you want to include. You do this as follows:

   package_data= {"mypackage": ["vendor/dependencies/StreamingDataReader.zip"]}

Do not use include_package_data at the same time; this looks for data in MANIFEST.in and ignores the package_data parameter. setuptools is really badly designed for managing this. The best advice I've found, for both binary and source packaging, is to ensure that all your data is within the package directory, and to use package_data and nothing else.

Your second problem is that you want to be able to import StreamingDataReader . Unfortunately, as far as I can tell, there's no way to do this with pip using pip install and nothing else. You have two obvious choices:

(1) You can include a requirements.txt file and list the zip file as a requirement, and have people do pip install -r requirements.txt in advance of pip install ing your project. Your requirements.txt file would look like this:

./mypackage/vendor/dependencies/StreamingDataReader.zip

But requiring the user to do that may be an unacceptable level of pain, and it won't work for anyone who depends on your package.

(2) Brute force. Just use zipfile to expand the package.

I'm going to assume that StreamingDataReader.zip file is a Python package zip, with a single directory at the toplevel (I'm assuming it's called StreamingDataReader ) which contains a toplevel setup.py and a subdirectory, let's assume it's called StreamingDataReader , which contains the actual software. In other words, I assume your zip looks like this:

StreamingDataReader/
StreamingDataReader/setup.py
StreamingDataReader/StreamingDataReader/__init__.py

etc. I'm also going to assume that you're running Python 3.3 or later, in which having an __init__.py file is no longer required to create a package hierarchy. (In previous versions, you'd need to manipulate sys.path , which isn't a big deal, but it's now easier.) So vendor/__init__.py isn't doing any work for you.

This solution, by the way, assumes that the folks who run your software will have write permission on the directory which contains it.

Put the following in your toplevel __init__.py :

import zipfile, pathlib

_mpath = pathlib.Path(__file__).resolve().parent
_sdrDir = mpath.joinpath("vendor", "dependencies", "StreamingDataReader")
if not _sdrDir.is_dir():
    _z = zipfile.Zipfile(sdrDir.parent.joinpath("StreamingDataReader.zip"), "r")
    _z.extractall(sdrDir.parent)
    _z.close()
    del _z
# cleanup
del _mpath
del _sdrDir

Once you do this, you can

import mypackage.vendor.dependencies.StreamingDataReader.StreamingDataReader as StreamingDataReader

wherever you want.

Then, finally, in the setup.py for your package, you'll need to include the dependencies for StreamingDataReader , because you're not recursively installing it with pip .

Yes, that's a lot of work. But as far as I know, there's no facility in the Python installation mechanism for otherwise installing a package from a zip file you provide yourself.