Is it Possible to Install Part of Python Package Via Pip?

Question

I have an internal utility library that is used by many projects. There is quite a bit of overlap between the projects in the code they pull from the utility library, but as the library grows, so does the amount of extra stuff any individual project gets that it won't be using. This wouldn't be an issue if the library consisted of only python, but the library also bundles in binaries.

Example-

psycopg2 is used in a handful of places within the utility library, but not all projects need db access. Because the development environment isn't the same as the production environment, the utility library also includes psycopg2 binaries for the prod environment.

This grows with openssl libraries, pandas, numpy, scipy, pyarrow, etc. The result is that a small, 50 line, single purpose script that may need db access is bundled into a 100mb+ deployment package.

So what I'd like to do is carve up the utility library into pieces where the projects down stream can choose which pieces to pull in, but keep the utility library code in one easy-to-manage place. That way, this small single purpose application can choose to import internal-util@core , internal-util@db and not include internal-util@numpy and internal-util@openssl

Is what i'm describing possible to do?

Answer 1

Not directly, to my best knowledge. pip installs a package fully, or not at all.

However, if you're careful in your package about how you import things that may require psycopg2 or someotherlargebinarything , you could use the extras_require feature and thus have the package's users choose which dependencies they want to pull in:

setup(
  # ...
  name='myawesometoolbelt',
  extras_require={
    'db': ['psycopg2'],
    'math': ['numpy'],
  },
)

and then, in your requirements.txt , or pip invocation,

myawesometoolbelt[db,math]

Answer 2

Have you tried looking at pip freeze > requirements.txt and pip install -r requirements.txt ?

Once you generated your pip list via pip freeze, it is possible to edit which packages you want installed and which to omit from the requirements.txt generated.
You can then pip install -r requirements.txt the things you want back in.