pip install private package from local with external dependency

Question

I am using pip to install a few private packages in a virtual environment. Using python version 3.7.3 and pip version 18.1

I have copied the.whl file of that private package to a local directory. Now that package has external dependencies whith another private packages and also public packages.

By Public Package I meant the package that is available from public repositories (pypi or piwheels).

I have tried using

pip install -f /path/to/private/package/directory

Now, this is failing with below message:

Could not install packages due to an EnvironmentError: 404 Client Error: Not Found for url: https://pypi.org/simple/my_private_package

Then I have tried below two command (expecting this would fail)

pip install -f /path/to/private/package/directory --no-index

and

pip install --no-index --find-links=/path/to/private/package/directory private-package  --extra-index-url https://pypi.org/simple --extra-index-url  https://www.piwheels.org/simple

and those failed as expected with same error message

Could not find a version that satisfies the requirement public-package==x.x.x (from private-package->another-private-package) (from versions: )

I also tried

pip install /path/to/private/package/directory/private_package_x_x_x.whl

This failed too with below error message. Which means that another private package could not be found in public repository. This is obvious.

Could not install packages due to an EnvironmentError: 404 Client Error: Not Found for url: https://pypi.org/simple/another-private-package/

So the last attempt that was tried:

pip install /path/to/private/package/directory/private_package_x_x_x.whl --find-links=/path/to/private/package/directory

Unfortunately, no luck this time too.

I can think of a couple of workarounds:

1. Download all the dependencies and keep it in the local directory (/path/to/private/package/directory)
2. Create a private server and provide index information over http

Now, not happy with option 1 as I have to download every time a new public dependency is added. Option 2 may be a good option - I may go with it, if no other option is found.

Any input suggestion will be greatly appreciated.

Answer 1

Thanks for the direction @Matteo Zanoni .

I accidently found out that providing --find-links in the requirement.txt works like a charm. ie pip will search both the public URL and the links.

You can also provide --extra-index-url http://your-url.org/index

The searching order is

1. Default public url (ie pypi an pywheels)
2. extra-index-url
3. find-links

Below solution is working for me. So, I have created a requirement.txt with below content

--find-links /path/to/private/package/directory

my-private-package

And here is the command output

(venv) user@host:/home/user> install -r requirement.txt
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Looking in links: /path/to/private/package/directory
Processing /path/to/private/package/directory/my_private_package-x.x.x.whl
Processing /path/to/private/package/directory/another_private_package-x.x.x.whl
Collecting public-package==x.x.x
  Using cached https://www.piwheels.org/simple/public-package/public_package-x.x.x-py3-none-any.whl (xx kB)
Installing collected packages: public-package, another-private-package, my-private-package
Successfully installed public-package-x.x.0 another-private-package-x.x.x my-private-package-x.x.x
(venv) user@host:/home/user> 

The dependency hierarchy was:

my-private-package
     |
     |-- another-private-package-x.x.x
              |
              |-- public-package-xxx 

---

If you are still reading this, this is how exactly I realised that --find-links can be added to requirement.txt and it will work with url.

To find out what are the dependencies in my private packages, I searched for how to find dependencies without installing the package. It took me to this answer List Python packages that will be installed from requirements.txt

python -m pip install pip-tools
pip-compile requirements.txt --output-file requirements-all.txt

Now, the pip-compile created the requirements-all.txt with --find-links in it with all the required dependencies in it.

So, to try out, I removed all the dependencies and just kept one package ie my-private-package and voila it worked..!!