If we want use S3 to host Python packages, how can we tell pip where to find the newest version?
Question
we are trying to come up with a solution to have AWS S3 to host and distribute our Python packages.
Basically what we want to do is using python3 setup.py bdist_wheel to create a wheel. Upload it to S3. Then any server or any machine can do pip install $http://path/on/s3 . (including a virtualenv in AWS lambda) (We've looked into Pypicloud and thought it's an overkill.)
Creating package and installing from S3 work fine. There is only one issue here: we will release new code and give them different versions. If we host our code on Pypi, you can upgrade some packages to their newest version by calling pip install package --upgrade .
But if you host your packages on S3, how do you let pip know there's a newer version exists? How do you roll back to an older version by simply giving pip the version number? Is there a way to let pip know where to look for different version of wheels on S3?
2 answers
solution1
13 2019-08-19 08:28:18
AWS S3 can be used as a pypi server with minimal configuration and no other additional dependencies.
Suppose you want to use the bucket mypackages for hosting your private packages - awesomepy and rattlesnake .
After you have created the bucket in S3, go to its properties, click on Static website hosting card and enable the option Use this bucket to host a website . Note down the Endpoint mentioned above the radio button.
Below is how you have to arrange the folders and files:
mypackages
|
|---awesomepy
| |---awesomepy-1.10.0-py27-none-any.whl
| |---awesomepy-1.10.3-py27-none-any.whl
| |---index.html
|
|---rattlepy
|---rattlesnake-0.1.1-py27-none-any.whl
|---index.html
The index.html files in the package folders will have content like below
<html>
<head>
<title>Links</title>
</head>
<body>
<h1>Links</h1>
<a href='awesomepy-1.10.0-py27-none-any.whl'>awesomepy-1.10.0-py27-none-any.whl </a>
<br />
<a href='awesomepy-1.10.3-py27-none-any.whl'>awesomepy-1.10.3-py27-none-any.whl </a>
</body>
</html>
Wheel packages have to follow proper naming convetion. Refer to PEP 427 for details. For the dummy package - awesomepy-1.10.3-py27-none-any.whl , we are assuming,
- Language implementation is Python 2.7 -
py27 - ABI tag is none -
none - Platform is any -
any
Finally you can install the packages using the below command. Ensure your packages have different names than that of pypi packages. Prefixing the package name with your company initials is one simple solution.
pip install awesomepy rattlesnake==0.1.1 --extra-index-url=<s3 Endpoint> --trusted-host=<s3 Endpoint without http>
Upgrading packages using pip install awesomepy --upgrade will also work with this approach.
Also, modify your bucket permissions to restrict the packages to intended clients. One way to do this is:
Select Permissions tab in the bucket --> Edit Bucket Policy
Original Credit: My ex-manager, Adlon :)
solution2
2 2017-07-14 13:03:20
Shameless plug: I had the exact same problem, and wasn't happy with any of the existing solutions, so I wrote a small Lmabda/Terraform combo that does the trick: https://github.com/berislavlopac/plambdapi
Hope it helps!
solution3
0 2020-06-14 00:49:29
What about wrapping up the whl file (eg yourpkg-1.0-py3-none-any.whl) inside another zip file (eg yourpkg.zip) with a deterministic name. Then you can set up some cron scripts to check locally whether the deterministic key has a new s3 file, and if so then unzip the whl and install it.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.