Trying to publish a Poetry package to AWS CodeArtifact. It supports pip
which should indicate that it supports poetry
as well since poetry
can upload to PyPi servers.
I've configured the domain like so:
export CODEARTIFACT_AUTH_TOKEN=`aws codeartifact get-authorization-token --domain XXXX --domain-owner XXXXXXXXXXXX --query authorizationToken --output text`
poetry config repositories.the_aws_repo https://aws:$CODEARTIFACT_AUTH_TOKEN@XXXX-XXXXXXXXXXXX.d.codeartifact.eu-central-1.amazonaws.com/pypi/XXXX/simple/
poetry config pypi-token.the_aws_repo $CODEARTIFACT_AUTH_TOKEN
But I'm getting 404 when trying to publish the package:
❯ poetry publish --repository the_aws_repo -vvv
No suitable keyring backend found
No suitable keyring backends were found
Using a plaintext file to store and retrieve credentials
Publishing xxx (0.1.5) to the_aws_repo
- Uploading xxx-0.1.5-py3-none-any.whl 100%
Stack trace:
7 ~/.poetry/lib/poetry/_vendor/py3.8/clikit/console_application.py:131 in run
129│ parsed_args = resolved_command.args
130│
→ 131│ status_code = command.handle(parsed_args, io)
132│ except KeyboardInterrupt:
133│ status_code = 1
6 ~/.poetry/lib/poetry/_vendor/py3.8/clikit/api/command/command.py:120 in handle
118│ def handle(self, args, io): # type: (Args, IO) -> int
119│ try:
→ 120│ status_code = self._do_handle(args, io)
121│ except KeyboardInterrupt:
122│ if io.is_debug():
5 ~/.poetry/lib/poetry/_vendor/py3.8/clikit/api/command/command.py:171 in _do_handle
169│ handler_method = self._config.handler_method
170│
→ 171│ return getattr(handler, handler_method)(args, io, self)
172│
173│ def __repr__(self): # type: () -> str
4 ~/.poetry/lib/poetry/_vendor/py3.8/cleo/commands/command.py:92 in wrap_handle
90│ self._command = command
91│
→ 92│ return self.handle()
93│
94│ def handle(self): # type: () -> Optional[int]
3 ~/.poetry/lib/poetry/console/commands/publish.py:77 in handle
75│ )
76│
→ 77│ publisher.publish(
78│ self.option("repository"),
79│ self.option("username"),
2 ~/.poetry/lib/poetry/publishing/publisher.py:93 in publish
91│ )
92│
→ 93│ self._uploader.upload(
94│ url,
95│ cert=cert or get_cert(self._poetry.config, repository_name),
1 ~/.poetry/lib/poetry/publishing/uploader.py:119 in upload
117│
118│ try:
→ 119│ self._upload(session, url, dry_run)
120│ finally:
121│ session.close()
UploadError
HTTP Error 404: Not Found
at ~/.poetry/lib/poetry/publishing/uploader.py:216 in _upload
212│ self._register(session, url)
213│ except HTTPError as e:
214│ raise UploadError(e)
215│
→ 216│ raise UploadError(e)
217│
218│ def _do_upload(
219│ self, session, url, dry_run=False
220│ ): # type: (requests.Session, str, Optional[bool]) -> None
My AWS IAM user has permission to do this since I gave it the relevant permissions in the repo.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::XXXXXXXXXXXX:user/ShayN"
},
"Action": [
"codeartifact:AssociateExternalConnection",
"codeartifact:CopyPackageVersions",
"codeartifact:DeletePackageVersions",
"codeartifact:DeleteRepository",
"codeartifact:DeleteRepositoryPermissionsPolicy",
"codeartifact:DescribePackageVersion",
"codeartifact:DescribeRepository",
"codeartifact:DisassociateExternalConnection",
"codeartifact:DisposePackageVersions",
"codeartifact:GetPackageVersionReadme",
"codeartifact:GetRepositoryEndpoint",
"codeartifact:ListPackageVersionAssets",
"codeartifact:ListPackageVersionDependencies",
"codeartifact:ListPackageVersions",
"codeartifact:ListPackages",
"codeartifact:PublishPackageVersion",
"codeartifact:PutPackageMetadata",
"codeartifact:PutRepositoryPermissionsPolicy",
"codeartifact:ReadFromRepository",
"codeartifact:UpdatePackageVersionsStatus",
"codeartifact:UpdateRepository"
],
"Resource": "*"
}
]
}
What am I missing?
The problem is the /simple/
at the end of the repo url. This part should only be added when pulling from that repo, not when publishing to it. If you look closely to the documentation of AWS CodeArtifact on how to publish with twine
, you'll see that it's also not there.
This works:
# This will give the repo url without the /simple/ part
# Example: https://<my-domain>-<domain-owner-id>.d.codeartifact.<region>.amazonaws.com/pypi/<my-repo>/
# Note the lack of the "aws:auth-token@" part
export CODEARTIFACT_REPOSITORY_URL=`aws codeartifact get-repository-endpoint --domain my-domain --domain-owner domain-owner-id --repository my-repo --format pypi --query repositoryEndpoint --output text`
# This will give the token to access the repo
export CODEARTIFACT_AUTH_TOKEN=`aws codeartifact get-authorization-token --domain my-domain --domain-owner domain-owner-id --query authorizationToken --output text`
# This specifies the user who accesses the repo
export CODEARTIFACT_USER=aws
# Now use all of these when configuring the repo in poetry
poetry config repositories.<my-repo-name-for-poetry> $CODEARTIFACT_REPOSITORY_URL
poetry config http-basic.<my-repo-name-for-poetry> $CODEARTIFACT_USER $CODEARTIFACT_AUTH_TOKEN
Note that the authentication token will expire when your AWS login session ends. Hence, you'll have to set the http-basic.<my-repo-name-for-poetry>
with the new token every time it expires.
FYI, I had the same problem and it took me hours to figure this out. But in the end, more carefully reading the documentation should have helped me.
This Python script can configure pip, poetry, and twine to work with your artifactory. Run eg
python ./codeartifact_login.py configure --tool poetry
poetry publish --repository your-artifactory
#!/usr/bin/env python
import os
import boto3
import fire
class CodeArtifact(object):
@staticmethod
def run(cmd: str):
print(os.popen(cmd=cmd).read())
@staticmethod
def configure(tool: str = "twine"):
if tool not in ["twine", "pip", "poetry"]:
raise RuntimeError(f"Not recognise tool: {tool}")
domain = "your-artifactory"
repo = "your-repo"
account_id = boto3.client('sts').get_caller_identity().get('Account')
region = boto3.session.Session().region_name
if tool == "poetry":
CODEARTIFACT_REPOSITORY_URL = f"https://{domain}-{account_id}.d.codeartifact.{region}.amazonaws.com/pypi/{repo}/"
CodeArtifact.run(cmd=f"poetry config repositories.{domain} {CODEARTIFACT_REPOSITORY_URL}")
CODEARTIFACT_AUTH_TOKEN = boto3.client('codeartifact').get_authorization_token(
domain=domain,
domainOwner=account_id,
durationSeconds=12 * 3600
)["authorizationToken"]
CodeArtifact.run(cmd=f"poetry config http-basic.{domain} aws {CODEARTIFACT_AUTH_TOKEN}")
else:
CodeArtifact.run(
cmd=f"aws codeartifact login --tool {tool} --domain {domain} --domain-owner {account_id} --repository {repo}")
if __name__ == '__main__':
fire.Fire(CodeArtifact) # configure(tool="poetry")
References:
EDIT: See the accepted answer, it works!
If someone gets here from a Google search, here's the situation according to when I'm writing this (19 Dec 2020):
No built-support for this in poetry
. You can install from AWS CodeArtifact using it, but not upload unless you're OK with putting secrets in your pyproject.toml
file (the renewing URL with the token). My workaround is to upload using twine
(just follow AWS's guide for that) and install using poetry
(need to add a poetry.toml
file AND add CodeArtifact as a source in pyproject.toml
).
For anyone trying to get Poetry running on Windows Powershell you can use environment variables to get things working. Since windows commands cannot handle the long AWS authentication tokens, it only works if you use environment variables.
Hopefully these steps and command examples help:
$Env:AWS_ACCESS_KEY_ID={{ awsaccesskeyidhere }}
$Env:AWS_SECRET_ACCESS_KEY={{ secretaccesskeyhere }}
$Env:AWS_SESSION_TOKEN={{ abcdefghijklmnopqrstuvwxyz }}
$Env:CODEARTIFACT_AUTH_TOKEN=aws --region us-east-1 codeartifact get-authorization-token --domain {{ repo_domain }} --domain-owner {{ aws_account }} --query authorizationToken --output text
You can check if this worked by running echo $Env:CODEARTIFACT_AUTH_TOKEN
. If it worked, you'll see a long string of text. Also make sure the AWS region is the one you're actually using.
This part was not quite as obvious for me. The key here is to use the UPPERCASE name of your repository in the environment variable name, and it should be the same name in the next step as well. More information in the Poetry documentation for using environment variables .
$Env:POETRY_HTTP_BASIC_MYPRIVATEREPO_USERNAME=echo aws
$Env:POETRY_HTTP_BASIC_MYPRIVATEREPO_PASSWORD=$Env:CODEARTIFACT_AUTH_TOKEN
As another example, if your repo was named " secretprojectrepo ", you would use: $Env:POETRY_HTTP_BASIC_SECRETPROJECTREPO_USERNAME=echo aws
The repository name here should be lowercase. Notice the URL does not include the "/simple" on the end. If you copy it from the AWS Console you'll need to remove it since Poetry isn't using that protocol for uploading.
poetry config repositories.myprivaterepo https://myprivaterepo-1234567890.d.codeartifact.us-east-1.amazonaws.com/pypi/myprivaterepo
Note: You can get the URL from the "manual setup" instructions when you View the Connection Instructions in the AWS Console. Should be the format https://aws:$CODEARTIFACT_AUTH_TOKEN@myprivaterepo-1234567890.d.codeartifact.us-east-1.amazonaws.com/pypi/myprivaterepo/simple/
. Just remove the credentials part, "aws:$CODEARTIFACT_AUTH_TOKEN@" from the URL since we're using environment variables for providing the authentication information to Poetry.
poetry build
poetry publish -r myprivaterepo
To set up pulling from the repository, follow steps above for logging in to AWS and configuring Poetry. Next you'll need a section in your pyproject.toml
that indicates to Poetry which repository to use as the source. Additional information: Poetry Documentation
[[tool.poetry.source]]
name = "myprivaterepo"
url = "https://myprivaterepo-1234567890.d.codeartifact.us-east-1.amazonaws.com/pypi/myprivaterepo/simple/"
default = true
Two important notes:
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.