GitHub action - publishing to both TestPyPI and PyPI fails

Question

I set up a GitHub workflow for publishing my Python project to (Test)PyPI according to this guide: https://packaging.python.org/en/latest/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/ resulting in this workflow: https://github.com/felixocker/ontor/blob/main/.github/workflows/publish-to-pypi.yml

Whenever I push to the main branch, a new TestPyPI release is created:

- name: Publish distribution to Test PyPI
  uses: pypa/gh-action-pypi-publish@master
  with:
    password: ${{ secrets.TEST_PYPI_API_TOKEN }}
    repository_url: https://test.pypi.org/legacy/

This works just fine, cp., eg, commit 15a3bf6

When I push with a tag, this should push to TestPyPI as usual, but the tag should also trigger pushing to PyPI:

- name: Publish distribution to PyPI
  if: startsWith(github.ref, 'refs/tags')
  uses: pypa/gh-action-pypi-publish@master
  with:
    password: ${{ secrets.PYPI_API_TOKEN }}

However, this fails with a 400 error, cp., eg, commit b450e0f:

ERROR    HTTPError: 400 Bad Request from https://test.pypi.org/legacy/
         File already exists. See https://test.pypi.org/help/#file-name-reuse
         for more information.

Interestingly, the error already occurs in the step for pushing to TestPyPI, and pushing to PyPI is skipped after that. Also, when I tried uploading the latest version to PyPI manually, I got the same error. After that manual attempt, the version was available via PyPI, though.

Really not sure what I am missing here - so pointers are greatly appreciated :)

#################################################

Edit #1:

The behavior makes sense to me for the commits 15a3bf6 and b5cade1, as I did not change the version number here.

What I just realized thanks to @Indra's response and the action overview is that the workflow is executed twice for the same commit: b450e0f in runs #33 and #34
Are there two runs because tags are treated as pushes of their own (which might be indicated by "v0.4.2" vs "main")?
I pushed the commit and the respective tag using

git push --atomic origin main <tag>

which should be simultaneous, though.
If so, is there a way to keep the intended behavior: "publish to TestPyPI whenever I push to main, publish to PyPI whenever there is a tag" without having two runs, assuming that I update the version?

Answer 1

In your error message it should be clear that your workflow attempt to upload the same package version to TestPyPI. PyPI doesn't allow to upload package with the same version more than once. See here . That's why you get File already exists error.

In your TestPI workflow:

- name: Publish distribution to Test PyPI
  uses: pypa/gh-action-pypi-publish@master
  with:
    password: ${{ secrets.TEST_PYPI_API_TOKEN }}
    repository_url: https://test.pypi.org/legacy/

GitHub actions will upload the package to TestPyPI whenever you push to the repository. If your package point to the same version when you push more than once, the workflow will fail because the package already uploaded to TestPyPI on the first push.

• Commit 15a3bf6

Everything works fine. Version 0.3.7 was uploaded to TestPyPI on GitHub push to main branch.

• Commit b5cade1

Package version still point to 0.3.7. GitHub push to main branch trigger actions and try to reupload the package to TestPyPI and failed because the version already exists.

GitHub action skip upload to PyPI because the commit is not tagged.

• Commit f732bda

Package version changed to 0.4.0. The first push successfully build and upload the package to TestPyPI, but subsequent upload failed. The error still the same, GitHub actions try to upload the package with the same version.

So on.

You could change the TestPyPI workflow to upload only on tagged commit or you could change the package version whenever you push to the GitHub repository. The first solution would be more feasible.

TestPyPI:

- name: Publish distribution to Test PyPI
  if: startsWith(github.ref, 'refs/tags')
  uses: pypa/gh-action-pypi-publish@master
  with:
    password: ${{ secrets.TEST_PYPI_API_TOKEN }}
    repository_url: https://test.pypi.org/legacy/

PyPI:

- name: Publish distribution to PyPI
  if: startsWith(github.ref, 'refs/tags')
  uses: pypa/gh-action-pypi-publish@master
  with:
    password: ${{ secrets.PYPI_API_TOKEN }}

Answer 2

So here is what happened:
Apparently, GitHub treats the pushes for a commit and a tag separately even when using

git push --atomic origin main <tag>

via SSH, see this discussion .
This results in the github action being run twice, as it is triggered by every push. The version number is already being used for TestPyPI in the first run, yielding a file already exists error in the second.

What worked for me:
Making the two cases for pushing to TestPyPI and PyPI exclusive according to this discussion as such:

    - name: Publish distribution to Test PyPI
      if: ${{ !startsWith(github.ref, 'refs/tags') }}
      uses: pypa/gh-action-pypi-publish@master
      with:
        password: ${{ secrets.TEST_PYPI_API_TOKEN }}
        repository_url: https://test.pypi.org/legacy/
    - name: Publish distribution to PyPI
      if: startsWith(github.ref, 'refs/tags')
      uses: pypa/gh-action-pypi-publish@master
      with:
        password: ${{ secrets.PYPI_API_TOKEN }}

Whenever a commit without a tag is pushed to GitHub, this triggers the push to TestPyPI. When a tag is pushed, the version is released to PyPI. A caveat is that the commit must be handled by the action before the tag, which, in my case, did not work for commit f732bda. Also, as mentioned by @Indra, the version number must be incremented for the action to work. Alternatively, a distinction between regular and hierarchical tags, eg, "v0.3.0/beta", could be made as described here .