Automate PyPI publishing

[mbollmann]

Resolves #8305

With the help of our AI friends, I’ve tried to create a workflow that should automate publishing new releases of the acl-anthology package to PyPI.

It’s supposed to work as follows:

• Whenever a change to python/pyproject.toml (which is where a version bump would happen) is pushed to master, the check job of this workflow checks if a Git tag named py-v{{ $VERSION }} already exists.
  • It also checks if CHANGELOG.md appears to have been updated correctly, as a sanity-check. (This is not an issue when using the just prepare-new-release recipe, but a safeguard in case we bump the version some other way.)
  • It also triggers this check on a PR to master, and posts a message in the PR if a new release would be triggered upon merging.
• If it doesn’t, this triggers the release workflow, which:
  • Creates a new py-v{{ $VERSION }} tag and pushes it.
  • Builds and publishes the package using uv build and uv publish, respectively.

In terms of credentials, this should work because PyPI supports “trusted publishing” based on the repo name, workflow name, and environment the workflow runs in (named pypi here). I’ve already set this up on the PyPI side.

Steps we should take:

• On this repo, under “Settings → Environments”, create a new environment named “pypi” and restrict its access to the master branch, just to prevent mistakes. (must be done by @mjpost)
• It would be good if @mjpost and potentially other maintainers also created an account on pypi.org so I can add someone as co-maintainer there, so that I’m not the single point of failure for admin access there.

[github-actions]

Build successful. Some useful links:

• Complete site preview: https://preview.aclanthology.org/automate-pypi-publishing
• Potential changes of interest:

This preview will be removed when the branch is merged.

[mbollmann]

Also leaving Claude Sonnet’s wisdom here on how to test this workflow before merging – I’m not going to take a closer look at this today anymore though:

1. Test the check job logic on the PR itself

The version-parsing, tag-existence check, and changelog check don't need OIDC, an environment, or PyPI at all — they're just shell logic. You can validate all of it by temporarily adding a pull_request trigger:

on:
 push:
   branches:
     - master
   paths:
     - 'python/pyproject.toml'
 pull_request:          # temporary, for testing
 workflow_dispatch:

Push a commit to your PR branch that bumps the version (with and without a matching changelog entry, to test both pass and fail cases), and watch the check job run for real. Since release depends on check's output and (per our earlier setup) only does anything meaningful when triggered appropriately, this is safe as long as release's steps that mutate state (tag push, publish) don't accidentally fire on a pull_request event. Worth adding an explicit guard for extra safety:

release:
 needs: check
 if: needs.check.outputs.is-new == 'true' && github.event_name != 'pull_request'

Once you're happy, remove the pull_request: trigger before merging — or actually, consider keeping it (with that guard in place) as a permanent, free PR-time check that catches "forgot to update the changelog" before merge rather than after, on master, when it's more annoying to fix.

2. Test the OIDC / build / tag / publish mechanics against TestPyPI

This is the part that genuinely can't be verified without actually exercising GitHub's OIDC token exchange and PyPI's verification of it — but you don't need to point it at real PyPI to do that. TestPyPI supports trusted publishing the same way:

• Add a second trusted publisher entry, this time on test.pypi.org, for the same repo and workflow filename, with a distinct environment name (e.g. testpypi).
• Create that testpypi environment in repo settings — you can leave it unrestricted (or scoped to your working branch) since it's not your real release gate.
• Temporarily point uv publish at TestPyPI (uv publish --index testpypi, assuming you've defined that index in pyproject.toml as in uv's docs) and use a disposable version like 0.0.0.dev1 and a scratch tag prefix like test-py-v, so nothing collides with real release history.

To actually trigger this from your unmerged branch: workflow_dispatch normally only shows up in the GitHub UI for workflows already registered on the default branch. The practical workaround (documented in GitHub's community discussions) is to temporarily rely on the pull_request trigger from step 1 to get the workflow "seen" by GitHub, then dispatch it against your branch explicitly via the CLI:

gh workflow run "Tag & publish to PyPI" --ref your-branch-name

This gets you a real end-to-end run — OIDC exchange, the python/ vs root dist/ quirk, tag creation and push with contents: write, and an actual upload — all against TestPyPI, with zero risk to your real package.

[mjpost]

I created the environment. I'm mjpost on pypi. We have similar auto-publish functionality setup on sacrebleu if you want a reference point.

[mbollmann]

Oh cool, I'll do a comparison with the sacrebleu one before I test this tomorrow!

[mjpost]

mjpost/sacrebleu@5af5a3a

[mjpost]

I guess this is a bit old already and the tooling is probably better (I don't remember creating a pypi environment, for example), but perhaps useful.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.95%. Comparing base (b34d07e) to head (370c53b).

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #9400   +/-   ##
=======================================
  Coverage   96.95%   96.95%           
=======================================
  Files          36       36           
  Lines        3710     3710           
=======================================
  Hits         3597     3597           
  Misses        113      113           

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[mbollmann]

I’ve successfully got this to run with TestPyPI on a throwaway branch (#9410) and added a few more improvements to the workflow.

I also bumped setup-uv to the latest version in all workflows and gave them all more descriptive names while I was at it. :)

This is ready.