Tag and release pypdfium2 fork of ctypesgen

[tiran]

We would like to build pypdfium2 with a custom build of libpdfium in an offline environment. It would make our lives easier if you could tag and release the current head of main on GitHub. We prefer to use released versions and like to avoid "random" git commits in our build pipeline.

You get extra kudos if you push your fork to PyPI, too.

Some ideas

pypdfium2-ctypesgen would be logical name for this fork of ctypesgen.

You could either stick to the version schema of upstream and release the package as 1.2.0. Or you could break with upstream versioning and align the version numbers with pypdfium2. The current HEAD of main is used for pypdfium2 4.30.0, so you could release the package as pypdfium2_ctypesgen-4.30.0.

[mara004]

Wow, I didn't expect a CPython dev to be interested in my amateur projects!

I'm glad the code is being used, but sorry to disappoint: I don't plan to make releases of pypdfium2-ctypesgen or publish it to PyPI, and I'm reluctant to rename the package.

This project isn't meant for a release-based development model. The intent is just to have our pypdfium2 and ctypesgen codebases coupled together to facilitate cooperative development (i.e. the HEAD of pypdfium2 should use the HEAD of ctypesgen).

Prefer to see this as an external dev branch of ctypesgen, or as a setup-time utility module of pypdfium2, rather than as a real own project.

[mara004]

It would make our lives easier if you could tag and release the current head of main on GitHub.

Note that our sdist releases auto-pin pypdfium2-ctypesgen to the current HEAD at compile time, so if you install from the sdist you should not have to do any tricky determining which ctypesgen commit a pypdfium2 release was built with.

That is, as long as you don't need to install with caller-provided data files. Otherwise, ctypesgen has to be managed on the caller side. I gather the main reason for downstreams to use that is currently offline setup, which is a bit unfortunate, as that could theoretically be done with an internal ctypesgen call.
We might be able to add new setup capabilities to better handle this case. Like taking a full pdfium version (to avoid web calls to resolve it from the build cipher), a path to custom pdfium headers (e.g. $PDFIUM_HEADERS), and new system-path: and bundle: strategies for $PDFIUM_PLATFORM.

[tiran]

The reason for my interest in your project is a bit ... complicated. :) I have filed the ticket as an engineer at Red Hat.

Long story short, pypdfium2 is a dependency of the IBM Deep Search package Docling, which itself is becoming a dependency of InstructLab, which is the core of RHEL AI. For various reasons like supply chain security and SBOM, we are not using any Python wheels and other binaries. Instead we build everything from upstream sources, either as RPMs or as Python wheels. The Python wheels span a range of trivial, pure Python packages to PyTorch builds for different hardware accelerators like AMD ROCm and NVIDIA CUDA.

We have created Fromager pipeline, which resolves dependencies, retrieves sdists, and then build each wheel in an isolated virtual env without network access. All build time dependencies must be resolved and pre-fetched before a build. The approach works out of the box for most Python packages, but there are some oddballs that download code at build time. (CMake FetchContent is popular in the AI space.) In general, we don't bother upstream projects with our downstream policies. We only engage with upstream, when we think that a change is going to help upstream and the broader Python community in the long run.

pypdfium2 is on the oddballs. We are building our own libpdfium.so, so we need to build a custom pypdfium2. Since we install all build-time dependencies offline, PDFIUM_PLATFORM="system:$PDFIUM_VER" does not work for us out of the box. The setting triggers a download of this fork at build time.

I see an opportunity to make everybody's life a bit easier by releasing and publishing this fork under a new name on PyPI.

1. release the fork as pypdfium2_ctypesgen on PyPI.
2. change https://github.com/pypdfium2-team/pypdfium2/blob/4.30.0/pyproject.toml#L10 to install pypdfium2_ctypesgen

Then you can remove the complicated ctypesgen git fetch code from https://github.com/pypdfium2-team/pypdfium2/tree/4.30.0/setupsrc/pypdfium2_setup and replace it with a simple import ctypesgen + importlib.metadata.metadata("pypydfium2_ctypesgen") to verify that the correct fork is installed. For local development and testing, you can still install ctypesgen locally in a virtual env.

If you are interested, then I can work with you and create PRs. If you don't think it's worth your while, then that's also okay. We can work around the issues internally.

[mara004]

Since we install all build-time dependencies offline, PDFIUM_PLATFORM="system:$PDFIUM_VER" does not work for us out of the box. The setting triggers a download of this fork at build time.
Then you can remove the complicated ctypesgen git fetch code from https://github.com/pypdfium2-team/pypdfium2/tree/4.30.0/setupsrc/pypdfium2_setup and replace it with a simple import ctypesgen + importlib.metadata.metadata("pypydfium2_ctypesgen") to verify that the correct fork is installed.

This is a bit imprecise: The download of this fork is triggered through standard pyproject.toml dependency means, which you can prevent by passing --no-build-isolation/--no-isolation to pip/build, and instead prepare ctypesgen on the caller side as you like due to security policies etc. This is not related to PDFIUM_PLATFORM, and no code in pypdfium2_setup clones or installs ctypesgen. (Only for creating the sdist itself, there is a web call to get the ctypesgen HEAD commit hash for pinning, and some tmp editing of pyproject.toml, which is indeed a bit dirty.)
However, PDFIUM_PLATFORM=system:$PDFIUM_VER does involve other web calls: to resolve the $PDFIUM_VER short cipher to a full version (e.g. 6611 -> 128.0.6611.0), and to download pdfium headers. These issues I intend to address, as stated above (in the meantime, use caller-provided data files).

The only remaining problem I see then is figuring out the right ctypesgen commit hash for a pypdfium2 release on your side. Parsing it out of the sdist's pyproject.toml shouldn't be too difficult. If you feel like this should be made more accessible, I could add the hash e.g. to autorelease/record.json or to the auto-generated changelog headers.

While I am interested in working with downstreams to improve the offline installs situation, I do not want to move away from "smart" branch tracking. The release-free model is quite intentional. If people install from pypdfium2's git main, I want them to also get latest ctypesgen git by default, not just a snapshot from the previous pypdfium2 release date. One reason is to avoid breakage when we make API changes but have not released them yet. I do not want to be forced to make releases every time we change the API, as we tend to experiment (consider the kind of changes that are made one day and re-changed the other day across both codebases).
Even historically, when we didn't have the fork yet, pypdfium2 was tracking ctypesgen git due to lack of recent releases at that time. See ctypesgen#129 and ctypesgen#201 for background.

My concern here is that you want us to put pypdfium2 information into ctypesgen, by tagging/releasing in sync with pypdfium2. However, ctypesgen is technically an independent project, and your suggestion would fall apart if this work were ever upstreamed and we still wanted to track a branch, as ctypesgen cannot make its release schedule or tagging dependent on a downstream project. So I think this is not logically the right solution.

PS: Before you spend time on PRs, it's best if we first agree on what to change, and some changes I might prefer to do myself.

[mara004]

After over a year, this issue came back to my mind.

In retrospect, I think you've got a point here. It isn't ideal to rely on a setup dependency from GitHub in a PyPI release.
Let's just assume GitHub would move to a new URL. All our previous sdists would break because they rely on the old URL. That's not good.

On the other hand, if we had ctypesgen releases on PyPI, and would depend on these in sdists, we'd be immune to that sort of breakage. We don't rely on foreign URLs, just a hermetic set of packages that could be re-uploaded anywhere.

(FWIW, I don't think the GitHub scenario is plausible, it just shows an issue inherent to the current approach.)

[mara004]

The main question is how to implement this, because we'd have to release ctypesgen just on before a pypdfium2 release, so it would need to be integrated into pypdfium2's release automation.

I'm not sure there's a good way to do cross-repository workflow triggers, but maybe we could just release ctypesgen from within pypdfium2's main workflow?

[mara004]

I'd still want to keep the head-to-head development approach and track latest ctypesgen from git in pypdfium2's main, but instead of replacing it to a github commit pin, we'd just change it to a package version on release.