fix: relax tokenizers bound to support Python 3.14 - #99
Open
jyje wants to merge 4 commits into
Open
Conversation
Scoped, branch-only workflow to verify whether libs/upstage installs and passes its chat-model integration test on Python 3.13/3.14. Doesn't touch the shared _test.yml/_integration_test.yml templates used by every libs/* package.
tokenizers switched to abi3 (stable ABI) wheels starting at 0.21.0, making them forward-compatible with new CPython versions (including 3.14) without a rebuild. The previous pin (^0.20.0, i.e. <0.21.0) was stuck on pre-abi3 releases, which have no Python 3.14 wheel and fail to build from source (PyO3 in tokenizers 0.20.x doesn't recognize 3.14). Relaxes the pin to >=0.21.0,<1.0.0 and regenerates poetry.lock, which now resolves tokenizers 0.23.1. Verified no regression: all 293 unit tests pass (including the tokenizer-specific ones, test_upstage_tokenizer_solar_pro2/solar_mini), and all 11 integration tests in test_chat_models.py pass against a real Upstage API key. Related: langchain-ai#34 (a prior dependency-conflict report caused by this same pin, from a different angle).
The py314_pin_check.yml workflow was branch-only scaffolding used to capture before/after CI evidence for the tokenizers pin change (linked from the PR description). It isn't meant to ship upstream, so remove it from the PR diff — the recorded fork runs remain as evidence.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description:
tokenizersis pinned to^0.20.0(i.e.>=0.20.0,<0.21.0) inpyproject.toml— a narrow window that both blocks Python 3.14 and conflicts with sibling packages that need a newertokenizersno release in that range ships a Python 3.14 wheel, and building it from source fails — PyO3 bundled inside
tokenizers0.20.x doesn't recognize Python 3.14 yettokenizersalready solved this upstream: starting at0.21.0, it shipsabi3(stable ABI) wheels — forward-compatible with new CPython versions without a rebuildthis PR relaxes the bound to
>=0.21.0,<1.0.0, removing the<0.21ceiling solangchain-upstagecan co-resolve with packages such astransformers/vllmthat require a newertokenizersno source changes needed — this package only calls
Tokenizer.from_pretrained()and.encode(), both unchanged across that rangeTesting:
local (Python 3.13,
poetry install --with test,test_integration): 293 unit tests passed, 1 skipped (unrelated) — including both tokenizer-specific tests,test_upstage_tokenizer_solar_pro2andtest_upstage_tokenizer_solar_minilocal: all 11 tests in
tests/integration_tests/test_chat_models.pypassed against a real Upstage API keylocal:
ruff checkandruff format --diffboth clean on the 2-file diff (pyproject.toml,poetry.lock)CI, before vs. after — a scoped workflow run on my fork (the workflow itself is not part of this PR's diff, run only to capture this comparison):
^0.20.0)poetry installbefore: 3.14 fails with
error: the configured Python interpreter version (3.14) is newer than PyO3's maximum supported version (3.13), thenCargo build finished with "exit status: 101"while buildingtokenizers-0.20.3after: both versions install a prebuilt
abi3wheel (no compile step) and pass unit + integration testsIssue:
tokenizersrelease below0.21.0ships a 3.14 wheel (see Description).<0.21ceiling letslangchain-upstageco-resolve with packages (e.g.transformers/vllm) that require a newertokenizersthan the old pin allowed.Dependencies:
langchain-openai(already a dependency of this package) declares notokenizersconstraint of its own