refactor(server): consolidate core+graph+admin into modules/server (sub-PR 4 of 2026.5)#1073
Open
danh91 wants to merge 22 commits into
Open
refactor(server): consolidate core+graph+admin into modules/server (sub-PR 4 of 2026.5)#1073danh91 wants to merge 22 commits into
danh91 wants to merge 22 commits into
Conversation
…core) (#1068) Exported from jtlshipping/shipping-platform main via ./bin/export-karrio-patches --force per PRDs/SUBTREE_SYNC_WORKFLOW.md. 1445 files across modules/connectors (579), modules/core (198), apps/api (160), modules/manager (113), modules/pricing (75), modules/sdk (55), modules/events (46), modules/orders (39), modules/data (36), modules/documents (28), modules/graph (25), modules/admin (24) + packages, plugins, scripts, and PRDs. Major functional additions from shipping-platform: - rate sheet Excel/CSV import/export (apps/, packages/, modules/data/) - Playwright e2e package scaffold (packages/e2e/) - dhl_parcel_de improvements (optional dims, reference fallback) - carrier i18n translation expansions (dhl_parcel_de, asendia) - ruff-format reflow across the tree from SP's linter config 54 three-way conflicts resolved per SUBTREE_SYNC_WORKFLOW.md matrix (see PR body for per-file decisions). Smartkargo, FedEx signature defaults, rate-sheet UX variants, and graph archive/unarchive kept at upstream (ours); carrier i18n dicts merged; e2e fixtures and de locale compiled catalog copied directly from SP. Pre-excluded from apply: 10 phantom-deletions of SP-experimental credential/secret modules that never reached upstream, and 5 binary-new files that needed direct copy (xlsx fixtures, django.mo). No JTL-proprietary modules in diff.
Bundles six workstreams targeting the 2026.5.0 release: adopt dev-tooling from JTL shipping-platform (ruff + pre-commit, .claude/ skills/rules), consolidate OSS modules/core + modules/graph + modules/admin into a single auto-discovering modules/server/ package (scoped down from the full issue #431 vision, NOT ee/insiders/modules/admin), finalize the Helm chart for EKS, add a Huey HTTP producer layer so API and worker pods scale independently, add a Playwright golden-path E2E gate in CI before release tags, and run the scripted subtree-sync workflow (per SUBTREE_SYNC_WORKFLOW.md) to pull shipping-platform changes upstream. Each workstream lands as its own sub-PR; tag 2026.5.0 after all merge.
Picks up karrioapi/karrio-insiders#32 (158 files, orgs/automation/apps/audit). Completes Wave 0 of the 2026.5 umbrella on feat/2026.5 — paired with the karrio-side sync merged as #1068 (now in main).
Post-Wave-0-sync fixes — CI was failing on feat/2026.5 for two reasons introduced by the shipping-platform sync (#1068): 1. modules/huey/ exists in the tree (task-backend abstraction from SP) but was never added to requirements.server.dev.txt or requirements. build.txt. INSTALLED_APPS declares huey.contrib.djhuey, so Django AppConfig.create() fails with ModuleNotFoundError: No module named 'huey'. Add -e ./modules/huey to both files — its pyproject pulls huey>=2.5 transitively. 2. modules/sdk/karrio/lib.py:259 uses Python 3.12 type-parameter syntax (def to_list[T]: ...) but mypy.ini declared python_version = 3.10, so SDK typecheck fails. Bump to 3.12 to match the runtime.
Second round of post-sync CI stabilization fixes. 1. modules/sdk/karrio/core/utils/enum.py — after the mypy python_version bump to 3.12, the stricter type-checker flagged two helper signatures (asValue and asKeyVal) where the 'type' parameter shadows the Spec.type attribute. Behavior is intentional (mirrors how Spec is consumed by callers). Add targeted '# type: ignore[valid-type]' instead of renaming the param, which would be a breaking API change. 2. modules/huey/karrio/server/settings/huey.py — 'huey.contrib.djhuey' was being appended twice (once in settings/base.py, once in the auto- discovered huey settings). Django boot failed with 'Application labels aren't unique: djhuey'. Drop the duplicate from the huey module — base.py is the canonical registration; the huey module only needs to register 'karrio.server.huey' itself.
Sub-PR 1 of the 2026.5.0 umbrella release (PRDs/RELEASE_2026_5_PLATFORM_UPGRADE.md).
Skills added (new):
- create-extension-module — scaffold new modules/<name>/ extension
- django-graphql — schema layout + auto-discovery patterns
- django-rest-api — view/serializer/router patterns
- run-tests — decision table mapping changed files to test commands
Skills updated:
- review-implementation — expanded N+1 check, extension-pattern gate, security check
Rules added (new):
- commit-conventions.md — type(scope) format, karrio scopes, no AI footer lines
- extension-patterns.md — golden rule, namespace-package caveats, core hook points
Rules updated:
- code-style.md — merge JTL's i18n/Django-models/serializers guidance with karrio specifics
- testing.md — imports-at-top rule + mocking guidance + Django/SDK patterns
Also:
- AGENTS.md — expand Context Priority with .claude/{rules,skills} load order
- CLAUDE.md — update rules/skills index to match new set
All ports adapt JTL-specific references (modules/entitlements, karrio subtree
framing) to karrio equivalents (modules/orders, modules/events, etc.). Karrio-
specific rules preserved: import karrio.lib as lib, no pytest, 4-method carrier
test pattern, no legacy DP/SF/NF/DF/XP utilities.
…jango-rest-api, create-extension-module)
Rewrite three skills that were copied near-verbatim from the sister JTL
repo so they reflect karrio's architecture, not JTL's.
django-graphql:
- Replace invented schema examples with real references to
schemas/base/{__init__,types,inputs,mutations}.py and the admin graph.
- Document karrio utilities: Connection[T], Paginated, BaseInput,
BaseMutation, paginated_connection, is_unset, authentication_required,
password_required.
- Use Model.access_by(info.context.request) for tenant scoping,
admin.staff_required / superuser_required for admin schemas.
- Note extra_types = [] requirement and the pkgutil.iter_modules
auto-discovery flow in schema.py.
- Point at karrio.server.graph.tests.GraphTestCase and real test files.
- Cross-reference .claude/rules/extension-patterns.md for the
thin-__init__ / one-way-dependency rules.
django-rest-api:
- Anchor on modules/manager/views/shipments.py and modules/orders/views.py
as the canonical examples.
- Switch base classes to karrio.server.core.views.api.GenericAPIView /
APIView (LoggingMixin + token/JWT/OAuth2 auth + access_by-aware
get_queryset), not raw DRF views.
- Document ENDPOINT_ID (5-char hash), extend_schema with x-operationId,
PaginatedResult factory, LimitOffsetPagination subclass pattern,
Serializer.map(...).save().instance, process_dictionaries_mutations,
owned_model_serializer, AccessMixin, ErrorResponse / ErrorMessages.
- Show the self-registering router.urls.append pattern and the
reverse("karrio.server.<module>:<name>") test convention.
- Replace JTL stack references with karrio's real layout, settings
auto-discovery, and requirements.build.txt + requirements.server.dev.txt
+ bin/run-server-tests registration checklist.
create-extension-module:
- Use modules/orders/ as the canonical reference (REST + GraphQL +
signals) instead of fabricated examples.
- Document AppConfig.ready() + @utils.skip_on_commands() pattern,
signals.register_signals() with @utils.disable_for_loaddata, and the
correct settings auto-discovery path
(modules/<name>/karrio/server/settings/<name>.py).
- Call out the namespace-package rule: no __init__.py at karrio/,
karrio/server/, or any shared schemas/ path.
- Match pyproject.toml against the real orders module (setuptools,
namespaces = true, karrio_server_* dependencies).
- Move tests from single tests.py to the tests/ directory pattern used
across modules, and list all three required registration files
(requirements.build.txt, requirements.server.dev.txt,
bin/run-server-tests).
- Port ruff.toml from jtlshipping/shipping-platform (target py312, line 120, rules E/W/F/I/B/S/UP/SIM/DJ/T20) adapted to karrio's modules/ layout; exclude ee/, community/, connectors' generated schemas, and templates. - Add .pre-commit-config.yaml with ruff + ruff-format + prettier + generic hooks. - Drop black + bandit from requirements.dev.txt; add ruff + pre-commit. - Add new lint job to .github/workflows/tests.yml running pre-commit --all-files on push and pull_request. Refs PRDs/RELEASE_2026_5_PLATFORM_UPGRADE.md (sub-PR 2).
Generated by:
ruff check --fix --unsafe-fixes . (ruff 0.15.11)
ruff format .
Only auto-fixable rule violations were applied; no hand edits here. AST
smoke check ('python -c "import ast; ast.parse(...)"' across
modules/ + apps/) passes on every touched file. Excludes in ruff.toml
prevent ee/, community/, node_modules, connectors' generated schemas,
and mapper.py/mapper-like auto-generated files from being touched.
Refs PRDs/RELEASE_2026_5_PLATFORM_UPGRADE.md (sub-PR 2, commit 2 of 3).
After commits 1+2, 'ruff check .' reported 44 findings across 16 files.
Grouped by rule:
B904 (9) : add 'raise ... from e' in graph mutations rate-sheet validators
E722 (10): narrow bare 'except:' to 'except Exception:' in GraphQL type resolvers (intentional fallbacks; noqa BLE001)
S110 (9) : noqa with justification — sentry/tracing/probe best-effort paths
F401 (5) : remove 2 unused graph schema imports; add __all__ in sdk i18n;
noqa on conditional loguru availability probe
B023 (2) : bind 'e=e' in smartkargo tracking failsafe lambdas (immediate-call, but satisfies linter)
E402 (2) : noqa for intentionally conditional/deferred imports
F811 (2) : **real duplicate class definitions** — FIXME + noqa pending human review
S301 (1) : noqa for constance pickle deserialization (pattern matches constance internals)
SIM102(1) : collapse nested 'if' in pricing fee-capture signal
SIM105(1) : noqa — logging not configured yet during settings bootstrap
UP007 (1) : convert 'typing.Union[...]' → 'X | Y' in abstract serializer
Real-bug escalations (gates human review):
- modules/connectors/dhl_parcel_de/tests/dhl_parcel_de/test_shipment.py:64
class TestDPDHLGermanyShippingOptionOverrides defined twice; the second
copy silently shadows the first and drops 2 test methods.
- modules/core/karrio/server/core/serializers.py:1250
class ShippingDocument defined twice; the second copy (with extra
'print_format' field) shadows the first.
Post-fix: 'ruff check .' → 'All checks passed!'
'ruff format --check .' → '1548 files already formatted'
AST parse smoke check across modules/ + apps/ → OK
Refs PRDs/RELEASE_2026_5_PLATFORM_UPGRADE.md (sub-PR 2, commit 3 of 3).
Adds a new STEP 5b between Python deps and Node deps that activates the venv and runs `pre-commit install --install-hooks` so the ruff + ruff-format hooks (added in commit 1) are wired into the developer's .git/hooks/pre-commit on first setup. No-ops gracefully if .venv/karrio or .pre-commit-config.yaml is missing; prints a manual-fix hint on failure.
chore(claude): port skills + rules from shipping-platform (sub-PR 1 of 2026.5)
chore(ruff): adopt ruff + pre-commit (sub-PR 2 of 2026.5)
0108_alter_secret_key_version_to_char and 0110_merge_0108_branches came in via the shipping-platform sync, but reference a Secret model that never existed upstream. The 10 source files (models/secret.py, credential_manager.py, secret_manager.py, rotation.py, management commands, the 0102 initial secret-storage migration, etc.) were phantom-deletions in the sync patch and intentionally skipped — so the 'alter' migration has nothing to alter and the 'merge' migration only existed to reconcile that dead branch. Chain is now linear again: 0107 -> 0108_clear_dhl_parcel_de... -> 0109_cleanup_legacy_system_rate_sheets.
Add labelless, notification, address_validation, and transit_label
fields to ServiceLevelFeaturesType and ServiceLevelFeaturesInput.
Admin test_update_rate_sheet_accepts_all_service_level_feature_flags
queries these fields; they landed in shipping-platform but the Wave 0
sync kept upstream's ServiceLevelFeaturesType (which lacked them)
per the conflict matrix default of "keep upstream". The test was
picked up from SP's side of the merge, so GraphQL introspection
rejected the query ("Cannot query field 'labelless' on type
'ServiceLevelFeaturesType'"). Ship the four fields so the type
and the test line up.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Sub-PR 4 of the 2026.5.0 umbrella (see
PRDs/RELEASE_2026_5_PLATFORM_UPGRADE.md, tracks #1065). Consolidates the three OSS core server packages (modules/core,modules/graph,modules/admin) into a singlemodules/server/package so every new cross-cutting server change touches onepyproject.toml, one-e ./modules/serverentry, and one install layer — eliminating the recurring class of "forgot to register a module" bugs captured in JTL issue #431.All import paths (
karrio.server.core.*,karrio.server.graph.*,karrio.server.admin.*) continue to resolve identically. Auto-discovery (pkgutil.iter_modules()ingraph/schema.py+admin/schema.py) was already in place on the base branch and is preserved untouched.Not in scope (per PRD D1):
ee/insiders/modules/admin,modules/manager|orders|events|documents|data|pricing|proxy|huey|sdk|soap, connectors — all untouched.Changes
Refactor
refactor(server): scaffold modules/server/ package…modules/server/{pyproject.toml, README.md, MANIFEST.in}; distribution namekarrio_server_modules(avoids collision with existingkarrio_serveratapps/api/pyproject.toml— judgment call, documented below)refactor(server): move modules/core -> modules/servergit mvpreserves history. Scope = everything undermodules/core/karrio/server/(not justcore/— the package also ownedconf.py,openapi.py,samples.py,filters/,iam/,providers/,serializers/,tracing/,user/)refactor(server): move modules/graph -> modules/serversettings/graph.pyrefactor(server): move modules/admin -> modules/serversettings/admin.pyrefactor(server): convert modules/{core,graph,admin}/ to shim packagesdependencies = ["karrio_server_modules"]; READMEs marked as shims; obsoleteMANIFEST.indeletedrefactor(server): update requirements + build/version scripts for modules/server-e ./modules/serverahead of the three shim entries (PRD E1/E2 — shims kept for 1 release cycle for pinned installs); alsosource.requirements{,.insiders,.platform}.txt,bin/build-and-release-packages,bin/update-package-versionsrefactor(server): adjust ruff + coverage paths for modules/server layoutruff.tomlper-file-ignores +.coveragercreport.includepoint at new pathsFile count totals
git mvmodules/server/MANIFEST.inremoved from shims (content absorbed intomodules/server/MANIFEST.in)Why
-e ./modules/serverentry replaces three.feat/2026.5-huey-http), which places the broker view undermodules/server/karrio/server/tasks/.Judgment calls / deviations from the prompt
karrio_server_modules, notkarrio_server. The PRD narrative reads "publishkarrio-serveras the new name" butapps/api/pyproject.tomlalready claims the namekarrio_server. Renamingapps/apiis an out-of-scope rename. Usingkarrio_server_modulesavoids the collision; external import paths are unchanged.ruff.tomlper-file-ignores reference the source paths, the moves would have surfaced 171 pre-existing (now un-ignored) violations. Adjusting the ignores to the new paths (commit 7) preserves the previous lint floor rather than actually fixing 171 historical style nits.docker/{api,insiders}/Dockerfilealready doCOPY . /temp/app/(not explicit per-module copies). Pip resolves the new-e ./modules/serverline viarequirements.build.txtinside the existing COPY, so nothing to edit.bin/run-server-testsuntouched. Prompt said "match existing idiom." Current idiom useskarrio test karrio.server.core.tests karrio.server.graph.tests karrio.server.admin.tests— those labels still resolve correctly because the import paths are unchanged.feat/2026.5, commits7767d84e5 fix(providers): drop orphaned Secret-model migrations(upstream) and mygit mvof the same files conflicted. Accepted the deletion — those migrations were orphaned and intentionally removed onfeat/2026.5.Verification
Run from repo root inside the activated env.
python -c "import karrio.server.core, karrio.server.graph, karrio.server.admin; print('OK')"OK— all three resolve frommodules/server/karrio/server/{core,graph,admin}/karrio check --tag database --fail-level ERRORSystem check identified no issues (0 silenced)../bin/run-server-tests --exclude-insidersFound 453 test(s)./All server tests completed successfully/ exit 0./bin/run-sdk-testsOK; pre-existing failures for nonexistent providers (asendia,parcelone,mydhl) are unrelatedruff check modules/server modules/core modules/graph modules/adminAll checks passed!pip install -e ./modules/serverSuccessfully installed karrio_server_modules-2026.1.29; existing shim installs of `karrio_server_coregrep -E "^-e \./modules/(core|graph|admin)$" requirements.{build,server.dev}.txtgrep -E "^-e \./modules/server$" requirements.{build,server.dev}.txtMigration / rollback (per PRD § Migration & Rollback)
karrio-server-core|graph|admincontinue to install from PyPI; the2026.5.xrelease will publish them as dependency-only shims per PRD E2.References
PRDs/RELEASE_2026_5_PLATFORM_UPGRADE.md§§ Workstream Breakdown (sub-PR 4), Module consolidation (core-only), Key Architecture Decisions D1–D3, Edge Cases E1/E2/E8, Migration & Rollback.claude/rules/extension-patterns.md§ "Namespace Packages — NEVER Add__init__.pyto Shared Paths"