fix: submodule support — reinstate clone, fix pull, add hasSubmodules flag

[rpapani]

Summary

Full git-submodule support for both standard and BYOG Cloud Manager repos, plus the data-access schema fields downstream consumers need to drive the runtime.

The branch sits on top of main and contains five logical commits, each independently reviewable:

1. fix: clone functionality to include submodules (#1413) — cherry-pick of the original submodule-clone fix that was reverted in Revert "fix: clone functionality to include submodules (#1413)" #1549 because the Lambda git-layer was missing the submodule helper binaries. With spacecat-git-layer:4 now carrying git-submodule and git-submodule--helper (see adobe/spacecat-infrastructure#470), this can safely go back in. The commit also carries the per-operation git timeout bump that landed alongside the original fix: GIT_OPERATION_TIMEOUT_MS defaults to 600_000 ms (10 min, up from 120s) and is now overridable via env var — recursive clones traverse multiple submodule repos and the old 2-minute budget was too tight.

2. fix: propagate --recurse-submodules to incremental pull — the original PR patched clone but not pull. The incremental-update path uses pull and was still skipping submodules.

3. feat: add hasSubmodules field to site.code — initial schema work. Superseded in shape by commits 4 and 5 below; included for review continuity.

4. refactor: replace code.hasSubmodules with code.metadata.submodules — moves submodule metadata into a nested metadata.submodules: { external, urls } object so additional fields can be added without further schema churn.

5. refactor: precomputed submoduleMap for BYOG submodule clones — the substantive change. Replaces the previous runtime name-lookup logic with a flat, onboarding-populated submoduleMap array of { path, url } entries. The cm-client iterates it mechanically and writes each entry into .git/config before running submodule update. All name disambiguation, URL classification, and collision handling is done at onboarding; the runtime is purely mechanical.

What's new in commit 5 (the meat of the change)

Submodule pipeline split by parent repo type

• Standard parent: git clone --recurse-submodules and pull --recurse-submodules work natively. .gitmodules URLs resolve to the same host as the parent (git.cloudmanager.adobe.com/{orgName}/...); the existing org-scoped Basic-auth extraheader covers every submodule transparently. No additional configuration needed.
• BYOG parent: --no-recurse-submodules at clone time, followed by a second pass driven by submoduleMap. The CM repo service proxy can't serve relative or SSH .gitmodules URLs (they resolve to name-based paths the proxy rejects), so we rewrite each submodule's URL in .git/config to a CM-reachable form before running submodule update. .gitmodules itself is never modified — the working tree stays clean.

submoduleMap data contract

"site.code.metadata.submodules.submoduleMap": [
  { "path": "sub-byog",     "url": "https://cm-repo.adobe.io/api/program/{pid}/repository/{numericId}.git" },
  { "path": "sub-standard", "url": "https://git.cloudmanager.adobe.com/{orgName}/{repoName}/" }
]

Each entry says: "for the submodule at <path> in .gitmodules, write <url> into .git/config submodule.<path>.url." The runtime decides which auth scope to apply by parsing each url's host — proxy URLs get the BYOG triple (Bearer + x-api-key + x-gw-ims-org-id), git.cloudmanager.adobe.com URLs get a Basic header from CM_STANDARD_REPO_CREDENTIALS[programId] scoped to {origin}/{orgName}/. Both scopes coexist on a single git submodule update invocation.

Why this is one map driven by URLs (not branched by repo type at runtime)

By baking the URL form into the map at onboarding, the runtime has no need to know what type the underlying repo is — the URL is the source of truth for both routing AND auth-scope selection. This handles every BYOG configuration we've encountered:

• Pure-BYOG (parent and all submodules tunnel through the proxy): single Bearer scope.
• Mixed BYOG + standard (BYOG parent with standard-typed submodules): both scopes attached to the same submodule update invocation; git applies the right headers per-URL by prefix matching.

Other additions

• New exported constant GIT_CLOUD_MANAGER_HOST (= git.cloudmanager.adobe.com) so the standard-host literal is centralised.
• New helper #buildSubmoduleAuthArgs that walks the map once, collects unique URL hosts, and emits one -c http.<scope>.extraheader=... per scope (deduplicated).
• git submodule update --force --recursive is used unconditionally on the BYOG path. --force resets the submodule's working tree to whatever HEAD points at after fetch — required when the parent's pinned commit SHAs aren't reachable in the submodule's current commit graph (a real configuration we've seen, where submodule update would otherwise leave the working tree empty).
• cmProgramRepos (the previous, runtime-fuzzy-matched name→id map field) is removed throughout. It was unreleased; no production data exists in that shape.

Files changed

File	Change
packages/spacecat-shared-cloud-manager-client/src/index.js	New GIT_CLOUD_MANAGER_HOST constant; clone() / pull() accept submoduleMap; new #buildSubmoduleAuthArgs; #resolveByogSubmodules rewritten to iterate submoduleMap and run submodule update --force --recursive
packages/spacecat-shared-cloud-manager-client/test/cloud-manager-client.test.js	Test rewrite for submoduleMap; new dual-scope auth test; defensive tests for invalid entries and unparseable URLs
packages/spacecat-shared-cloud-manager-client/README.md	New "Submodule handling" and "Known limitations" sections
packages/spacecat-shared-data-access/src/models/site/index.d.ts	SubmodulesMetadata.submoduleMap?: Array<{path, url}> with full JSDoc on URL forms and runtime auth-scope derivation
packages/spacecat-shared-data-access/src/models/site/site.schema.js	JSDoc on the code attribute documents the new field
packages/spacecat-shared-data-access/test/unit/models/site/site.model.test.js	Tests for metadata.submodules.submoduleMap storage + cross-import preservation

Known limitations (tracked in cm-client README.md)

The submodule pipeline doesn't cover every BYOG configuration. The cases below are documented as out-of-scope for this PR and would each land as a separate follow-up:

• submoduleMap not yet populated → parent clone preserved, warning logged, submodule fetches fail (expected until onboarding populates the map).
• Out-of-program submodules (.gitmodules references repos not onboarded to CM at all).
• Broken BYOG connections (repo status: ready in API but proxy 404s).
• Stale parent gitlinks (handled today via --force; documented trade-off is dirty-pointer state in git status).
• Recursive submodule depth > 1 (each level needs its own map).
• Cross-org standard submodules (would need a per-{programId, orgName} cred shape).
• Submodules on third-party hosts not behind the CM proxy.
• submoduleMap drift between imports.

Test plan

• cm-client unit tests pass — 80 tests, 100% coverage
• data-access unit tests pass — 1920 tests
• Manual end-to-end clone with --recurse-submodules for a standard parent — verified
• Manual end-to-end clone with --no-recurse-submodules + submoduleMap rewrite for a BYOG parent (single-scope and dual-scope cases) using the updated git Lambda layer :4 — verified end-to-end
• Downstream consumer verification via adobe/spacecat-import-worker#724

Related

• Supersedes revert Revert "fix: clone functionality to include submodules (#1413)" #1549
• Unblocks SITES-43089 autofix work
• Downstream consumer PR: adobe/spacecat-import-worker#724 — picks up this branch via gist tarball for pre-release testing, then consumes metadata.submodules.submoduleMap
• Lambda layer PR: adobe/spacecat-infrastructure#470 — bundles git-submodule + git-submodule--helper into spacecat-git-layer:4

🤖 Generated with Claude Code

[github-actions]

This PR will trigger a minor release when merged.