ci: split e2e into two self-contained waved CodeBuild batches - #3493
Closed
sarayev wants to merge 6 commits into
Closed
ci: split e2e into two self-contained waved CodeBuild batches#3493sarayev wants to merge 6 commits into
sarayev wants to merge 6 commits into
Conversation
Two e2e/CI reliability changes. jstreemap resolution: jstreemap 1.29.1-1.29.3 ship a broken UMD bundle that references a bare `self`, throwing `ReferenceError: self is not defined` at require time on Node. A root `resolutions` entry pins jstreemap to the last known-good 1.28.2 to keep the broken bundle out of this workspace. jstreemap is not currently resolved anywhere in yarn.lock, so this is a defensive guard and a no-op on the lockfile; its only effect is adding a jstreemap stanza to dependency_licenses.txt (regenerated here so verify_dependency_licenses_extract stays green). It does not affect the e2e CLI, which is installed separately via `npm install -g @aws-amplify/cli-internal` from the local Verdaccio registry and resolves its own transitive dependencies. e2e wave-staggering: every e2e shard previously depended only on `publish_to_local_registry`, so all 183 shards became runnable at once and overran the CodeBuild project's concurrent-build limit. Shards are now chained into a sliding window of E2E_WAVE_SIZE (90): the first 90 shards keep their upstream dependency and each later shard i depends on shard i-90, capping concurrency at 90 while preserving the deterministic shard ordering. Trade-off: CodeBuild `depend-on` starts a successor only after its predecessor SUCCEEDS, so a failed shard skips the shard 90 positions later in the chain (this matches the proven waving approach in amplify-cli). Because `depend-on` also requires a non-empty primary artifact from the predecessor, graphql_e2e_tests.yml (which had no artifacts block) now writes a small marker artifact. Testing: regenerated codebuild_specs/e2e_workflow.yml and validated the dependency graph (jobs 0-89 depend on publish_to_local_registry; job i depends on job i-90 for i>=90). Confirmed generator determinism, prettier, tsc --noEmit, and eslint (no new findings vs. baseline).
Split the single 183-shard e2e batch into two independent, self-contained CodeBuild batches (api+gql ~78, cdk ~105), each with its own prep/build/verify chain, cleanup, and an index-offset wave window (SPLIT_E2E_WAVE_SIZE=95) capping in-flight builds under the orchestrator fault ceiling. Adds a reconciliation self-check, cloudE2ESplit trigger, and wait-for-all-codebuild-split aggregator. Legacy single-batch path retained.
sarayev
changed the title
The jstreemap resolution does not affect category-api e2e (which installs @aws-amplify/cli-internal out-of-band via npm global, bypassing yarn resolutions). Removing it leaves this PR scoped purely to the e2e two-batch split. --- Prompt: Remove all jstreemap-related changes from the fix/pin-jstreemap-and-e2e-waving branch so the PR #3493 diff contains none of them, without disturbing the e2e split work. The jstreemap pin is a no-op for category-api because its e2e installs @aws-amplify/cli-internal out-of-band via npm global, so yarn resolutions/pins never reach it. Surgically remove only the jstreemap parts from package.json and dependency_licenses.txt, leaving all e2e-split changes intact, then commit and push.
Simone319
approved these changes
Jun 25, 2026
Skip deleting any Amplify app named DoNotDeleteAppToBypassGen1Deprecation in cleanup-e2e-resources getAmplifyApps, so the placeholder app (with a backend environment) persists per account/region. This keeps isExistingGen1Customer true and allows amplify init to create Gen1 apps in e2e.
sa-east-1 has limited Amplify service coverage (RDS/OpenSearch) and was producing e2e failures; remove it from the round-robin region assignment in split-e2e-tests.
…er shard The Gen1 EOL gate blocks `amplify init` unless the account+region already has an Amplify app with a backend environment. We seed a placeholder app `DoNotDeleteAppToBypassGen1Deprecation`, but CI cleanup keeps deleting it, which breaks subsequent e2e runs. Add an idempotent, self-healing `ensureGen1PlaceholderApp(region)` to amplify-e2e-core. It lists apps in the shard's region (CLI_REGION), and if the placeholder app is missing it creates it; it also ensures a backend environment named `test` exists. The function is idempotent on both the app name and the backend env name, and swallows/logs all errors so it never fails a run for an already-seeded account. Wire it into a jest `globalSetup` hook so it runs once per shard before any test (and therefore before any `amplify init`). The hook is added to both e2e packages that run `amplify init`: amplify-e2e-tests and amplify-graphql-api-construct-tests. Tested with `tsc` on amplify-e2e-core (clean compile); cloud e2e not run. --- Prompt: Add a step at the START of each category-api e2e run that ensures the placeholder app (+ backend env 'test') exists in that shard's account+region — idempotent and self-healing, mirroring amplify-cli's ensureGen1PlaceholderApp but with app name DoNotDeleteAppToBypassGen1Deprecation and env name test. Investigate the right hook, implement on fix/pin-jstreemap-and-e2e-waving, build, commit, and push without marking the PR ready or triggering e2e.
This was referenced Jun 26, 2026
Contributor
Author
sarayev
added a commit
that referenced
this pull request
Jun 26, 2026
## Description Splits the e2e CI infrastructure out of the mixed draft #3493 into a clean, self-contained change off `main`. This PR contains **only** the CI/e2e-infra file-set; the generation/conversation test fixes ship separately, and the generation IAM change is #3495. ### Two self-contained waved CodeBuild batches The single e2e batch is split into two independent batches — `api+gql` (~78 shards) and `cdk` (~108 shards) — each with its own prep/build/verify chain and cleanup. Every shard is chained into an index-offset wave window (`SPLIT_E2E_WAVE_SIZE`) so in-flight builds stay under the orchestrator's concurrent-build ceiling. A `cloudE2ESplit` trigger and a `wait-for-all-codebuild-split` aggregator drive and join the two batches. The legacy single-batch path is retained. ### Reconciliation self-check The generator now asserts that the split total equals the combined total with no missing/extra/duplicate/overlapping shards. Current run: **PASS (186 == 186)**. ### Region pool `sa-east-1` is removed from the round-robin region assignment — its limited Amplify service coverage (RDS/OpenSearch) was producing e2e failures. ### CDK OOM: shard-split via RUN_SOLO The five heavy CDK construct-deploy suites (utils, log-config, gsi-projection-type, ddb-iam-access, data-construct) were bin-packed into one shard and exhausted the worker heap (SIGKILL/OOM). They are added to the existing `RUN_SOLO` list so each deploys in its own shard. This grows the cdk batch to 108 and the combined workflow to 186; no test is dropped. ### Gen1 placeholder app: self-healing + cleanup-protect An idempotent, self-healing `ensureGen1PlaceholderApp(region)` is added to `amplify-e2e-core` and wired into a jest `globalSetup` hook in the two e2e packages that run `amplify init`, so the Gen1 deprecation-bypass placeholder app (and its `test` backend env) is recreated per shard before any test runs. `cleanup-e2e-resources` is updated to skip deleting `DoNotDeleteAppToBypassGen1Deprecation` so it persists across runs. ## How did you test this change? - Regenerated specs with `yarn ts-node ./scripts/split-e2e-tests.ts prod`; output is deterministic (no diff on re-run). - Reconciliation self-check: **PASS**, split total 186 == combined 186, no missing/extra/duplicate/overlap; max in-flight per batch within cap; no sa-east-1 assignments. ## Notes Draft. Supersedes the CI portion of #3493. Companion PRs: generation/conversation e2e test fixes (separate) and #3495 (generation IAM).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of changes
A single CodeBuild batch fanning out all ~183 e2e shards overwhelms the batch
orchestrator, which faults materially once more than ~100 builds are in flight.
This PR adds the ability to fire the suite as two independent, self-contained
batches, each waved to cap its own in-flight count. The legacy single-batch
path (
yarn cloud-e2e) is retained unchanged; the split path is additive.Summary:
e2e_workflow_api_gql.yml(amplify-e2e-tests +graphql-transformers-e2e-tests, ~78 shards) and
e2e_workflow_cdk.yml(amplify-graphql-api-construct-tests, ~105 shards). Each carries the full
prep chain (
build_linux,build_windows,test,verify_*,publish_to_local_registry) and its owncleanup_e2e_resources, so they runconcurrently and independently.
SPLIT_E2E_WAVE_SIZE = 95(first 95 depend onpublish_to_local_registry,each later shard
idepends oni - 95), capping in-flight shards per batchat 95. The combined spec keeps its existing
E2E_WAVE_SIZE = 90.yarn split-e2e-testsregenerates all threespecs and asserts the two split batches exactly cover the combined set
(78 + 105 = 183, no missing/extra/overlap) and carry the required non-test
jobs; throws on failure.
cloud-e2e-splitfires both batches and prints both BatchIDs;
triggerProjectBatchWithBuildspecstarts a batch with--buildspec-override;wait-for-all-codebuild-split.tspolls both batchesto terminal state and reports aggregate pass/fail;
.agent-docs/E2E_TESTING.mddocuments the split mode.
CDK / CloudFormation Parameters Changed
None.
Issue #, if available
N/A — operational/CI change.
Description of how you validated changes
yarn split-e2e-tests; generator isdeterministic.
each carries the required non-test jobs).
author after review.
Checklist
yarn testpassesBy submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.