Skip to content

test: fix @generation e2e (model → amazon.nova-lite-v1:0) - #3500

Closed
sarayev wants to merge 4 commits into
mainfrom
test/generation-conversation-e2e-fixes
Closed

test: fix @generation e2e (model → amazon.nova-lite-v1:0)#3500
sarayev wants to merge 4 commits into
mainfrom
test/generation-conversation-e2e-fixes

Conversation

@sarayev

@sarayev sarayev commented Jun 26, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Description

Fixes the @generation e2e test, which was failing because its directives referenced anthropic.claude-3-haiku-20240307-v1:0 — a model that is no longer reliably available for these test accounts, causing generation requests to error.

Model swap

All three @generation directives in schema-generation.graphql (summarize, generateRecipe, solveEquation) now use amazon.nova-lite-v1:0, which supports the structured/tool-use output the generation route requires.

Generation test guards

generation.test.ts now asserts errors is undefined and the generated recipe is not null before reading fields, so a model/route failure surfaces as a clear assertion rather than a downstream undefined access.

E2E status

The generation e2e passed in two CodeBuild batches:

  • d7085183
  • 441e2caa

These runs confirm the change both fixes the @generation test and introduces no regression.

Note: conversation e2e changes are out of scope for this PR and are owned separately (#3477). This branch was reduced to generation-only.

How did you test this change?

E2E runs (batches above) — generation suite SUCCEEDED.

Checklist

  • PR description included
  • yarn tests passing
  • Unit Tests added

@sarayev sarayev mentioned this pull request Jun 26, 2026
Closed
7 tasks
@sarayev
test: fix generation/conversation e2e (model id -> mistral plain FM, …
d3e8617 
…error/null guards)

Two Gen2-cdk e2e suites (generation, conversation) were failing for
reasons unrelated to product code. This change fixes the test fixtures
and adds fail-fast hygiene guards. Test-side only; no product behavior
changes.

Model ids
  The fixtures invoked anthropic.claude-3-haiku-20240307-v1:0, a legacy
  Bedrock model that has been auto-revoked in the e2e account; invoking
  it returned ResourceNotFoundException, AppSync resolved null, and the
  tests crashed on a null-deref.

  - generation: the @generation transformer's AppSync->Bedrock IAM role
    grants bedrock:InvokeModel only on the foundation-model ARN, so a
    cross-region inference profile fails with AccessDenied. The
    generation fixture is switched to the active on-demand foundation
    model mistral.mistral-large-2407-v1:0, which the e2e account can
    invoke under the existing IAM and which satisfies the basic
    text/scalar assertions.

  - conversation: routes via @aws-amplify/ai-constructs, which handles
    inference-profile IAM, so the conversation fixture uses the active
    inference profile us.anthropic.claude-haiku-4-5-20251001-v1:0.

Fail-fast guards
  generation.test.ts: assert the GraphQL response has no errors and that
  the generated recipe is not null before dereferencing, so a
  Bedrock/GraphQL error is printed instead of a TypeError.

  conversation.test.ts: replace the ineffective toBeDefined() check on
  the stream part with not.toBeNull(), guard the .length read when the
  part is null, and add a max-iteration / overall-timeout guard inside
  the subscription for-await loop so a non-streaming assistant fails
  fast with a descriptive message instead of cascading into the jest
  timeout.

---
Prompt: Split the mixed e2e/CI branch into a clean draft PR off main
containing only the generation/conversation e2e test fixes: switch the
dead legacy claude-3-haiku model id to a working model (generation ->
mistral plain on-demand FM, conversation -> working inference profile)
and add the error/null fail-fast guards. tsc the construct-tests
package and commit (no --no-verify).
@sarayev
sarayev force-pushed the test/generation-conversation-e2e-fixes branch from 67b24f3 to d3e8617 Compare June 26, 2026 09:20
@sarayev sarayev mentioned this pull request Jun 26, 2026
Closed
7 tasks
@sarayev
test(amplify-graphql-api-construct-tests): fix conversation stream `p…
84331e7 
…` assertions for haiku-4-5

The conversation happy-path e2e asserted every streamed
ConversationMessageStreamPart had a non-null `p` and aggregated `p`
into a non-empty string. That premise is wrong: `p` is optional stream
padding, not response text. The ai-constructs Bedrock converse adapter
only sets `p` when the underlying Bedrock contentBlockDelta carries a
string padding field, and never sets it on control frames
(contentBlockStop, the final stopReason chunk). claude-haiku-4-5 emits
no `p` padding at all, so the subscription delivers `p: null` on every
part, failing `expect(streamPart.p).not.toBeNull()` on the first frame
and cascading the other cases into timeouts.

Verified directly against
us.anthropic.claude-haiku-4-5-20251001-v1:0 via Bedrock ConverseStream:
the model returns the full pirate response (stopReason end_turn) with
text delivered through contentBlockDelta/contentBlockText and no `p`
padding on any frame. The assistant text is mapped correctly, so this
is a test-assertion issue, not a resolver bug.

Drop the contradictory not-null assertion, keep the `p` length check
when padding is present, and assert the aggregate streamed
contentBlockText is non-empty so the test still meaningfully verifies a
real streaming response was received.

---
Prompt: Gen2 PR #3500 conversation e2e shard failed at
conversation.test.ts:136 expect(streamPart.p).not.toBeNull() with model
us.anthropic.claude-haiku-4-5. Determine whether haiku-4-5 returns the
pirate text interleaved with null-p control frames (assertion is wrong,
fix it properly) or whether no assistant text arrives (real resolver
bug, report it, do not loosen the test). Gather evidence from the CI log
and the stream-part type/resolver, then fix or report. Commit to the
branch with a conventional message, no --no-verify, follow
CODING_GUIDELINES.md, and push.
Simone319 pushed a commit that referenced this pull request Jun 28, 2026
@sarayev @Simone319
test(amplify-graphql-api-construct-tests): fix conversation stream `p…
22510c0 
…` assertions for haiku-4-5

The conversation happy-path e2e asserted every streamed
ConversationMessageStreamPart had a non-null `p` and aggregated `p`
into a non-empty string. That premise is wrong: `p` is optional stream
padding, not response text. The ai-constructs Bedrock converse adapter
only sets `p` when the underlying Bedrock contentBlockDelta carries a
string padding field, and never sets it on control frames
(contentBlockStop, the final stopReason chunk). claude-haiku-4-5 emits
no `p` padding at all, so the subscription delivers `p: null` on every
part, failing `expect(streamPart.p).not.toBeNull()` on the first frame
and cascading the other cases into timeouts.

Verified directly against
us.anthropic.claude-haiku-4-5-20251001-v1:0 via Bedrock ConverseStream:
the model returns the full pirate response (stopReason end_turn) with
text delivered through contentBlockDelta/contentBlockText and no `p`
padding on any frame. The assistant text is mapped correctly, so this
is a test-assertion issue, not a resolver bug.

Drop the contradictory not-null assertion, keep the `p` length check
when padding is present, and assert the aggregate streamed
contentBlockText is non-empty so the test still meaningfully verifies a
real streaming response was received.

---
Prompt: Gen2 PR #3500 conversation e2e shard failed at
conversation.test.ts:136 expect(streamPart.p).not.toBeNull() with model
us.anthropic.claude-haiku-4-5. Determine whether haiku-4-5 returns the
pirate text interleaved with null-p control frames (assertion is wrong,
fix it properly) or whether no assistant text arrives (real resolver
bug, report it, do not loosen the test). Gather evidence from the CI log
and the stream-part type/resolver, then fix or report. Commit to the
branch with a conventional message, no --no-verify, follow
CODING_GUIDELINES.md, and push.

(cherry picked from commit 84331e7)
@sarayev
test(amplify-graphql-api-construct-tests): scope pr to generation-only
dfc4a1b 
Revert the conversation e2e changes (conversation.test.ts and
schema-conversation.graphql) back to their origin/main state so this
PR contains only the @generation e2e fix. The conversation changes are
owned by a separate PR and are removed here to avoid overlap.

The remaining diff swaps the @generation aiModel to
amazon.nova-lite-v1:0 and adds error/null guards to the generation
test, which has passed two e2e runs.

---
Prompt: Reduce PR #3500 to generation-only: revert the conversation-related changes back to origin/main, keep only the generation (nova-lite) changes, commit and push, and update the PR to be generation-only while keeping it draft.
@sarayev sarayev changed the title test: fix generation/conversation e2e (model id -> mistral plain FM, error/null guards) test: fix @generation e2e (model → amazon.nova-lite-v1:0) Jun 29, 2026
@sarayev
sarayev marked this pull request as ready for review June 29, 2026 13:43
@sarayev
sarayev requested a review from a team as a code owner June 29, 2026 13:43
@sarayev sarayev closed this Jun 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@soberm soberm soberm approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@sarayev @soberm