Skip to content

Commit 31f022a

Browse files
authored
Add add-wizard tuistory integration suite in dedicated CI job (#27205)
1 parent 62b8dab commit 31f022a

4 files changed

Lines changed: 350 additions & 19 deletions

File tree

.github/workflows/ci.yml

Lines changed: 73 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -75,7 +75,7 @@ jobs:
7575
- name: "CLI Completion & Other" # Remaining catch-all (reduced from original)
7676
packages: "./pkg/cli"
7777
pattern: "" # Catch-all for tests not matched by other CLI patterns
78-
skip_pattern: "^TestCompile[^W]|TestPoutine|TestSafeUpdate|TestMCPInspectPlaywright|TestMCPGateway|TestMCPAdd|TestMCPInspectGitHub|TestMCPServer|TestMCPConfig|TestLogs|TestFirewall|TestNoStopTime|TestLocalWorkflow|TestProgressFlagSignature|TestConnectHTTPMCPServer|TestCompileWorkflows_EmptyMarkdown|TestCompileWithZizmor|TestCompileWithPoutine|TestCompileWithPoutineAndZizmor|^TestAdd|^TestList|^TestUpdate|^TestAudit|^TestInspect|TestDockerBuild|TestDockerImage"
78+
skip_pattern: "^TestCompile[^W]|TestPoutine|TestSafeUpdate|TestMCPInspectPlaywright|TestMCPGateway|TestMCPAdd|TestMCPInspectGitHub|TestMCPServer|TestMCPConfig|TestLogs|TestFirewall|TestNoStopTime|TestLocalWorkflow|TestProgressFlagSignature|TestConnectHTTPMCPServer|TestCompileWorkflows_EmptyMarkdown|TestCompileWithZizmor|TestCompileWithPoutine|TestCompileWithPoutineAndZizmor|^TestAdd|^TestList|^TestUpdate|^TestAudit|^TestInspect|TestDockerBuild|TestDockerImage|TestTuistoryAddWizardIntegration"
7979
- name: "Workflow Compiler"
8080
packages: "./pkg/workflow"
8181
pattern: "TestCompile|TestWorkflow|TestGenerate|TestParse"
@@ -316,6 +316,13 @@ jobs:
316316
- name: Build gh-aw binary
317317
run: make build
318318

319+
- name: Upload gh-aw binary artifact
320+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
321+
with:
322+
name: gh-aw-linux-amd64
323+
path: gh-aw
324+
retention-days: 1
325+
319326
- name: Test update command (dry-run)
320327
env:
321328
GH_TOKEN: ${{ github.token }}
@@ -326,6 +333,71 @@ jobs:
326333
./gh-aw update --verbose
327334
echo "✅ Update command executed successfully" >> $GITHUB_STEP_SUMMARY
328335
336+
integration-add-wizard-tuistory:
337+
name: Integration Add-Wizard (tuistory)
338+
if: ${{ needs.changes.outputs.has_changes == 'true' }}
339+
needs:
340+
- changes
341+
- update
342+
runs-on: ubuntu-latest
343+
timeout-minutes: 20
344+
permissions:
345+
contents: read
346+
env:
347+
GH_AW_BINARY_DIR: /tmp/gh-aw-binary
348+
GH_AW_INTEGRATION_BINARY: /tmp/gh-aw-binary/gh-aw
349+
concurrency:
350+
group: ci-${{ github.ref }}-integration-add-wizard-tuistory
351+
cancel-in-progress: true
352+
steps:
353+
- name: Checkout code
354+
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
355+
356+
- name: Set up Go
357+
id: setup-go
358+
uses: actions/setup-go@4dc6199c7b1a012772edbd06daecab0f50c9053c # v6
359+
with:
360+
go-version-file: go.mod
361+
cache: true
362+
363+
- name: Set up Node.js
364+
uses: actions/setup-node@395ad3262231945c25e8478fd5baf05154b1d79f # v6
365+
with:
366+
node-version: "24"
367+
368+
- name: Download dependencies
369+
run: go mod download
370+
371+
- name: Verify dependencies
372+
run: go mod verify
373+
374+
- name: Download gh-aw binary artifact
375+
uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16 # v4
376+
with:
377+
name: gh-aw-linux-amd64
378+
path: ${{ env.GH_AW_BINARY_DIR }}
379+
380+
- name: Prepare gh-aw binary
381+
run: chmod +x "${GH_AW_INTEGRATION_BINARY}"
382+
383+
- name: Run add-wizard tuistory integration tests
384+
env:
385+
GH_TOKEN: ${{ github.token }}
386+
run: |
387+
set -o pipefail
388+
go test -v -parallel=1 -timeout=15m -tags 'integration' -json \
389+
-run 'TestTuistoryAddWizardIntegration' \
390+
./pkg/cli/ \
391+
| tee test-result-integration-add-wizard-tuistory.json
392+
393+
- name: Upload test results
394+
if: always()
395+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
396+
with:
397+
name: test-result-integration-add-wizard-tuistory
398+
path: test-result-integration-add-wizard-tuistory.json
399+
retention-days: 14
400+
329401
js-integration-live-api:
330402
if: ${{ needs.changes.outputs.has_changes == 'true' && (github.ref == 'refs/heads/main') }}
331403
needs:
Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
# ADR-27205: Integration Testing Interactive TUI Workflows via Tuistory
2+
3+
**Date**: 2026-04-19
4+
**Status**: Draft
5+
**Deciders**: pelikhan, Copilot
6+
7+
---
8+
9+
## Part 1 — Narrative (Human-Friendly)
10+
11+
### Context
12+
13+
The `add-wizard` command drives an interactive terminal UI (TUI) flow — it prompts the user for repository information and confirmation before applying file changes. Existing integration tests in `pkg/cli/` exercise non-interactive commands by running the compiled `gh-aw` binary in a subprocess; they cannot interact with prompts or validate TUI-driven behavior. As the wizard becomes a user-critical path, there is a need to verify that the interactive flow works end-to-end: prompts appear in the expected order, keyboard input is accepted correctly, and cancellation leaves the repository in a clean state.
14+
15+
### Decision
16+
17+
We will use [tuistory](https://www.npmjs.com/package/tuistory) — a Node.js TUI automation tool — via `npx` to drive the `add-wizard` TUI in integration tests. Tests launch a named tuistory session that wraps the `gh-aw` subprocess in a pseudo-terminal, then send keyboard events and poll for expected text output. This approach treats the TUI as a black box from the user's perspective, validating the full interactive experience without modifying production code to add test hooks.
18+
19+
### Alternatives Considered
20+
21+
#### Alternative 1: Unit-test TUI components with mocked prompts
22+
23+
The prompt library used by `add-wizard` could be wrapped behind an interface, and unit tests could inject a fake implementation that returns predefined responses. This approach runs fast and in-process, but it does not validate that the production prompt rendering or keyboard-event handling works correctly. A bug in the TUI library integration would not be caught, and the test would diverge from the real user experience over time. [TODO: verify which prompt library is used and whether it exposes a testing interface]
24+
25+
#### Alternative 2: Expect-style scripting with `expect` / `pexpect`
26+
27+
Traditional `expect` or Python `pexpect` scripts can drive interactive CLIs by matching on stdout patterns and sending text. This is a mature and widely understood technique. However, it introduces a Python dependency into a Go project, and `expect` patterns are fragile against minor changes in prompt wording or terminal control codes. Tuistory offers the same interaction model in JavaScript with a higher-level API and named sessions that simplify cleanup.
28+
29+
#### Alternative 3: Extend the CLI with a non-interactive mode for all prompts
30+
31+
The `add-wizard` could accept all inputs as flags (e.g., `--repo`, `--confirm`) so tests can bypass TUI prompts entirely. This would enable simple subprocess-based testing. However, it alters the production CLI surface and may encourage misuse of flags in scripts where the interactive flow is the intended UX. It also leaves the interactive TUI path untested.
32+
33+
### Consequences
34+
35+
#### Positive
36+
- The happy path and cancellation path of the interactive wizard are validated against a real subprocess, catching regressions in prompt ordering, text, and keyboard handling.
37+
- Tests are isolated: each test creates a fresh temporary directory with an initialized git repository, ensuring no cross-test pollution.
38+
- Tuistory's named sessions enable deterministic cleanup via a deferred `close` call, even when the test panics or is cancelled.
39+
- The test gracefully skips when `npx` or tuistory is unavailable, preventing false failures on developer machines without Node.js.
40+
41+
#### Negative
42+
- Integration tests now require Node.js (`npx`) in the CI runner and on developer machines that wish to run them locally.
43+
- Tuistory is an external JavaScript tool installed at test time via `npx -y`; it is not pinned to a specific version, introducing a risk of breaking changes from upstream.
44+
- TUI-based tests are inherently slower and more timing-sensitive than unit tests; `waitForTuistoryText` uses polling with hardcoded timeouts (up to 120 seconds per step).
45+
- Adding a dedicated CI job (`integration-add-wizard-tuistory`) increases overall pipeline time and resource consumption.
46+
47+
#### Neutral
48+
- A shared `gh-aw` binary artifact is introduced in the `update` CI job and downloaded by the new integration job, requiring the `update` job to run before `integration-add-wizard-tuistory`. This changes the CI dependency graph.
49+
- The `GH_AW_INTEGRATION_BINARY` environment variable allows CI to inject a pre-built binary path into `TestMain`, avoiding a redundant build. Developers who do not set this variable get the original local-build fallback.
50+
- `TestTuistoryAddWizardIntegration` is excluded from the CLI catch-all matrix job to prevent duplicate execution.
51+
52+
---
53+
54+
## Part 2 — Normative Specification (RFC 2119)
55+
56+
> The key words **MUST**, **MUST NOT**, **REQUIRED**, **SHALL**, **SHALL NOT**, **SHOULD**, **SHOULD NOT**, **RECOMMENDED**, **MAY**, and **OPTIONAL** in this section are to be interpreted as described in [RFC 2119](https://www.rfc-editor.org/rfc/rfc2119).
57+
58+
### Test Isolation
59+
60+
1. Each integration test **MUST** create its own temporary directory and initialize a fresh git repository within it before launching the wizard under test.
61+
2. Tests **MUST** clean up their temporary directory on exit, using `defer os.RemoveAll(...)` or equivalent, regardless of test outcome.
62+
3. Tests **MUST NOT** depend on or mutate the repository in which the test process is running.
63+
64+
### Tuistory Session Lifecycle
65+
66+
1. Each test **MUST** use a unique session name (e.g., derived from the test name and `time.Now().UnixNano()`) to avoid collisions when tests run in parallel.
67+
2. Tests **MUST** defer a `tuistory -s <session> close` call immediately after a successful launch so the session is always terminated.
68+
3. Tests **SHOULD** skip (not fail) when `npx` or tuistory is unavailable in the environment, using `t.Skip(...)` with an explanatory message.
69+
4. Tests **MUST NOT** assume a specific tuistory version; behavior differences across minor versions **SHOULD** be handled by keeping assertions against stable, version-independent output strings.
70+
71+
### CI Integration
72+
73+
1. The `integration-add-wizard-tuistory` CI job **MUST** depend on the `update` job and download the `gh-aw-linux-amd64` artifact rather than building the binary independently.
74+
2. The CI job **MUST** set `GH_AW_INTEGRATION_BINARY` to the downloaded binary path so `TestMain` uses it without triggering a local build.
75+
3. `TestTuistoryAddWizardIntegration` **MUST** be excluded from any catch-all CLI matrix job via `skip_pattern` to ensure it runs only in its dedicated job.
76+
4. Test results **MUST** be uploaded as a CI artifact using the naming convention `test-result-integration-<suite-name>` with a retention period of at least 14 days.
77+
78+
### Pre-built Binary Injection
79+
80+
1. `TestMain` **MUST** check the `GH_AW_INTEGRATION_BINARY` environment variable before attempting a local build.
81+
2. If the environment variable is set, `TestMain` **MUST** verify the file exists and is accessible; if not, it **MUST** panic with a descriptive error rather than silently falling back to a local build.
82+
3. When the environment variable is absent, `TestMain` **MAY** build the binary locally using `make build`.
83+
84+
### Conformance
85+
86+
An implementation is considered conformant with this ADR if it satisfies all **MUST** and **MUST NOT** requirements above. Failure to meet any **MUST** or **MUST NOT** requirement constitutes non-conformance.
87+
88+
---
89+
90+
*This is a DRAFT ADR generated by the [Design Decision Gate](https://github.com/github/gh-aw/actions/runs/24633266047) workflow. The PR author must review, complete, and finalize this document before the PR can merge.*
Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
//go:build integration
2+
3+
package cli
4+
5+
import (
6+
"fmt"
7+
"os"
8+
"os/exec"
9+
"path/filepath"
10+
"strings"
11+
"testing"
12+
"time"
13+
14+
"github.com/github/gh-aw/pkg/fileutil"
15+
"github.com/stretchr/testify/assert"
16+
"github.com/stretchr/testify/require"
17+
)
18+
19+
type addWizardTuistorySetup struct {
20+
tempDir string
21+
binaryPath string
22+
workflowPath string
23+
}
24+
25+
func setupAddWizardTuistoryTest(t *testing.T) *addWizardTuistorySetup {
26+
t.Helper()
27+
28+
tempDir, err := os.MkdirTemp("", "gh-aw-add-wizard-tuistory-*")
29+
require.NoError(t, err, "Failed to create temp directory")
30+
31+
// Initialize git repository required by add-wizard preconditions.
32+
gitInit := exec.Command("git", "init")
33+
gitInit.Dir = tempDir
34+
output, err := gitInit.CombinedOutput()
35+
require.NoError(t, err, "Failed to initialize git repository: %s", string(output))
36+
37+
gitConfigName := exec.Command("git", "config", "user.name", "Test User")
38+
gitConfigName.Dir = tempDir
39+
_ = gitConfigName.Run()
40+
41+
gitConfigEmail := exec.Command("git", "config", "user.email", "test@example.com")
42+
gitConfigEmail.Dir = tempDir
43+
_ = gitConfigEmail.Run()
44+
45+
binaryPath := filepath.Join(tempDir, "gh-aw")
46+
err = fileutil.CopyFile(globalBinaryPath, binaryPath)
47+
require.NoError(t, err, "Failed to copy gh-aw binary")
48+
49+
err = os.Chmod(binaryPath, 0755)
50+
require.NoError(t, err, "Failed to make gh-aw binary executable")
51+
52+
workflowPath := filepath.Join(tempDir, "local-test-workflow.md")
53+
workflowContent := `---
54+
name: Local Add Wizard Integration
55+
on:
56+
workflow_dispatch:
57+
engine: copilot
58+
---
59+
60+
# Local Add Wizard Integration
61+
62+
This workflow is used by add-wizard tuistory integration tests.
63+
`
64+
err = os.WriteFile(workflowPath, []byte(workflowContent), 0644)
65+
require.NoError(t, err, "Failed to write local workflow fixture")
66+
67+
return &addWizardTuistorySetup{
68+
tempDir: tempDir,
69+
binaryPath: binaryPath,
70+
workflowPath: workflowPath,
71+
}
72+
}
73+
74+
func runTuistory(t *testing.T, args ...string) (string, error) {
75+
t.Helper()
76+
77+
cmd := exec.Command("npx", append([]string{"-y", "tuistory"}, args...)...)
78+
output, err := cmd.CombinedOutput()
79+
return string(output), err
80+
}
81+
82+
func waitForTuistoryText(t *testing.T, sessionName string, text string, timeoutMs int) {
83+
t.Helper()
84+
output, err := runTuistory(t, "-s", sessionName, "wait", text, "--timeout", fmt.Sprintf("%d", timeoutMs))
85+
require.NoError(t, err, "Expected tuistory to find %q. Output: %s", text, output)
86+
}
87+
88+
func TestTuistoryAddWizardIntegration(t *testing.T) {
89+
const launchTimeoutMs = 30000 // 30 seconds
90+
91+
if _, err := exec.LookPath("npx"); err != nil {
92+
t.Skip("npx not available, skipping tuistory add-wizard integration test")
93+
}
94+
95+
versionOutput, err := runTuistory(t, "--version")
96+
if err != nil {
97+
t.Skipf("tuistory is not usable in this environment: %v (%s)", err, versionOutput)
98+
}
99+
100+
setup := setupAddWizardTuistoryTest(t)
101+
defer func() {
102+
_ = os.RemoveAll(setup.tempDir)
103+
}()
104+
105+
sessionName := fmt.Sprintf("gh-aw-add-wizard-%d", time.Now().UnixNano())
106+
command := fmt.Sprintf("%s add-wizard ./%s --engine copilot --skip-secret", setup.binaryPath, filepath.Base(setup.workflowPath))
107+
108+
launchArgs := []string{
109+
"launch", command,
110+
"-s", sessionName,
111+
"--cwd", setup.tempDir,
112+
"--cols", "140",
113+
"--rows", "40",
114+
"--env", "CI=",
115+
"--env", "GO_TEST_MODE=",
116+
"--timeout", fmt.Sprintf("%d", launchTimeoutMs),
117+
}
118+
119+
launchOutput, err := runTuistory(t, launchArgs...)
120+
if err != nil {
121+
t.Skipf("tuistory launch is not usable in this environment: %v (%s)", err, launchOutput)
122+
}
123+
124+
defer func() {
125+
_, _ = runTuistory(t, "-s", sessionName, "close")
126+
}()
127+
128+
// No git remote in the test repository forces add-wizard to prompt for owner/repo.
129+
waitForTuistoryText(t, sessionName, "Enter the target repository (owner/repo):", 120000)
130+
131+
typeOutput, err := runTuistory(t, "-s", sessionName, "type", "github/gh-aw")
132+
require.NoError(t, err, "Failed to type repository slug. Output: %s", typeOutput)
133+
134+
enterOutput, err := runTuistory(t, "-s", sessionName, "press", "enter")
135+
require.NoError(t, err, "Failed to press enter after repository slug. Output: %s", enterOutput)
136+
137+
waitForTuistoryText(t, sessionName, "Do you want to proceed with these changes?", 120000)
138+
139+
cancelOutput, err := runTuistory(t, "-s", sessionName, "press", "ctrl", "c")
140+
require.NoError(t, err, "Failed to send Ctrl+C to add-wizard session. Output: %s", cancelOutput)
141+
142+
// Collect complete session output and assert cancellation occurred before changes were applied.
143+
readOutput, err := runTuistory(t, "-s", sessionName, "read", "--all")
144+
require.NoError(t, err, "Failed to read tuistory output after cancellation")
145+
assert.True(t,
146+
strings.Contains(readOutput, "confirmation failed") || strings.Contains(readOutput, "interrupted"),
147+
"Expected cancellation-related output, got:\n%s",
148+
readOutput,
149+
)
150+
151+
addedWorkflowPath := filepath.Join(setup.tempDir, ".github", "workflows", filepath.Base(setup.workflowPath))
152+
_, statErr := os.Stat(addedWorkflowPath)
153+
assert.ErrorIs(t, statErr, os.ErrNotExist, "Workflow file should not be created when add-wizard is cancelled")
154+
}

0 commit comments

Comments
 (0)