Skip to content

Re-enable the snapshot acceptance test block once Pipes snapshot creates stop timing out #1106

Description

@kaidaguerre

The snapshot block has been removed from the acceptance-test matrix in
.github/workflows/11-test-acceptance.yaml because publishing a snapshot to
turbot-ops/clitesting times out. This issue tracks putting it back.

Why it was disabled

tests/acceptance/test_files/snapshot.bats publishes three snapshots to
turbot-ops/clitesting (one per output format). Creates against that workspace
take longer than the API gateway's 60s limit, so the publish returns
504 Gateway Timeout and the test fails.

Measured 2026-08-17, three consecutive POST requests to
/api/v0/org/turbot-ops/workspace/clitesting/snapshot with a 23.6KB payload:

attempt status time
1 504 60.458s
2 504 60.444s
3 504 60.443s

All three are the gateway's ceiling, not a measurement of how long the create
takes — the real duration is only known to be above 60s.

Reads, deletes and workspace lookups against the same workspace all answer in
under a second. Only creates are slow.

Why this leaks snapshots

The server completes the insert even when the gateway has already given up. Two
of the three probe requests above landed as available rows in the workspace
while the client received a 504 and no snapshot ID. Nothing client-side can
clean those up, because the ID is what a delete needs and it was never returned.

That is a second, independent leak from the one where a failing assertion skips
an inline delete. #1099 fixes that first path by moving the delete into
teardown(), and is still worth landing — but it cannot help here, because the
teardown has no ID either.

What the disabled tests actually covered

The three tests assert CSV, JSON and table output formatting in snapshot mode.
The upload is incidental to exercising --snapshot — steampipe's copy of this
file carries a comment saying outright that these are "not snapshot
creation/upload" tests, and the powerpipe tests have the same shape. So the
coverage lost is output formatting, not the publish path.

What needs to happen before re-enabling

  1. Snapshot creates against the CI workspace need to return under 60s. The
    workspace holds ~2,018 visible snapshots. An earlier theory that create
    latency scales with row count now looks doubtful: a large number of rows
    were deleted a week before the measurements above and the latency did not
    move.
  2. A scheduled cleanup for orphaned snapshots, since a timed-out create leaves
    a row no test can identify.

Once creates are reliably fast, restore the - "snapshot" entry to the
test_block matrix.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions