The snapshot block has been removed from the acceptance-test matrix in
.github/workflows/11-test-acceptance.yaml because publishing a snapshot to
turbot-ops/clitesting times out. This issue tracks putting it back.
Why it was disabled
tests/acceptance/test_files/snapshot.bats publishes three snapshots to
turbot-ops/clitesting (one per output format). Creates against that workspace
take longer than the API gateway's 60s limit, so the publish returns
504 Gateway Timeout and the test fails.
Measured 2026-08-17, three consecutive POST requests to
/api/v0/org/turbot-ops/workspace/clitesting/snapshot with a 23.6KB payload:
| attempt |
status |
time |
| 1 |
504 |
60.458s |
| 2 |
504 |
60.444s |
| 3 |
504 |
60.443s |
All three are the gateway's ceiling, not a measurement of how long the create
takes — the real duration is only known to be above 60s.
Reads, deletes and workspace lookups against the same workspace all answer in
under a second. Only creates are slow.
Why this leaks snapshots
The server completes the insert even when the gateway has already given up. Two
of the three probe requests above landed as available rows in the workspace
while the client received a 504 and no snapshot ID. Nothing client-side can
clean those up, because the ID is what a delete needs and it was never returned.
That is a second, independent leak from the one where a failing assertion skips
an inline delete. #1099 fixes that first path by moving the delete into
teardown(), and is still worth landing — but it cannot help here, because the
teardown has no ID either.
What the disabled tests actually covered
The three tests assert CSV, JSON and table output formatting in snapshot mode.
The upload is incidental to exercising --snapshot — steampipe's copy of this
file carries a comment saying outright that these are "not snapshot
creation/upload" tests, and the powerpipe tests have the same shape. So the
coverage lost is output formatting, not the publish path.
What needs to happen before re-enabling
- Snapshot creates against the CI workspace need to return under 60s. The
workspace holds ~2,018 visible snapshots. An earlier theory that create
latency scales with row count now looks doubtful: a large number of rows
were deleted a week before the measurements above and the latency did not
move.
- A scheduled cleanup for orphaned snapshots, since a timed-out create leaves
a row no test can identify.
Once creates are reliably fast, restore the - "snapshot" entry to the
test_block matrix.
The
snapshotblock has been removed from the acceptance-test matrix in.github/workflows/11-test-acceptance.yamlbecause publishing a snapshot toturbot-ops/clitestingtimes out. This issue tracks putting it back.Why it was disabled
tests/acceptance/test_files/snapshot.batspublishes three snapshots toturbot-ops/clitesting(one per output format). Creates against that workspacetake longer than the API gateway's 60s limit, so the publish returns
504 Gateway Timeoutand the test fails.Measured 2026-08-17, three consecutive
POSTrequests to/api/v0/org/turbot-ops/workspace/clitesting/snapshotwith a 23.6KB payload:All three are the gateway's ceiling, not a measurement of how long the create
takes — the real duration is only known to be above 60s.
Reads, deletes and workspace lookups against the same workspace all answer in
under a second. Only creates are slow.
Why this leaks snapshots
The server completes the insert even when the gateway has already given up. Two
of the three probe requests above landed as
availablerows in the workspacewhile the client received a 504 and no snapshot ID. Nothing client-side can
clean those up, because the ID is what a delete needs and it was never returned.
That is a second, independent leak from the one where a failing assertion skips
an inline delete. #1099 fixes that first path by moving the delete into
teardown(), and is still worth landing — but it cannot help here, because theteardown has no ID either.
What the disabled tests actually covered
The three tests assert CSV, JSON and table output formatting in snapshot mode.
The upload is incidental to exercising
--snapshot— steampipe's copy of thisfile carries a comment saying outright that these are "not snapshot
creation/upload" tests, and the powerpipe tests have the same shape. So the
coverage lost is output formatting, not the publish path.
What needs to happen before re-enabling
workspace holds ~2,018 visible snapshots. An earlier theory that create
latency scales with row count now looks doubtful: a large number of rows
were deleted a week before the measurements above and the latency did not
move.
a row no test can identify.
Once creates are reliably fast, restore the
- "snapshot"entry to thetest_blockmatrix.