[backports] Add .backports.yml inventory, validator, and dry-run trigger (#19210)

[mrodm]

TL;DR

Implements the backport branch inventory described in issue #19210:
a .backports.yml file that tracks all active backport-* branches,
a Go package with validator and active-branch checker that runs in CI,
and a pipeline step that dry-runs integrations-backport
for each newly added inventory entry.

Proposed commit message

Add .backports.yml inventory, validator, and dry-run trigger

WHAT

Introduces a structured inventory of all active backport branches in
.backports.yml (59 entries at time of writing). Each entry records:

- package: aws
  branch: backport-aws-3.17
  base_version: "3.17.0"
  base_commit: "5b593f6681"
  maintained_until: null
  archived: false

A branch is considered inactive if archived: true OR
maintained_until is set to a date in the past.

Supporting tooling added alongside the inventory:

• dev/backports/inventory.go — Go implementation that provides:
  • ValidateInventory (run via mage validateBackportsInventory):
    checks every entry for required fields, valid semver base_version,
    hex base_commit, that the referenced package exists in packages/,
    and detects duplicate branch names and package/version pairs.
  • CheckActive (run via mage checkBackportBranchActive):
    evaluates whether a single branch is active, covering both the
    archived: true flag and an expired maintained_until date.
• .buildkite/scripts/check_backports_inventory.sh — CI wrapper that
  runs the Go validator and blocks the pipeline on failure.
• .buildkite/scripts/trigger_backport_dryrun.sh — for each entry
  that is new in the PR (absent from the base branch), uploads a
  trigger step that runs integrations-backport in DRY_RUN=true
  mode to validate base_commit and base_version before the entry
  lands on main.

New mage targets in magefile.go:

Target	Description
ValidateBackportsInventory	Validates .backports.yml schema (required fields, semver, hex SHA, package existence, duplicates)
CheckBackportBranchActive <branch> [-asjson]	Reports whether a branch is active; exit 0 = active, 1 = inactive, 2 = error

CI wiring in .buildkite/pipeline.yml:

• check-backports-inventory step runs the Go validator on every PR
  that touches .backports.yml or the validator source.
• trigger-backport-dryrun step runs the dry-run trigger, gated on
  check-backports-inventory passing.

.buildkite/pipeline.backport.yml is updated so triggered dry-run
builds skip the human-input gate (input-variables step).

dev/scripts/README.md added to document that dev/scripts/ scripts
are not involved in package testing and are therefore excluded from
non_package_patterns.

Test coverage:

• dev/backports/inventory_test.go — Go unit tests for both the
  validator (ValidateInventory) and the active-branch check
  (CheckActive), including duplicate-detection cases and the known
  security_detection_engine@8.17.7 exception.
• run_dev_scripts_tests.sh refactored to delegate to per-script test
  files via direct bash calls (missing file now fails the build).

WHY

Without a structured inventory, automation cannot reliably determine
which backport-* branches are active, maintained, or stale.
This inventory is the foundation for the backport tooling described
in the parent epic (#19016).

Author's Checklist

• Ensure all debugging changes are removed before merging.
• Verify base_commit values for backport-security_detection_engine-* branches (noted as approximations in the file header).
• Test triggering integrations-backport pipeline https://buildkite.com/elastic/integrations/builds/44100
• Check archived values for the entries in .backports.yml

How to test this PR locally

# Run the Go inventory validator
mage validateBackportsInventory

# Check whether a specific branch is active
dev/scripts/backport_check_active.sh --branch backport-aws-3.17
dev/scripts/backport_check_active.sh --branch backport-aws-3.17 --asjson

# Run the test suite
mage -v check
# or 
go test ./dev/backports/...

The trigger-backport-dryrun step can be observed in the Buildkite
build for this PR — it should queue a dry-run for any entry added or
absent from main's .backports.yml. Example: https://buildkite.com/elastic/integrations/builds/44100

Related issues

• Closes backport: introduce .backports.yml inventory and active-branch check script #19210
• Relates Improve package backport tooling for routine multi-version support #19016

---

This PR was generated with the assistance of Claude (claude-sonnet-4-6).

[github-actions]

TL;DR

The failing Buildkite job is the backport-branch creation step, and it fails because the configured BASE_COMMIT (65b2a04d98) is not found in the branch being checked (HEAD in this run). Update the backport inputs/branch selection so commit validation runs against the intended source branch and a valid commit SHA.

Remediation

• Ensure SOURCE_BRANCH resolves to the intended branch (typically main) before commitExists runs in .buildkite/scripts/backport_branch.sh (SOURCE_BRANCH assignment near line 43; validation at lines 294-297).
• Provide a valid BASE_COMMIT from that branch via Buildkite metadata/input step before running .buildkite/scripts/backport_branch.sh.
• Re-run the integrations-backport build after updating branch/metadata values.

Investigation details

Root Cause

create-backport-branch exits in the base-commit validation path:

• Script logic validates BASE_COMMIT with commitExists "$BASE_COMMIT" "$SOURCE_BRANCH" at .buildkite/scripts/backport_branch.sh line 295.
• In this failing run, logs show SOURCE_BRANCH: HEAD, and then validation fails with The commit 65b2a04d98 doesn't exist in the branch HEAD.

This is a configuration/pipeline input mismatch (invalid commit/branch context for backport validation), not a package test failure.

Evidence

• Build: https://buildkite.com/elastic/integrations/builds/44099
• Job/step: integrations-backport → Creating the backport branch
• Key log excerpts:
  • --- SOURCE_BRANCH: HEAD
  • The commit 65b2a04d98 doesn't exist in the branch HEAD
  • Error: The command exited with status 1

Verification

• Not run (detective workflow is read-only; analysis based on provided Buildkite logs and repository script logic).

Follow-up

If this run was intended as a PR-only dry-run, gate the branch-creation step so it does not execute commit validation without explicit UI input metadata.

Note

🔒 Integrity filter blocked 2 items

The following items were blocked because they don't meet the GitHub integrity level.

• [backports] Add .backports.yml inventory, validator, and dry-run trigger (#19210) #19362 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #19362 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[mrodm]

This file was added to the Pull Request for reference on how the initial .backports.yml was built.

But this was for a one-shot.
I think it can be removed. WDYT ?

[mrodm]

To be removed before merging.
Keep it for now in case it is needed to push more changes.

[mrodm]

This will trigger a DRY_RUN build of the integrations-backport for each new entry detected.

For this first time, it has been set that it will not trigger any build. This is detected when in the main branch there is no .backports.yml file.

It has also been set in the .buildkite/pipeline.yml file that this step is just triggered in Pull Requests targeting main brnach.

    if_changed:
      - ".backports.yml"
    if: |
      build.env('BUILDKITE_PULL_REQUEST') != "false" &&
      build.env('BUILDKITE_PIPELINE_SLUG') == "integrations" &&
      build.env('BUILDKITE_PULL_REQUEST_BASE_BRANCH') == "main"

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 23c6012

History

• 💚 Build #44217 succeeded d7fa665
• 💚 Build #44214 succeeded 3e36fd9
• 💚 Build #44192 succeeded da0a5a0
• 💚 Build #44161 succeeded 1436a4e
• 💚 Build #44159 succeeded f60e65e
• 💚 Build #44157 succeeded 58215a1

cc @mrodm

[mrodm]

Since mage 1.16.0, optional parameters are supported.

The optional parameter have the name of the argument. So here it would be

mage checkBackportBranchActive -asjson

Do you prefer another name for this argument/parameter?
Some other options:

• -tojson
• -json

If we want to use -json , it would be needed to do some renamings in the imports too and other changes

--- magefile.go
+++ magefile.go
@@ -8,7 +8,7 @@ package main
 
 import (
        "context"
-       "encoding/json"
+       stdjson "encoding/json"
        "fmt"
        "io"
        "os"

Probably it is better (or expected by the users) to use -json , WDYT ?