feat: add elastic-package requires update command to bump requires pins from the registry

[teresaromero]

What

Adds elastic-package requires update — a new subcommand that reads requires.input and requires.content entries from manifest.yml, queries the package registry for the latest published versions, and writes the updated pins back to the file.

Key behaviours:

• Only applies to integration packages with a requires block; other package types are skipped.
• Respects conditions.kibana.version: a newer dependency version is only proposed when its Kibana constraint overlaps with the integration's.
• When a newer version exists but requires a higher Kibana version, a warning is printed suggesting to bump conditions.kibana.version.
• --dry-run reports proposals without modifying the file.
• --format json emits machine-readable output suited for CI batch jobs.
• Normalises constraint-style pins (e.g. ^0.3.0) to exact semver on write.

Output example:

➜  elastic-package requires update -C packages/nginx_integration_otel  --dry-run                 
WARN package nginx_otel 0.5.0 is available but requires kibana ^9.4.0; integration conditions.kibana.version is ^9.2.0 — consider bumping conditions.kibana.version
Package: nginx_otel_integration
Code owner: elastic/ecosystem
Requires updates:
╭─────────┬──────────────────┬─────────┬──────────┬────────┬──────────────────────────────╮
│  KIND   │    DEPENDENCY    │ CURRENT │ PROPOSED │ KIBANA │           WARNING            │
├─────────┼──────────────────┼─────────┼──────────┼────────┼──────────────────────────────┤
│ input   │ filelog_otel     │ 0.1.0   │ 0.2.0    │ ^9.2.0 │                              │
├─────────┼──────────────────┼─────────┼──────────┼────────┼──────────────────────────────┤
│ input   │ nginx_otel_input │ 0.1.0   │ 0.2.0    │ ^9.2.0 │                              │
├─────────┼──────────────────┼─────────┼──────────┼────────┼──────────────────────────────┤
│ content │ nginx_otel       │ 0.3.0   │ -        │        │ package nginx_otel 0.5.0     │
│         │                  │         │          │        │ is available but requires    │
│         │                  │         │          │        │ kibana ^9.4.0; integration   │
│         │                  │         │          │        │ conditions.kibana.version    │
│         │                  │         │          │        │ is ^9.2.0 — consider bumping │
│         │                  │         │          │        │ conditions.kibana.version    │
╰─────────┴──────────────────┴─────────┴──────────┴────────┴──────────────────────────────╯
Dry run: manifest.yml was not modified

➜  elastic-package requires update -C packages/nginx_integration_otel  --dry-run --format json
WARN package nginx_otel 0.5.0 is available but requires kibana ^9.4.0; integration conditions.kibana.version is ^9.2.0 — consider bumping conditions.kibana.version
{
  "package": "nginx_otel_integration",
  "codeowner": "elastic/ecosystem",
  "proposals": [
    {
      "kind": "input",
      "package": "filelog_otel",
      "current": "0.1.0",
      "proposed": "0.2.0",
      "kibana_constraint": "^9.2.0"
    },
    {
      "kind": "input",
      "package": "nginx_otel_input",
      "current": "0.1.0",
      "proposed": "0.2.0",
      "kibana_constraint": "^9.2.0"
    },
    {
      "kind": "content",
      "package": "nginx_otel",
      "current": "0.3.0",
      "proposed": "",
      "warning": "package nginx_otel 0.5.0 is available but requires kibana ^9.4.0; integration conditions.kibana.version is ^9.2.0 — consider bumping conditions.kibana.version"
    }
  ]
}



➜  elastic-package requires update -C packages/nginx_otel  --dry-run                             
Package: nginx_otel
Code owner: elastic/obs-infraobs-integrations
package type is "content"; requires update only applies to integration packages with requires

Why

Integration packages pin exact versions of input/content packages in manifest.yml. Keeping those pins up to date today requires manually checking the registry and editing the file — error-prone at scale across the integrations repository.

requires update makes bumping pins a single command, safe to run in CI across all integration packages, with --format json output for grouping PRs by code owner.

Kibana compatibility check

To determine whether a newer dependency version is compatible, the command checks whether the integration's and dependency's conditions.kibana.version constraints share any valid Kibana version.

• Resolves Automatically update package dependencies integrations#17404 (partially)

[teresaromero]

not sure this will be used on the automated script but i believe is worth enabling it for local use

[mrodm]

This assumes that constraints are somehow ordered, isn't it?

conditions:
  kibana:
    version: "^8.0.0 || ^9.0.0"

Would it work successfully with this other constraint ?

conditions:
  kibana:
    version: "^9.0.0 || ^8.0.0"

Based on the minVersionFromConstraintBranch function.

Not sure if this is happening now in the current packages.

[teresaromero]

The new subset check evaluates each || branch independently, so order does not affect the result. Both orderings are covered by the OR ordering invariant test cases in compatibility_test.go.

[mrodm]

I'm not sure if this is doing what we expect (or what I think I would expect).

For instance, an integration package with this kibana constraint:

conditions:
  kibana:
    version: "~9.1.0 || ^9.4.0""

and a new input package dependency (set as required) update with kibana constraint:

conditions:
  kibana:
    version: "^9.4.0""

Would return true in package.CheckConditions because the constraint is validated with the part of the constraint ^9.4.0. However, that integration package supports some Elastic stacks that this new input package would not be valid (9.1.x).

And this could be even more complicated to calculated if both packages contain constraints with || or ~ instead of ^.

Example of a package with several constraints with OR:
https://github.com/elastic/integrations/blob/cc8af15ae202a23ee52f5099793c40cce3ce50f2/packages/microsoft_sentinel/manifest.yml#L13

There are also opened constrains (that I guess they would be less problematic here):
https://github.com/elastic/integrations/blob/cc8af15ae202a23ee52f5099793c40cce3ce50f2/packages/cloud_asset_inventory/manifest.yml#L17

I'm not sure about the approach to take here. @jsoriano could you please chime in here too?

[jsoriano]

Yes, I think we need to reconsider the algorithm here. What we really would need to check is that the integration constraint is a subset of the dependency constraint. Added some comments about this in the tests.

[teresaromero]

Replaced the overlap check with a subset check. The algorithm splits the integration constraint on || and probes each branch with three representative versions (floor, floor+1 patch, major ceiling) against the dependency constraint. If any probe satisfies the branch but not the dependency, it returns false.

[jsoriano]

The high-level approach looks good, but some details may need another look.

[jsoriano]

Redundant requires?

Suggested change

	Integration packages with `requires` can depend on input or content packages by declaring them under
	`requires` in `manifest.yml`. Depending on the package type, dependencies are resolved
	Integration packages can depend on input or content packages by declaring them under
	`requires` in `manifest.yml`. Depending on the package type, dependencies are resolved

[teresaromero]

I've rephrase it. thnks

[jsoriano]

Thanks 👍

Maybe instead of "requires" we can say "required packages", or "dependencies".

Suggested change

	## Integrations with requires and the package registry
	## Integrations with dependencies and the package registry

Suggested change

	## Integrations with requires and the package registry
	## Integrations with required packages and the package registry

Or backquote requires so it refers to the setting in the manifest.

Suggested change

	## Integrations with requires and the package registry
	## Integrations with `requires` and the package registry

[jsoriano]

Could we add a test case like this one? This would be a common case when integrations start supporting new majors.

Suggested change

	},
	{
	name: "strict-greater range does not overlap higher constraint",
	integration: "^9.5.0 || ^10.0.0",
	dependency: "^9.6.0",
	want: false,
	},

[jsoriano]

A more realistic case that should be also a non-overlapping range, would be this one:

Suggested change

	integration: ">=9.4.0,<9.6.0",
	dependency: "^9.6.0",
	integration: "^9.4.0",
	dependency: "^9.6.0",

[jsoriano]

This should fail. This means that the integration doesn't have any kibana restriction, but the input has. The resulting built integration should also have ^9.6.0, not an empty constraint.

Suggested change

	name: "empty integration constraint always overlaps",
	integration: "",
	dependency: "^9.6.0",
	want: true,
	name: "empty integration constraint always overlaps",
	integration: "",
	dependency: "^9.6.0",
	want: false,

[jsoriano]

Try to reuse internal/yamledit, it already has helpers for the kind of functionality needed here.

[teresaromero]

Replaced hand-rolled YAML helpers with yamledit.NewDocumentBytes, GetSequenceNode, SetKeyValue, and Write.

[jsoriano]

Instead of having a single method that resolves and optionally applies version dumps, and given that we already have the type for proposals, maybe we should have two methods, one to resolve, and another one to apply.

func Resolve(opts Options) ([]UpdateProposal, error)
func Apply(manifest []bytes, proposals []UpdateProposal) ([]bytes, error)

[teresaromero]

Done. Resolve handles resolution only; Apply rewrites manifest bytes in memory; the command layer owns the disk write. Dry-run is a natural consequence of skipping Apply.

[jsoriano]

This method gets the versions by querying from all the derived kibana versions. This makes several queries to the registry, not sure if this will be fully accurate for what we need.

Maybe an initial approach while we don't have another way to get packages from a given constraint from the registry is to query all versions of a package with a single request (with all=true), and then do the filtering locally.

[teresaromero]

fetchAllRevisions now issues a single request with All: true and filters locally in filterByKibanaSubset.

[jsoriano]

Umm, we probably don't want to update to prerelease versions, at least by default. This can be another reason to just query all versions of the package and then filter locally. An exception could be a dependency that only has prerelease versions.

There could be a flag to decide if we want or not to update to prerelease versions, but getting them should not be the default.

[teresaromero]

as all the otel input packages are still a prerelease, that is why i configured as so, i will check for the flag so we can use it instead

[jsoriano]

Yeah, we can keep an exception for packages that only have prerelease versions. But for GA packages that use input packages that have GA versions we should definitely default to GA versions.

Btw, in principle requires can be used too with non-OTel packages, so we could create now an integration package that requires input packages with GA versions.

[jsoriano]

Nit. We could have some unit tests for this using httptest and/or with go-vcr to record real interactions.

[jsoriano]

LGTM, thanks for addressing the comments.

If you don't add the flag to add the changelog in this PR please create a follow up issue (or PR).

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 7ad0939

History

• 💔 Build #8095 failed d3cf953
• 💚 Build #8049 succeeded 4e30654
• 💚 Build #8029 succeeded e682a05
• 💔 Build #8028 failed b6aada7

[teresaromero]

If you don't add the flag to add the changelog in this PR please create a follow up issue (or PR).

i've opened a followup here #3626

[mrodm]

Tested locally and it looks great!
Thanks!