Add dynamic signal types system test support

[MichelLosier]

Resolves: #3298

Enhances system tests support for otel input packages with dynamic_signal_types with:

• Detects datastreams in ES based on expected dataset and namespace
• Adds signal_types config field to declare what signal types to explicitly expect in the test

Callouts

I explored a bit the following ask:

If elastic-package becomes able to detect the data streams created for a given policy, we could remove the logic we have to build the data stream names where it is expected to find data.

From the Fleet side we don't track what data streams actually get created concretely in ES for a policy, nor do data streams in ES point back to policies in the metadata so it seems to me we are left with constructing expected datastream and index template names in some form. Happy to explore further though if folks feel otherwise.

Additionally, the datastream discovery polls until it first gets a result of matching datastreams *-dataset-namespace, this though assumes that all expected signals will be produced in the first ingest cycle. In the case for sql_server_input_otel this works if I omit the signal_types option in the logs system test, although the test fails of course on field assertion, but both metric and log data streams are found. If we think there maybe some variability for the default cause we want to handle here, we can look at a longer wait poll strategy (configurable wait), or goroutine the discovery to poll and enqueue found datastreams as tests run.

How to test:

• Build locally and run elastic-package test system -v against the changes in sql_server_input_otel package in this PR: Update sql_server_input_otel logs signal test integrations#17794
  • This package was also added as a test package