Fix ignored fields system test query

[mturac]

Closes #3524

What changed

• Read the _ignored metadata field from doc values in the runtime field script used by system tests.
• Add a regression test that locks the query to doc-values lookup and prevents the stored-fields lookup from coming back.

Tests

• go test ./internal/testrunner/runners/system
• make format
• git diff --check

[jsoriano]

/test

[jsoriano]

test integrations

[jsoriano]

Change LGTM, let's wait to tests.

[elastic-vault-github-plugin-prod]

Created or updated PR in integrations repository to test this version. Check elastic/integrations#19022

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: a152c0f

Failed CI Steps

• :go: Integration test (false positive): prometheus
• :go: Integration test: sql_server_input_otel
• :go: Integration test: mysql_input_otel
• :junit: Junit annotate

History

[jsoriano]

This change produces failures in many packages, see elastic/integrations#19022

They may be legit, but we need to think on an strategy to enable this in a controlled way, so we don't need to review and fix all packages at once.

[mturac]

Thanks for checking this. That makes sense, I had only scoped the change to the query behavior, but the package-level fallout means the rollout needs to be narrower.

I’ll rework this so the stricter ignored-fields check can be enabled in a controlled way instead of forcing all affected packages to be fixed in one pass.