crowdstrike: remove FDR ingest-time cache processor in favor of LOOKUP JOIN

packages/crowdstrike/_dev/build/docs/README.md

@@ -352,12 +352,8 @@ FROM logs-crowdstrike.fdr-*
 | LIMIT 20
 ```
 
-**Elasticsearch 8.19+** is required for `LOOKUP JOIN` to resolve an alias. Use `crowdstrike_lookup.aidmaster` as in the example above. On **releases before 8.19**, `LOOKUP JOIN` must target the concrete transform destination index instead: in Kibana go to **Stack Management** → **Transforms**, open the CrowdStrike latest aidmaster transform, and use the **destination_index** name shown there (that name can change with the integration version).
-
 **Using enriched fields:** Enrichment from the lookup is under the `crowdstrike.info.host.*` namespace (e.g. `crowdstrike.info.host.hostname` for hostname, `crowdstrike.info.host.cid` for customer ID). Use these fields in dashboards and detection rules when building on query-time enrichment.
 
-**Ingest-time versus query-time:** The FDR integration’s **Enrich Host and User Metadata** option (`enrich_metadata`, on by default) uses the Elastic Agent (Filebeat) metadata cache to attach `aidmaster` and `userinfo` to events at ingest time. If you rely on query-time host enrichment only (transform + `LOOKUP JOIN` above), set **Enrich Host and User Metadata** to **Off** so host metadata is not applied twice. Turning it off also disables ingest-time enrichment from `userinfo`; if you still need user fields from `userinfo` on every document, keep ingest-time enrichment enabled or supplement with a separate query pattern. Disabling **Enrich Host and User Metadata** automatically makes **Keep Original Host and User Metadata** option (`keep_metadata`) ineffective and the metadata events are retained.
-
 ### Query-time user metadata enrichment (LOOKUP JOIN)
 
 A second transform maintains the latest user metadata per host-user pair from `UserIdentity` and `UserLogon` sensor events in a lookup index. Unlike `userinfo` directory data (which requires [Falcon Discover](https://www.crowdstrike.com/platform/exposure-management/falcon-discover/) and covers only Windows), sensor events are available to all FDR customers on all platforms (Windows, macOS, Linux, ChromeOS). You can enrich FDR events with user metadata at query time using ES|QL [`LOOKUP JOIN`](https://www.elastic.co/docs/reference/query-languages/esql/commands/lookup-join).
@@ -393,11 +389,7 @@ FROM logs-crowdstrike.fdr-*
 | LIMIT 20
 ```
 
-**Elasticsearch 8.19+** is required for `LOOKUP JOIN` to resolve an alias. Use `crowdstrike_lookup.userinfo` as in the examples above. On **releases before 8.19**, `LOOKUP JOIN` must target the concrete transform destination index instead: in Kibana go to **Stack Management** → **Transforms**, open the CrowdStrike latest userinfo transform, and use the **destination_index** name shown there (that name can change with the integration version). If you use both host and user lookups on releases before 8.19, you will need two concrete destination index names — one for aidmaster and one for userinfo — both obtainable from **Stack Management** → **Transforms**.
-
-**Using enriched fields:** Enrichment from the user lookup is under the `crowdstrike.info.user.*` namespace (e.g. `crowdstrike.info.user.name` for username, `crowdstrike.info.user.domain` for UPN domain, `crowdstrike.info.user.logon_type` for logon type). Use these fields in dashboards and ES|QL detection rules when building on query-time enrichment. Note that detection rules using EQL, threshold, or KQL operate on stored documents and cannot use `LOOKUP JOIN` — those rule types continue to rely on ingest-time cache enrichment for user metadata.
-
-**Ingest-time versus query-time:** The same **Enrich Host and User Metadata** option (`enrich_metadata`) that controls ingest-time host enrichment also controls ingest-time user enrichment from `userinfo` directory data. Query-time user enrichment via the transform is additive — it works regardless of whether ingest-time enrichment is enabled. If you rely on query-time enrichment only, set **Enrich Host and User Metadata** to **Off** so metadata is not applied twice. If both are active, user metadata may appear under `crowdstrike.info.user.*` from both the ingest-time cache and the query-time lookup; the values should be consistent but the ingest-time cache is populated from `userinfo` while the query-time lookup uses sensor events, so field availability may differ.
+**Using enriched fields:** Enrichment from the user lookup is under the `crowdstrike.info.user.*` namespace (e.g. `crowdstrike.info.user.name` for username, `crowdstrike.info.user.domain` for UPN domain, `crowdstrike.info.user.logon_type` for logon type). Use these fields in dashboards and ES|QL detection rules when building on query-time enrichment.
 
 #### ES|QL dashboard panels
 

packages/crowdstrike/changelog.yml

@@ -1,4 +1,31 @@
 # newer versions go on top
+- version: "4.0.0"
+  changes:
+    - description: >-
+        Remove ingest-time cache processor from FDR. Users with custom KQL, EQL, or threshold
+        queries filtering on `crowdstrike.info.*` as stored fields must rewrite those queries
+        as ES|QL using LOOKUP JOIN against `crowdstrike_lookup.aidmaster` or
+        `crowdstrike_lookup.userinfo`.
+      type: breaking-change
+      link: https://github.com/elastic/integrations/pull/19434
+    - description: >-
+        Remove `enrich_metadata`, `keep_metadata`, `metadata_ttl`, `metadata_cache_capacity`,
+        and `metadata_cache_write_interval` configuration variables from FDR. Aidmaster and
+        userinfo metadata events are now always indexed regardless of previous settings.
+        Metadata enrichment is now handled exclusively by query-time LOOKUP JOIN.
+      type: breaking-change
+      link: https://github.com/elastic/integrations/pull/19434
+    - description: >-
+        Aidmaster and userinfo events are now always indexed instead of being dropped after
+        cache enrichment, providing a reliable source for the LOOKUP JOIN transforms.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/19434
+    - description: Remove metadata file sorting from FDR SQS notification parsing script.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/19434
+    - description: Remove dead-code ingest pipeline processors that read from `crowdstrike.info.*` fields populated by the cache.
+      type: bugfix
+      link: https://github.com/elastic/integrations/pull/19434
 - version: "3.21.0"
   changes:
     - description: Use new `release` field for agentless deployment mode to establish as beta.

packages/crowdstrike/data_stream/fdr/_dev/test/pipeline/test-fdr-cspm-iom-evaluation.log-expected.json

@@ -36,6 +36,7 @@
                 },
                 "created": "2025-10-13T03:59:08.974Z",
                 "crn": "aws|123456789012|global|AWS::Account|123456789012",
+                "event_simpleName": "CloudSecurityIOMEvaluation",
                 "findings": [
                     {
                         "name": "Encryption Enabled",
@@ -58,8 +59,7 @@
                 },
                 "revision": 7,
                 "status": "Unresolved",
-                "url": "https://us-east-1.console.aws.amazon.com/ec2/home?region=us-east-1#Volumes:",
-                "event_simpleName": "CloudSecurityIOMEvaluation"
+                "url": "https://us-east-1.console.aws.amazon.com/ec2/home?region=us-east-1#Volumes:"
             },
             "event": {
                 "action": "CloudSecurityIOMEvaluation",

packages/crowdstrike/data_stream/fdr/_dev/test/pipeline/test-fdr-fim-rule-matched-linux.log-expected.json

@@ -24,9 +24,9 @@
                 "PolicyRuleSeverity": 3,
                 "RUID": "1000",
                 "cid": "2cc98db1a47b4c98b913c94d43bfab70",
+                "event_simpleName": "FileIntegrityMonitorRuleMatched",
                 "id": "ca65aa54-f7b9-453b-8ef1-99a5b2c8e3c4",
-                "name": "FileIntegrityMonitorRuleMatchedLinV9",
-                "event_simpleName": "FileIntegrityMonitorRuleMatched"
+                "name": "FileIntegrityMonitorRuleMatchedLinV9"
             },
             "device": {
                 "id": "2e3d9c94d9c34764860b1f3b444c6d4d"
@@ -186,9 +186,9 @@
                 "PreviousFileAttributesLinux": "16",
                 "RUID": "0",
                 "cid": "2cc98db1a47b4c98b913c94d43bfab70",
+                "event_simpleName": "FileIntegrityMonitorRuleMatched",
                 "id": "784c387c-806a-4add-a54f-83bc938d022d",
-                "name": "FileIntegrityMonitorRuleMatchedLinV9",
-                "event_simpleName": "FileIntegrityMonitorRuleMatched"
+                "name": "FileIntegrityMonitorRuleMatchedLinV9"
             },
             "device": {
                 "id": "2e3d9c94d9c34764860b1f3b444c6d4d"
@@ -346,9 +346,9 @@
                 "RUID": "1000",
                 "SecurityInformationLinux": "1",
                 "cid": "2cc98db1a47b4c98b913c94d43bfab70",
+                "event_simpleName": "FileIntegrityMonitorRuleMatched",
                 "id": "48cd83c0-62ba-471c-a6a2-fa5309195dde",
-                "name": "FileIntegrityMonitorRuleMatchedLinV9",
-                "event_simpleName": "FileIntegrityMonitorRuleMatched"
+                "name": "FileIntegrityMonitorRuleMatchedLinV9"
             },
             "device": {
                 "id": "2e3d9c94d9c34764860b1f3b444c6d4d"
@@ -500,9 +500,9 @@
                 "PolicyRuleSeverity": 3,
                 "RUID": "1000",
                 "cid": "2cc98db1a47b4c98b913c94d43bfab70",
+                "event_simpleName": "FileIntegrityMonitorRuleMatched",
                 "id": "54049e9e-d8c2-41c8-8822-98687f7a3608",
-                "name": "FileIntegrityMonitorRuleMatchedLinV9",
-                "event_simpleName": "FileIntegrityMonitorRuleMatched"
+                "name": "FileIntegrityMonitorRuleMatchedLinV9"
             },
             "device": {
                 "id": "2e3d9c94d9c34764860b1f3b444c6d4d"

packages/crowdstrike/data_stream/fdr/_dev/test/pipeline/test-fdr-fim-rule-matched-windows.log-expected.json

@@ -23,9 +23,9 @@
                 "PolicyRuleSeverity": 3,
                 "RegType": "1",
                 "cid": "2cc98db1a47b4c98b913c94d43bfab70",
+                "event_simpleName": "FileIntegrityMonitorRuleMatched",
                 "id": "fa2d4a8a-df61-4d4a-b1da-d04140d2faf0",
-                "name": "FileIntegrityMonitorRuleMatchedV11",
-                "event_simpleName": "FileIntegrityMonitorRuleMatched"
+                "name": "FileIntegrityMonitorRuleMatchedV11"
             },
             "device": {
                 "id": "05831d09e02c4949a44cf99ffa54f2ed"
@@ -170,9 +170,9 @@
                 },
                 "PolicyRuleSeverity": 1,
                 "cid": "2cc98db1a47b4c98b913c94d43bfab70",
+                "event_simpleName": "FileIntegrityMonitorRuleMatched",
                 "id": "a9e7eae2-3f8e-44c9-b847-40cf2af49e2b",
-                "name": "FileIntegrityMonitorRuleMatchedV11",
-                "event_simpleName": "FileIntegrityMonitorRuleMatched"
+                "name": "FileIntegrityMonitorRuleMatchedV11"
             },
             "device": {
                 "id": "05831d09e02c4949a44cf99ffa54f2ed"

packages/crowdstrike/data_stream/fdr/_dev/test/pipeline/test-fdr-geoip-disabled.log-expected.json

@@ -26,9 +26,9 @@
                 "PreviousFileAttributesLinux": "16",
                 "RUID": "0",
                 "cid": "2cc98db1a47b4c98b913c94d43bfab70",
+                "event_simpleName": "FileIntegrityMonitorRuleMatched",
                 "id": "784c387c-806a-4add-a54f-83bc938d022d",
-                "name": "FileIntegrityMonitorRuleMatchedLinV9",
-                "event_simpleName": "FileIntegrityMonitorRuleMatched"
+                "name": "FileIntegrityMonitorRuleMatchedLinV9"
             },
             "device": {
                 "id": "2e3d9c94d9c34764860b1f3b444c6d4d"