spike(apisix): opt-in Coraza WAF proxy-wasm image + Helm wiring

[blarghmatey]

What are the relevant tickets?

N/A. Follow-up spike from the investigation on #5096/#5099 (bot/scanner noise mitigation at the APISIX layer).

Description (What does it do?)

Spike, not a full rollout: builds a custom APISIX image bundling the Coraza WAF proxy-wasm filter, registers its build pipeline, and opts pre-production applications clusters into a low-risk long-horizon soak test before considering it for production.

Why Coraza over other options evaluated:

• chaitin-waf is already in our loaded APISIX plugin list, but it isn't self-contained — it proxies every request to a separate self-hosted Chaitin SafeLine WAF detection-engine service (APISIX >=3.5.0, SafeLine >=5.6.0) that would need to be deployed and operated as new infrastructure.
• Coraza runs in-process as a proxy-wasm filter — no separate service, no network hop per request.
• Confirmed against the actual upstream build script (api7/apisix-build-tools's build-apisix-base.sh) that wasm-nginx-module is compiled into every standard apisix-base build unconditionally (--add-module=../wasm-nginx-module-..., not an opt-in variant) — so the wasm/proxy-wasm runtime our stock apache/apisix image already ships is sufficient; no custom OpenResty compile needed.
• The only gap is getting the compiled coraza-proxy-wasm.wasm binary into the image. Confirmed via the coraza-proxy-wasm project's own README that OWASP Core Rule Set "comes embedded in the extension" at compile time by the upstream project — no separate ruleset files to deliver or mount.

Changes:

• dockerfiles/apisix-waf/Dockerfile: two-stage build — COPY --from=ghcr.io/corazawaf/coraza-proxy-wasm:0.6.0 /plugin.wasm onto apache/apisix:3.17.0-ubuntu (tracks APISIX_CHART_VERSION's default appVersion).
• src/ol_concourse/pipelines/container_images/apisix_waf.py: build-and-push-to-ECR pipeline, following the existing ol_python_base.py/kubewatch pattern (container_build_task + ensure_ecr_task + PutStep). Published as mitodl/apisix-waf:latest (+ git-short-ref secondary tag), ECR only (not Docker Hub — internal security tooling, not meant for public distribution). Deliberately not chained into a pulumi_jobs_chain — image builds here never trigger a deploy on their own.
• src/ol_concourse/pipelines/container_images/meta.py: registers apisix-waf-docker in PIPELINE_CONFIGS so the meta-pipeline auto-creates/updates it on merge, same as every other image-build pipeline.
• src/ol_infrastructure/infrastructure/aws/eks/apisix_official.py: new opt-in, per-cluster Pulumi config (apisix_custom_image_repository, apisix_custom_image_tag, apisix_wasm_plugins) that only diverges from today's behavior when a stack explicitly sets it, following the existing eks_config.get(...) or <default> pattern already used throughout this file for other per-cluster tunables (apisix_min_replicas, apisix_memory, etc.). pullPolicy switches to Always when a custom image is configured (since :latest needs re-checking on pod restart); the stock image path is unaffected.
• Pulumi.applications.CI.yaml / Pulumi.applications.QA.yaml: opt in with the three new config keys. applications.Production is deliberately not touched.
• dockerfiles/README.md: documents the new image per the directory's existing convention.

Why this is low-risk despite being "live" on two clusters: wasm.enabled: true loads the coraza-filter plugin globally, but no ApisixRoute or shared-plugin-config references it by name yet, so it's inert with respect to live traffic. What this soak-tests is narrower and specific: does APISIX start up and run stably over time on the custom image with the wasm runtime active. That's the operational confidence needed before attaching the plugin to any real route.

Screenshots (if appropriate):

N/A - infrastructure/backend only.

How can this be tested?

• pulumi preview against applications.QA and applications.CI, both configured: identical isolated diff on each (image.repository/image.tag/image.pullPolicy + new wasm block), 1 resource updated, 167 unchanged.
• pulumi preview against applications.Production: zero apisix-related diff (only the unrelated, already-known pending Traefik NLB fix from fix(eks): preserve real client IPs on the Traefik gateway NLB #5099 shows up).
• Dockerfile passes the repo's pre-commit Dockerfile-linting hook; all changed Python/YAML files pass ruff, mypy, and yamllint.

Not done in this PR (deliberately, this is a spike, not a full rollout):

• The Concourse pipeline is registered but hasn't been run yet on merge to actually confirm — first real build happens on merge via the meta-pipeline.
• No route has coraza-filter attached — it isn't wired into the shared OLApisixSharedPluginsConfig any app depends on, and won't be until the soak test on CI/QA shows it's stable.
• Production is not opted in.

Additional Context

Suggested next step after this merges and the soak test has run for a while with no stability issues: attach coraza-filter to a single low-traffic test route on one of the opted-in clusters (directives: Include @demo-conf, Include @crs-setup-conf, SecRuleEngine On, Include @owasp_crs/*.conf) to confirm it actually blocks known scanner traffic patterns (/wls-wsat/, /seeyon/htmlofficeservlet, etc.) before considering it for the shared plugin config any real app traffic depends on.

[Copilot]

Pull request overview

Adds an opt-in path to run APISIX with the Coraza proxy-wasm WAF filter by (a) defining a custom image build pipeline and Dockerfile, and (b) wiring Helm values in the EKS APISIX deployment to optionally reference that image and register wasm plugins.

Changes:

• Introduces a new apisix-waf Dockerfile and a Concourse pipeline to build/publish the image to ECR.
• Adds per-stack Pulumi config knobs to optionally override the APISIX image repo/tag and enable wasm plugins.
• Updates applications CI/QA stack configs to set those new knobs (currently contradicting the PR’s “no cluster opted-in” description).

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/ol_infrastructure/infrastructure/aws/eks/Pulumi.applications.QA.yaml	Opts the QA stack into the custom APISIX WAF image + wasm plugin configuration.
src/ol_infrastructure/infrastructure/aws/eks/Pulumi.applications.CI.yaml	Opts the CI stack into the custom APISIX WAF image + wasm plugin configuration.
src/ol_infrastructure/infrastructure/aws/eks/apisix_official.py	Adds opt-in Helm wiring for custom image repo/tag and wasm plugins.
src/ol_concourse/pipelines/container_images/meta.py	Registers the new apisix-waf container-image pipeline in the container-images meta pipeline list.
src/ol_concourse/pipelines/container_images/apisix_waf.py	Defines the Concourse pipeline to build and push the apisix-waf image to ECR.
dockerfiles/README.md	Documents the new apisix-waf image and how it’s published.
dockerfiles/apisix-waf/Dockerfile	Adds the two-stage image that copies the Coraza .wasm artifact into an APISIX base image.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

src/ol_infrastructure/infrastructure/aws/eks/apisix_official.py:225

• apisix_custom_image_repository and apisix_custom_image_tag are treated as independent optional config values, but an incomplete configuration (only one set) will produce invalid Helm values (e.g., custom repo with chart default tag that likely doesn’t exist, or overriding the stock image tag unintentionally). Consider enforcing that these two settings must be provided together so misconfiguration fails fast during preview/up.

    apisix_custom_image_repository = eks_config.get("apisix_custom_image_repository")
    apisix_custom_image_tag = eks_config.get("apisix_custom_image_tag")
    # e.g. [{"name": "coraza-filter", "priority": 7999,
    #        "file": "/usr/local/apisix/coraza-filter.wasm"}]
    apisix_wasm_plugins = eks_config.get_object("apisix_wasm_plugins") or []

[shaidar]

This should be turned into a project that includes some developers for a full blown assessment on how best to handle this. We already have existing tools/services in place that could be leveraged, however there's an additional cost associated with enabling those features. There are also some "free" tools such as the one proposed and others that could perform the job however it's added complexity and additional things to manage.
Turning this into an official project allows us to assess what the associated costs are, what are we willing to accept/live with, and make a decision on whether we spend the extra money or go down the path of tool assessment and deployment.