Skip to content

README.md

Latest commit

 

History

History
202 lines (157 loc) · 7.67 KB

README.md

File metadata and controls

202 lines (157 loc) · 7.67 KB
Raw

Component Test Harness

Validate AICR components end-to-end with a single command. No GPU hardware required for most components.

Quick Start

make build
make component-test COMPONENT=cert-manager

Auto-detects the test tier, creates a Kind cluster, deploys the component, and runs its health check. Components detected as scheduling tier are redirected to the KWOK infrastructure (make kwok-e2e) and exit with code 2 — no Kind cluster is created for those.

Test Tiers

Tier What it validates Cluster needs Example
scheduling Pods schedule on correct nodes Kind + KWOK Any component with overlays
deploy Component deploys and health checks pass Kind cert-manager, kai-scheduler
gpu-aware GPU-dependent components init against fake GPUs Kind + nvml-mock gpu-operator

Auto-Detection

The harness reads recipes/registry.yaml to determine the tier:

Has health check? GPU references? Detected tier
No No scheduling
Yes No deploy
Yes Yes gpu-aware
No Yes gpu-aware (warns about missing health check)

Override with TIER=:

make component-test COMPONENT=gpu-operator TIER=gpu-aware

Or set testTier in registry.yaml:

- name: my-component
  testTier: gpu-aware
  helm: ...

Dependencies are not deployed. The harness bundles and deploys exactly one component; dependencyRefs declared on the component's overlay/mixin refs are NOT installed — for chart-backed and manifest-only components alike. If the component hard-depends on another component (e.g. nodewright-customizations applies a Skyhook CR whose CRD ships with nodewright-operator; kubeflow-trainer needs cert-manager webhooks), install the dependency first with KEEP_CLUSTER=true so the EXIT-trap cleanup does not immediately uninstall it — for example make component-test COMPONENT=nodewright-operator KEEP_CLUSTER=true — or expect the deploy step to fail on the bare test cluster. The deploy script prints a warning listing undeployed dependencies: for manifest-only components these come from the exact overlay ref the recipe was synthesized from; for chart-backed components (whose synthesized recipe is registry defaults, tied to no variant) the warning lists the union across recipe variants, so some entries may be variant-only (e.g. a GB200 leaf's DRA driver) and not needed by the default configuration. Platform-specific dependencies (e.g. the *-ocp*/OLM chains, whose OperatorGroup and Subscription manifests need OLM CRDs) cannot be pre-installed with this Kind-based harness — test those against a matching cluster.

Make Targets

# Full test (auto-detect tier, create cluster, deploy, health check, cleanup)
make component-test COMPONENT=cert-manager

# Individual steps (for debugging)
make component-detect COMPONENT=cert-manager     # Show detected tier
make component-cluster                     # Create/reuse Kind cluster
make component-deploy COMPONENT=cert-manager     # Deploy component only
make component-health COMPONENT=cert-manager     # Run health check only
make component-cleanup COMPONENT=cert-manager    # Uninstall component

# Keep cluster for debugging
KEEP_CLUSTER=true make component-test COMPONENT=cert-manager

# Delete cluster entirely
make component-cleanup DELETE_CLUSTER=true

Environment Variables

Global

Variable Default Purpose
COMPONENT (required) Component name from registry.yaml
TIER (auto-detected) Override: scheduling, deploy, gpu-aware
CLUSTER_NAME aicr-component-test Kind cluster name
KUBECONFIG (auto) Path to kubeconfig
KEEP_CLUSTER false Preserve cluster after test
DEBUG false Extra debug logging

Cluster (ensure-cluster.sh)

Variable Default Purpose
KIND_NODE_IMAGE from .settings.yaml Kind node image
KIND_CONFIG tools/component-test/kind-config.yaml Kind config file
CLUSTER_WAIT_TIMEOUT 120s Node readiness timeout

GPU Mock (setup-gpu-mock.sh)

Variable Default Purpose
NVML_MOCK_VERSION from .settings.yaml nvml-mock version
NVML_MOCK_IMAGE ghcr.io/nvidia/nvml-mock Image override
GPU_PROFILE a100 GPU profile: a100, h100, gb200
GPU_COUNT 8 GPUs per node
DRIVER_VERSION auto from profile Mock driver version (e.g., 550.163.01)
MOCK_READY_TIMEOUT 300s DaemonSet readiness timeout

Deploy (deploy-component.sh)

Variable Default Purpose
HELM_TIMEOUT 300s Helm install timeout
HELM_NAMESPACE from registry.yaml Override namespace
HELM_VALUES (none) Extra --values file
HELM_SET (none) Extra --set overrides (comma-separated)
AICR_BIN auto-detected from dist/ Path to aicr binary

Health Check (run-health-check.sh)

Variable Default Purpose
HEALTH_CHECK_TIMEOUT 5m Chainsaw assert timeout
HEALTH_CHECK_FILE from registry.yaml Override health check path
CHAINSAW_BIN chainsaw Path to chainsaw binary

Cleanup (cleanup.sh)

Variable Default Purpose
DELETE_CLUSTER false Delete the Kind cluster
FORCE_CLEANUP false Skip confirmation prompts

Debugging a Failure

# 1. Run with KEEP_CLUSTER to preserve state
KEEP_CLUSTER=true make component-test COMPONENT=cert-manager

# 2. Inspect pods
kubectl -n cert-manager get pods
kubectl -n cert-manager describe pod <pod-name>
kubectl -n cert-manager logs <pod-name>

# 3. Re-run just the health check
make component-health COMPONENT=cert-manager

# 4. Re-deploy after fixing
COMPONENT=cert-manager bash tools/component-test/cleanup.sh
make component-deploy COMPONENT=cert-manager
make component-health COMPONENT=cert-manager

# 5. Clean up when done
make component-cleanup COMPONENT=cert-manager DELETE_CLUSTER=true

Adding GPU-Aware Testing

For components requiring GPU resources: ensure .settings.yaml has component_test.nvml_mock_version. GPU references in values.yaml or registry entries auto-detect; override via TIER=gpu-aware or set testTier: gpu-aware in registry.yaml. Customize: GPU_PROFILE=h100 GPU_COUNT=4 make component-test ....

Troubleshooting

Issue Check
aicr binary not found Run make build first
Component not found in registry Verify component name matches recipes/registry.yaml
Health check file not found Create recipes/checks/<component>/health-check.yaml
Kind cluster creation fails Check Docker is running, kind is installed
Helm install timeout Increase HELM_TIMEOUT, check pod events
chainsaw not found Run make tools-setup
nvml-mock not ready Increase MOCK_READY_TIMEOUT, check DaemonSet logs

Architecture

make component-test COMPONENT=cert-manager
         │
    detect-tier.sh          → scheduling | deploy | gpu-aware
         │
    ensure-cluster.sh       → Reuse or create Kind cluster
         │
    setup-gpu-mock.sh       → (gpu-aware only) Deploy nvml-mock
         │
    deploy-component.sh     → Bundle + helm install
         │
    run-health-check.sh     → Chainsaw health check
         │
    cleanup.sh              → Uninstall + optionally delete cluster

All scripts are independently runnable and accept environment variables for override. Configuration defaults come from .settings.yaml.