FlashInfer CI Infrastructure

Infrastructure-as-code for FlashInfer's public CI system. This repository manages:

• AWS Infrastructure: VPC, subnets, security groups, IAM roles
• Self-Hosted Runners: GitHub Actions runners on EC2 (spot, on-demand, capacity blocks)
• Automation: Lambda functions for runner scaling, cleanup, and management
• CI/CD: Automated deployment via GitHub Actions with environment protection

Architecture

┌─────────────────────────────────────────────────────────────────┐
│  GitHub (flashinfer-ai/flashinfer)                              │
│    └── Webhook ──► API Gateway ──► Lambda (scale-up)            │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  AWS (us-west-2)                                                │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │  VPC (10.0.0.0/16)                                      │    │
│  │  ├── Public Subnets  ──► Runners (spot/on-demand)       │    │
│  │  └── Public Subnets  ──► CB Runners (H100/B200)         │    │
│  └─────────────────────────────────────────────────────────┘    │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │  Lambda Functions                                       │    │
│  │  ├── scale-up        (launch runners on job queue)      │    │
│  │  ├── scale-down      (terminate idle runners)           │    │
│  │  ├── cb-scale-up     (launch CB runners)                │    │
│  │  └── cb-manager      (check CB status)                  │    │
│  └─────────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────────┘

Quick Start

# Clone with submodules
git clone --recursive https://github.com/flashinfer-ai/ci-infra.git
cd ci-infra

# Set up environment
cp scripts/setup-env.sh.example scripts/setup-env.sh
# Edit setup-env.sh with your AWS credentials and GitHub App secrets
source scripts/setup-env.sh

# Deploy
cd terraform
terraform init
terraform plan
terraform apply

Available Runners

All runners are registered at the organization level and available to all repositories in flashinfer-ai.

Spot Runners (Cost-Optimized)

Type	GPU	runs-on Labels	Instance Types
CPU x64	-	[self-hosted, linux, x64, cpu, spot]	r5/r6a/r6i/r7a/r7i (12-16xlarge), m5/m6a/m6i/m7a/m7i (24xlarge)
CPU ARM64	-	[self-hosted, linux, arm64, cpu, spot]	r6g/r7g/r8g (8-16xlarge), m6g/m7g (16-24xlarge)
GPU T4	SM75	[self-hosted, linux, x64, gpu, sm75, spot]	g4dn (2xlarge, 4xlarge, 8xlarge)
GPU A10G	SM86	[self-hosted, linux, x64, gpu, sm86, spot]	g5 (2xlarge, 4xlarge, 8xlarge)

On-Demand Runners (Reliable)

Type	GPU	runs-on Labels	Instance Types
CPU x64	-	[self-hosted, linux, x64, cpu, on-demand]	r6a/r6i/r7a/r7i (16xlarge), m6a/m6i (24xlarge)
CPU ARM64	-	[self-hosted, linux, arm64, cpu, on-demand]	r6g/r7g/r8g (16xlarge), m6g/m7g (24xlarge)
GPU T4	SM75	[self-hosted, linux, x64, gpu, sm75, on-demand]	g4dn (2xlarge, 4xlarge, 8xlarge)
GPU A10G	SM86	[self-hosted, linux, x64, gpu, sm86, on-demand]	g5 (2xlarge, 4xlarge, 8xlarge)

Multi-GPU Spot Runners (4 runners per instance: 4x 2-GPU):

Type	GPU	runs-on Labels	Instance	Pricing
A100 2-GPU	SM80	[self-hosted, linux, x64, gpu, a100, 2gpu]	p4d.24xlarge	Spot + on-demand fallback

Capacity Block Runners (4 runners per instance: 4x 2-GPU):

Type	GPU	runs-on Labels	Instance	Pricing
H100 2-GPU	SM90	[self-hosted, linux, x64, gpu, h100, 2gpu]	p5.48xlarge	Capacity Block
B200 2-GPU	SM100	[self-hosted, linux, x64, gpu, b200, 2gpu]	p6-b200.48xlarge	Capacity Block

All multi-GPU nodes run 4 runners: 4x 2-GPU runners (GPUs 0-1, 2-3, 4-5, 6-7).

See terraform/templates/runner-configs/*.yaml for the full list of instance types.

Capacity Block Regions

Capacity Blocks are only offered in some regions at a time and expire, so CB runners can be launched outside the home region. The cb-scale-up Lambda stays in us-west-2 and tries it first, then each region in var.cb_secondary_regions (currently us-east-1 then us-west-1), launching into whichever has an active Capacity Block. Runners register with the same labels wherever they boot, so workflows do not need to know which region served them.

Each secondary region gets its own VPC, security group, instance role and launch template from terraform/modules/cb-region. To follow a new reservation into a new region, add an entry to var.cb_secondary_regions, a provider alias in terraform/providers.tf, and a module block in terraform/cb-secondary-regions.tf; set enabled = false to drop a region.

License

Apache License 2.0 - see LICENSE file for details.