Skip to content

Add page-granular CPU offloading foundation - #450

Open
Lanoxia wants to merge 19 commits into
ovg-project:mainfrom
Lanoxia:zixuan/cpu-offload-control-plane
Open

Add page-granular CPU offloading foundation#450
Lanoxia wants to merge 19 commits into
ovg-project:mainfrom
Lanoxia:zixuan/cpu-offload-control-plane

Conversation

@Lanoxia

@Lanoxia Lanoxia commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add page-granular CPU offloading while preserving each kvcached virtual page ID.
  • Store complete KV page bundles in bounded pinned CPU memory and restore them transactionally.
  • Add PageAllocator and KVCacheManager offload/restore state, transition guards, memory accounting, and Python bindings.
  • Add policy and planning primitives that exclude active pages, respect CPU capacity, and estimate transfer-versus-recompute cost.
  • Add provider-neutral GPU validation, benchmarks, a self-hosted workflow, and reproducible artifact collection.

Safety model

Offload commits a complete CPU copy before releasing the GPU mapping. Restore maps the same virtual page ID, copies the complete payload back, and only then makes the page available again. Failed restore operations retain the CPU copy and roll back the GPU mapping.

Validation

  • CPU Tests pass on Python 3.9, 3.10, 3.11, 3.12, and 3.13.
  • Pre-commit, GPU-free C++ tests, and MyPy on Python 3.9-3.13 pass.
  • 26 focused CPU-offload policy, geometry, workflow, and binding-contract tests pass.
  • Real VMM validation passed on two NVIDIA H20 GPUs: 80 offload/restore round trips per GPU, byte-for-byte restoration, 256 MiB reclaimed per cycle, and dual-GPU isolation.
  • Measured transfer throughput was approximately 50 GB/s device-to-host and 54 GB/s host-to-device.

Scope

This PR provides the kvcached page-level data path, control plane, safety invariants, and validation infrastructure. Engine-specific automatic victim discovery and pressure-policy integration can build on these explicit page offload/restore primitives in follow-up work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants