rdma4py

High-performance RDMA for Python.

Read the PyTorch developer blog introduction to rdma4py.

Documentation for all packages is published at d4l3k.github.io/rdma4py.

Benchmarks

See the benchmark overview and the detailed backend reports: EFA multi-lane, ibverbs multi-QP, and NVMe-oF/RDMA to GPU.

efa/ - AWS EFA and SRD bindings

Low-level bindings for AWS Elastic Fabric Adapter, including SRD SEND and one-sided RDMA, EFA direct-verbs queries, and torch-friendly GPUDirect memory registration. No Python runtime dependencies and no torch/CUDA linkage.

Install the EFA package from PyPI:

pip install efa

See efa/README.md for system setup, API examples, EFA peer rules, GPUDirect usage, source installation, and hardware testing.

ibverbs/ — low-level libibverbs bindings

Pythonic, Cython-based bindings to libibverbs intended as the foundation for building high-performance RDMA libraries in Python, with first-class GPUDirect support (register a GPU device pointer or an exported dma-buf fd). No runtime dependencies, no torch/CUDA linkage.

Install the ibverbs package from PyPI:

pip install ibverbs

See ibverbs/README.md for system setup, the API, quickstart, feature coverage, source installation, and testing instructions.

nvmeof/ - NVMe over Fabrics RDMA initiator

Userspace NVMe/RDMA controller and namespace access layered on ibverbs, with keyed SGL data placement directly into registered host or CUDA GPU memory. It is shipped separately because NVMe protocol policy is above the verbs layer.

pip install nvmeof

See nvmeof/README.md for target requirements, host and GPU examples, ordering rules, and current protocol scope.

experiments/ — research prototypes

Experimental code is kept outside the installable packages. The fault-tolerant GPUNetIO all-reduce implements a PyTorch-nightly ProcessGroup, timeout recovery, dynamic membership, and exact-byte NCCL comparison on top of ibverbs. The dual-NIC TCP experiment forces multi-flow traffic through two peer NICs on one host and verifies 200 Gb/s line rate with physical hardware counters.

License

BSD-3-Clause. See LICENSE.