Can Ampere Altra Max + ALTRAD8UD-1L2T support Direct PCIe peer-to-peer DMA from BlueField-3 NIC → GPU memory?

[dping]

Can Ampere Altra Max + ALTRAD8UD-1L2T support Direct PCIe peer-to-peer DMA from BlueField-3 NIC → GPU memory with below setup?

Setup CPU: Ampere Altra M128-26 (ARM64)

• Motherboard: ASRock Rack ALTRAD8UD-1L2T

• GPU: NVIDIA RTX 6000 Ada

• NIC: NVIDIA BlueField-3

• CPU: Ampere Altra M128-26 (ARM64)

Looks like Ampere Altra uses multiple isolated PCIe host bridges and ARM SMMU enforces strict DMA isolation

Also No routing of peer transactions between host bridges. So GPUDirect RDMA cannot cross those boundaries.

Is that our understand correct? Is there any way to make it work?

Thank you very much

[quocbao]

Ampere CPUs are not designed for your use case

If you want “PCIe peer-to-peer”, go looking for servers having the bridge in the red circle.

If you want true high-speed peer-to-peer GPU bandwidth, go looking for SXM GPU.