This 4029GP-TRT2 server consists of 10 x 1080Ti + 2x Xeon Silver 4114. I suppose TRT2 is a double-root since the the official site did not say it is a single root complex.
It seems that in the uni-p2pDiabled mode, the bandwidth is much better than the double-root version (4028GR-TR), and is close to the single root version (TR2).
However, somehow I cannot make the P2P=enabled work. The program hangs when P2P=enabled. Maybe I didn't install CUDA or NCCL correctly.
4029GP-TRT2:
4028GR-TR bandwidth result from https://www.servethehome.com/single-root-or-dual-root-for-deep-learning-gpu-to-gpu-systems/
4028GR-TR2
It seems that in the uni-p2pDiabled mode, the bandwidth is much better than the double-root version (4028GR-TR), and is close to the single root version (TR2).
However, somehow I cannot make the P2P=enabled work. The program hangs when P2P=enabled. Maybe I didn't install CUDA or NCCL correctly.
4029GP-TRT2:
4028GR-TR bandwidth result from https://www.servethehome.com/single-root-or-dual-root-for-deep-learning-gpu-to-gpu-systems/
4028GR-TR2
