Linux, Infiniband, OpenSM, directly connected nodes

[cadillac]

Hi, i have two nodes directly connected via Mellanox ConnectX-3 VPI cards. Every tutorial mentions that even directly connected 2 node InfiniBand networks has to have a subnet manager running to be able to operate. I noticed that after starting OpenSM, and the links became active and up, the process can be shut down and the network operates without issues. Now i'm wondering if i could get away with only running the subnet manager when the infiniband network "changes", with an one-shot service which runs after hardware initialization, and exits. Does anybody have experience with this?

 

[necr]

Yes, there’s a flag for OpenSM for this as well (configure once and exit). If you’re using only IPoIB for basic IP connectivity, that’s enough. However, if you want a new RDMA connection to be formed, like a new storage session, you’d need OpenSM, same for some diag tools. Best practice is to keep it running as a service, it should be less than 5% CPU.

 

[cadillac]

Yes, there’s a flag for OpenSM for this as well (configure once and exit). If you’re using only IPoIB for basic IP connectivity, that’s enough. However, if you want a new RDMA connection to be formed, like a new storage session, you’d need OpenSM, same for some diag tools. Best practice is to keep it running as a service, it should be less than 5% CPU.

Thanks for the explanation!