Hello all,
I've been a bit of a lurker here for the past few years but finally decided I had a question worth posing. Forgive me if this is in the wrong section as it's both a bit of a hardware question as well as a software one. To set the stage a bit, I have two 40Gbps mellanox connectx-2 cards, one is in my main workstation while the other is in my dell r820 which is housing a pair of samsung 960 evos in raid 0 (yes, I know this is not redundant. It's only being used as a temporary cache). I've gone through the entire setup and configured the cards with the appropriate firmaware and have them talking to each other, I've set up IPoIB and have gotten good results testing the speed of the remote connection which are well above what I would get over standard sata 3 over a gigabit ethernet connection, but nowhere near as fast as what I see locally on the r820. Just to give a quick example over the IPoIB connection I get 2802.6MB/s Reads and 3044.3MB/s writes; locally I hit 5244.5MB/s and 3785.0MB/s respectively.
So the results are pretty much what I would have expected over the 10Gbps IPoIB link, which is a relatively simple setup. My question is since I'm using IPoIB I'm not getting the full benefit of having a 40Gbps card, and have recently been reading a bit about NVMEoF, has anyone here had experience with setting this up? I know another method would be iSCSI over RDMA, but I have no experience with that.
Hope to hear from the community soon! Thanks!
Ryan
I've been a bit of a lurker here for the past few years but finally decided I had a question worth posing. Forgive me if this is in the wrong section as it's both a bit of a hardware question as well as a software one. To set the stage a bit, I have two 40Gbps mellanox connectx-2 cards, one is in my main workstation while the other is in my dell r820 which is housing a pair of samsung 960 evos in raid 0 (yes, I know this is not redundant. It's only being used as a temporary cache). I've gone through the entire setup and configured the cards with the appropriate firmaware and have them talking to each other, I've set up IPoIB and have gotten good results testing the speed of the remote connection which are well above what I would get over standard sata 3 over a gigabit ethernet connection, but nowhere near as fast as what I see locally on the r820. Just to give a quick example over the IPoIB connection I get 2802.6MB/s Reads and 3044.3MB/s writes; locally I hit 5244.5MB/s and 3785.0MB/s respectively.
So the results are pretty much what I would have expected over the 10Gbps IPoIB link, which is a relatively simple setup. My question is since I'm using IPoIB I'm not getting the full benefit of having a 40Gbps card, and have recently been reading a bit about NVMEoF, has anyone here had experience with setting this up? I know another method would be iSCSI over RDMA, but I have no experience with that.
Hope to hear from the community soon! Thanks!
Ryan