IB Question

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

rich0dify

New Member
May 20, 2013
4
0
0
If I were to purchase a MHGH28-XTC (£55) for a storage server, and then 2xQlogic QLE7140's (£25) for the Hyper-V hosts and a couple of cables, would I be able to use the two ports on the MGH28-XTC to break out to get 10Gb to each of the Qlogic cards?

If this is possible, I should be able to saturate a RAID5 array on a PERC6i?

Cheers
 

dba

Moderator
Feb 20, 2012
1,477
184
63
San Francisco Bay Area, California, USA
I have not used those particular cards, but to answer your question as a general one, the answer is 'Yes', and I have done so in the past. Just remember that, without an Infiniband switch, you'll end up with two separate Infiniband networks. Each will require its own IP addresses (assuming IPoIB) and subnet manager. Assigning one IP per port is easy, and running multiple copies of the subnet manager isn't that difficult, but the setup will definitely require some additional configuration before it will work.

If I were to purchase a MHGH28-XTC (£55) for a storage server, and then 2xQlogic QLE7140's (£25) for the Hyper-V hosts and a couple of cables, would I be able to use the two ports on the MGH28-XTC to break out to get 10Gb to each of the Qlogic cards?

If this is possible, I should be able to saturate a RAID5 array on a PERC6i?

Cheers
 

shindo

New Member
May 7, 2013
26
0
1
WI, USA
I have not used those particular cards, but to answer your question as a general one, the answer is 'Yes', and I have done so in the past. Just remember that, without an Infiniband switch, you'll end up with two separate Infiniband networks. Each will require its own IP addresses (assuming IPoIB) and subnet manager. Assigning one IP per port is easy, and running multiple copies of the subnet manager isn't that difficult, but the setup will definitely require some additional configuration before it will work.
dba, do you have a link to a resource that would give more information about this? I don't have a switch, so I am hoping to be able to direct-connect two ESXi 5.1 hosts to a NAS4Free node and I'm not sure where I can run the subnet manager(s). I am hopeful that I can run opensm on the NAS4Free node, but I haven't been able to find any information regarding running two subnets. Another option would be to run one on each ESXi host, but I haven't really found any good documentation on how to go about that, either.

Thanks,
Shindo
 
Last edited:

dba

Moderator
Feb 20, 2012
1,477
184
63
San Francisco Bay Area, California, USA
If you always start the storage server first, just run OpenSM on that node. Or, even better, you can do what I did: Run opensm on every node. This is a no-no in a complicated IB network but works perfectly in this case.

Without a switch, you'll need one copy of subnet manager per IB port - so two copies on your storage server. Grab the guid for each port by running ibstat –p and then (on Linux for example) you can start opensm with opensm –g <0xguid number> -B

Use Google to find a more complete recipe - my notes are unusually sparse for this.

dba, do you have a link to a resource that would give more information about this? I don't have a switch, so I am hoping to be able to direct-connect two ESXi 5.1 hosts to a NAS4Free node and I'm not sure where I can run the subnet manager(s). I am hopeful that I can run opensm on the NAS4Free node since it is based on Solaris, but I haven't been able to find any information regarding running two subnets. Another option would be to run one on each ESXi host, but I haven't really found any good documentation on how to go about that, either.

Thanks,
Shindo
 
Last edited:

bp_968

New Member
Dec 23, 2012
45
0
0
dba, do you have a link to a resource that would give more information about this? I don't have a switch, so I am hoping to be able to direct-connect two ESXi 5.1 hosts to a NAS4Free node and I'm not sure where I can run the subnet manager(s). I am hopeful that I can run opensm on the NAS4Free node since it is based on Solaris, but I haven't been able to find any information regarding running two subnets. Another option would be to run one on each ESXi host, but I haven't really found any good documentation on how to go about that, either.

Thanks,
Shindo
Well when I did a point to point using various IB cards I ran the subnet manager on my windows client. No problems at all. You could probably run it inside a VM and run two copies on the same VM if you put each nic in a different virtual switch.

I got 550MB/s copying from a SSD to a RAMdrive over infiniband (IPoIB) which was the limit of my SSD. You should be able to get 700-800MB/s even with IPoIB.

EDIT: I forgot one of my cards during that testing session was a SDR only card. The Mellanox your looking at is a DDR card. If you use all DDR cards then you should see 1500-1700MB/s I believe. Thats an advantage over using a switch I think since you can get DDR/QDR ect rates without a really expensive switch (mine was cheap but SDR speeds only)

Ben