Deep breath...my infiniband journey begins

[whitey]

Ok all I just acquired a Mellanox IS5022 QDR IB switch. Have a need to test IPoIB. Hold my hand while I start digging up old threads around here research and give me a 'high level dummy overview'. I know this switch does not have a SNM so I will likely end up running opensm on a small appliance VM, what HCA's are the sweet spot?

Anything else I am missing before I start my journey?

 

[cesmith9999]

is there a reason to do this?

when I did this a few years ago, it worked. however, the cost of bridging to a normal network for doing things was expensive. that is one reason that I recommend 40 GB ethernet adapters and ROCEv2. then you have a normal network environment that just plugs in and works.

Chris

 

[whitey]

I dunno you tell me we have a HPC cluster at work that I have been told we need to test and sort out IPoIB connectivity to potentially migrate 1GbE connections off of for applications that require high bandwidth/low latency and a fatter pipe than what we are throwing at it currently.

If I am WAY off in left field please advise.

 

[whitey]

Is a Mellanox MHQH19B-XTR or a a VPI card more desirable, I guess my confusion comes in as I have heard that VPI cards do both IB and ethernet, but how does the IB switch deal with that or do you have to have some new-fangled IB switch that understands ethernet or bridges the two worlds. If the answer is I need to run the IB network seperate of the IP ethernet network that is fine as well maybe if we run mgmt plane over ethernet and high bandwidth/low latency apps over IB network.

Sorry for souding completely lost, definitely not my forte'

 

[OkiieDoe]

if the mellanox card is VPI then it defaults to infiniband not ethernet it should be plug n play and you would run the IB network separate from the ethernet network. in my home lab every server has dual 1gb for front end and dual 40gb IB connection for back end.

 

[Patrick]

Get VPI cards if possible @whitey

Also, make sure the model you get supports the speeds you want. For example, there are ConnectX-3 cards that only support 10GbE on the Ethernet side IIRC.

 

[whitey]

if the mellanox card is VPI then it defaults to infiniband not ethernet it should be plug n play and you would run the IB network separate from the ethernet network. in my home lab every server has dual 1gb for front end and dual 40gb IB connection for back end.

OK so this setup sounds like two fabrics (which is perfectly fine depending on use case), ethernet for front end and IB for backend...cool/thanks for clarification. So if my intentions are to just run IPoIB but over dedicated IB cards and IB switch fabric then a non-VPI card would work, I just am having a tough time for some reason wrapping my head around VPI and IB/eth running side-by-side and what would be the needed switch to support that (any old IB switch, IB switch w/ ethernet bridge/gateway, newest gen IB switch, etc.) or if I even need ethernet mode if I intend to run IPoIB onlfy for now and do not have a switch that can take advantage of a VPI card?

Hell I am so lost but I know there is a ton of info round' here 'somewhere' I know i saw a recent post w/ a VPI guide so it's off to hunt that or google-fu.

 

[Patrick]

@whitey - my suggestion on the VPI side was more that post-IB you may one day pick up a switch that supports 40GbE. If so, it is nice to not have to buy new cards. This is doubly so since you can often find VPI for the same or marginally more than IB only cards.

 

[OkiieDoe]

all you would need is a mellanox infiniscale IV switch with a mellanox connect-x 2/3 vpi cards running in IB mode (default) for IPoIB.

 

[whitey]

OK I got the Mellanox MHQH19B-XTR VPI cards which look like 40G IB cards and 10G ethernet VPI cards. Think this'll work for my initial tests. Are you saying that this card (well maybe a connectx-3 VPI card) could just natively take 40G ethernet and shut off IB mode and hook it to a regualr 40G ethernet switch w/ the QSFP port and be happy? If so I follow.

 

[Patrick]

Some of the CX-3 and newer cards can do FDR IB and 40GbE.

If you have the ones that do 10GbE that is great. You can do the software swap from IB to Ethernet and then hook them into an Ethernet switch. Mellanox makes relatively cheap QSFP to SFP+ adapters for this I believe.

 

[whitey]

Cool, looks like this is the adapter.

HP / MELLANOX QSFP+ to SFP+ Adapter MAM1Q00A-QSA 655874-B21

EDIT: Mind blowing, so I 'could' take this adapter, come off of the HCA and take that other (can't quite tell if it will take DAC of OM3/4 cabling) end up to a 10G SFP+ ethernet switch port or did I flip/flop that up?

Let's call this 'IB rookie question session', thanks guys!

 

[Patrick]

Hold off on buying if you do not need them yet though. They even have cable options available so lots of possibilities when that time comes.

Those adapters are starting to go well below $10/ ea as well Lot of 20 HP / MELLANOX QSFP+ to SFP+ Adapter 655874-B21 | eBay

 

[whitey]

MHZH29B-XTR MELLANOX CONNECTX-2 40GB IB Q NETWORKING ADAPTER | eBay

That's pretty handy right there, no need for converters but a lil' more $$$.

EDIT: My goodness, why have i never known/looked at these, this look like the best of both worlds, frontend ethernet connect up 10G, backend high speed net/stg fabric for IB. I must have been living under a rock. Kinda bummed I got those other cards now but they will do and that's $100 less 'splainin' I have to do...soo impatient...SMH.

http://www.mellanox.com/related-docs/prod_adapter_cards/ConnectX-2_QSFP_IB_and_ SFP_10GigE_Card_User_Manual_MHZH29.pdf

 

[Patrick]

I thought about not linking this... NEW* Mellanox ConnectX-3 MCX354A-FCBT Dual Port FDR 56.6 Gb/s Infiniband + 40GbE | eBay

Those do 40GbE as well.

There is a reason I have been simplifying the network down to 40GbE.

 

[J--]

I picked up some IBM 95Y3461 that are listed as ConnectX-3 EN cards, that supposedly will do 40 GbE. Not sure what the corresponding Mellanox part number is, but they're cheap on eBay (picked up two for $70 each).

 

[MiniKnight]

@J-- where are you finding them for $70? I saw this? IBM Mellanox 00D9552 CONNECTX-3 VPI 40GBE Adapter Infiniband Network Card | eBay but none of the p/n you have.

 

[Rand__]

@whitey you mind if I ask some stupid basic question in this thread? Bought an SX6015 today and realize from reading here that I have way to little info on IB coming from 10gbaseT. Many questions are likely identical so I didn't want to create a duplicate thread but don't want to spam yours either

 

[whitey]

I have no qualms with that, fire away!

 

[Rand__]

Thanks
1. Sorry for this very basic q, trying to get my hardware straight and am confused regarding adapters/transceivers
a. The above mentioned adapters are for converting QSFP to SFP+ (ie connecting to a regular 10gbE switch)? If I were to get a QSFP to 4xSFP+ cable I would not need those additionally?
b. Optical transceivers would only be needed if I wanted to use fiber instead of copper cables?

2. Are ib switches downward compatible? i.e. Could i get a DDR card and use it on a QDR switch (I assume so) ?
Answer:
As a standard-based solution, InfiniBand offers backward compatibility between the different speeds.
This allows users to expand their system and to connect newer parts to older existing ones.

3. Can I run multiple instances of opensm in the same network at the same time?
Answer:
There can be multiple subnet managers residing in a subnet as long as only one is active. Non-active subnet managers (StandbySubnet Managers) keep copies of the active subnet manager’s forwarding information and verify that the activesubnet manager is operational. If an active sub-net manager goes down, a standby subnet manager will take over responsibilities to ensure the fabric does not go down with it

4. What is the Mellanox Gateway license needed for (if at all)?
Answer:
Built with Mellanox's SwitchX-based switches, the InfiniBand to Ethernet gateway software license, or system, provides full port flexibility to toggle between 56Gb/s InfiniBand and either 10, 40 and 56Gb/s Ethernet with low 400ns latency.
With the new gateway functionality, the Mellanox edge switches can now be used for both InfiniBand and Ethernet switches at the same time with an integrated gateway capability between the two networks
-> Not sure how to activate that on my un-managed switch though


Edit:Found most of it by now, left in for others