Infiniband to Ethernet bridging question

renderfarmer

Member
Feb 22, 2013
249
1
18
New Jersey
Hi all. At the moment I have a Win2012 file server feeding my win8 workstation and seven win2008r2 render nodes connected via a Cisco Topspin 90 (SFS3001) 12 port IB switch with optional Ethernet Switch Module. It conveniently bridges my Ethernet LAN to my IPoIB network giving my IB machines internet access. However, my storage array is now faster than the Switch's 10Gbps speed so I'm looking to upgrade to 40Gbps.

My question is; what's the "best" way of bridging an IB network with my Ethernet LAN withouit the benefit of a multi fabric switch like the Topspin 90?

I really don't know that much about networking so "best" in this context means easiest.

Any advice is much appreciated.
 

Jeggs101

Well-Known Member
Dec 29, 2010
1,490
228
63
That setup is probably the easiest. what kind of render nodes btw?
 

renderfarmer

Member
Feb 22, 2013
249
1
18
New Jersey
Yes, of course. That's why I bought that switch a year ago and love it so much. That being said, I would love 40Gbps even more, and as far as I know there is no such thing as a 40Gbps IB switch with bridged Ethernet ports.

Would it be as simple as plugging my Ethernet into my file server and enable Internet Sharing in the control properties?

All of my render nodes are as follows:

Case Supermicro CSE-111TQ-563CB
Mobo Supermicro X9DRL-iF-O
CPU 2x Xeon E5-2620 Retail
HDD Intel 330 60GB
RISER Startech PEX8RISER
RAM Crucial (4x8GB) BLS2K8G3D1609ES2LX0
HSF Dynatron R15
IB Mellanox MHEA28-XTC
 
Last edited:

dba

Moderator
Feb 20, 2012
1,478
181
63
San Francisco Bay Area, California, USA
The Voltaire/Mellanox 4036E Infiniband switch is a version of the 4036 that includes a built-in Ethernet bridge plus two 10Gb Ethernet ports.

Yes, of course. That's why I bought that switch a year ago and love it so much. That being said, I would love 40Gbps even more, and as far as I know there is no such thing as a 40Gbps IB switch with bridged Ethernet ports.

Would it be as simple as plugging my Ethernet into my file server and enable Internet Sharing in the control properties?

All of my render nodes are as follows:

Case Supermicro CSE-111TQ-563CB
Mobo Supermicro X9DRL-iF-O
CPU 2x Xeon E5-2620 Retail
HDD Intel 330 60GB
RISER Startech PEX8RISER
RAM Crucial (4x8GB) BLS2K8G3D1609ES2LX0
HSF Dynatron R15
IB Mellanox MHEA28-XTC
 

Jeggs101

Well-Known Member
Dec 29, 2010
1,490
228
63
You could do something like that since you likely do not need massive ethernet bandwidth

What app are you rendering on?
 

renderfarmer

Member
Feb 22, 2013
249
1
18
New Jersey
The Voltaire/Mellanox 4036E Infiniband switch is a version of the 4036 that includes a built-in Ethernet bridge plus two 10Gb Ethernet ports.
Thanks, dba. I saw that on the site but but didn't realize a FW update would turn the 10GbE ports to 1GbE. These retail for around $15k and are rarer than rocking horse poop on eBay so I'll have to explore alternatives until one pops up. The 4036 can be had for under $1K. do you have any experience with the Voltair/Mellanox OS? Is it any more user friendly than Cisco's?
 
Last edited:

cactus

Moderator
Jan 25, 2011
829
77
28
CA
Get the 4036 and put a card in another box and do L3 routing. The eIPoIB will allow for L2 bridging on Linux, but I cant find information about when it will/was put into mainline.
 

renderfarmer

Member
Feb 22, 2013
249
1
18
New Jersey
I finally got around to trying windows built in internet connection sharing feature and it forces your computer to take on a 192.xxx.xxx.xxx IP. Not exacly helpful.

For fun I just tried bridging my ConnectX card and my built in intel Eth0 card and it did do it but a few seconds later I got my first BSOD in Windows 8 (DPC WATCHDOG VIOLATION).

Get the 4036 and put a card in another box and do L3 routing. The eIPoIB will allow for L2 bridging on Linux, but I cant find information about when it will/was put into mainline.
I'm really interested in following through on this but I'm a networking novice. I'm familair with the broad definition of L3 routing but can you please point me to a how-to on setting that up in windows server? Or suggest some google search terms that might lead me to a how-to on L3 routing in windows server? I've started looking around broadly but it's slim pickins so far.
 
Last edited:

dba

Moderator
Feb 20, 2012
1,478
181
63
San Francisco Bay Area, California, USA

renderfarmer

Member
Feb 22, 2013
249
1
18
New Jersey
If you have a windows box and want simple bridging, just visit the network connections control panel (control panel --> Network and Internet --> Network Connections). Select the two network connections that you want to bridge and then right click and select "bridge connections". Done.
I tried that for fun and got my first BSOD seconds later. When the system rebooted the bridged connection was gone. I just tried it in Win8 with the built in ethernet port that wasn't even connected so maybe that's what caused it. I'll give it a go this week on my win2012 machine with a live net connection.

Is that how you bridge your IB network with the internet?
 

renderfarmer

Member
Feb 22, 2013
249
1
18
New Jersey
Yes. I wish I had a bridging IB switch, but this way was easy and free.
Thanks! That sounds good to me. The minimal amount of traffic going from me to the web and back won't impact my network in the least so this sounds promising. I'll give it a try.
 

cactus

Moderator
Jan 25, 2011
829
77
28
CA
Windows does not have L2 bridging either. I am thinking this isn't a big thing for Mellanox because most new installs will have switches that have an Ethernet gateway built in. (4036E, dba) I was thinking of starting a thread on the Mellanox forum, even getting a "not going to happen any time soon" is better than nothing.

I read a post where the user used a NAT for his VMs in Hyper-V.

I searched a lot more for eIPoIB, but found nothing this year and it isn't in linux-next. From the mailing list stuff I did read, the problem is a good convention for passing MAC addresses though.
 

zer0sum

Active Member
Mar 8, 2013
395
145
43
Why don't you guys just run a little atom box with 2 NIC's in it that bridges the infiniband with the ethernet?
You could run vyatta or a similar router distro or even pfsense etc.
 

renderfarmer

Member
Feb 22, 2013
249
1
18
New Jersey
Windows does not have L2 bridging either. I am thinking this isn't a big thing for Mellanox because most new installs will have switches that have an Ethernet gateway built in. (4036E, dba) I was thinking of starting a thread on the Mellanox forum, even getting a "not going to happen any time soon" is better than nothing.

I read a post where the user used a NAT for his VMs in Hyper-V.

I searched a lot more for eIPoIB, but found nothing this year and it isn't in linux-next. From the mailing list stuff I did read, the problem is a good convention for passing MAC addresses though.
I installed a couple linux distros on my workstation using VirtualBox and was only able to get internet shared with them via NAT. It worked fine in both ubuntu and Fedora.

I watched a very interesting talk on this topic. http://www.youtube.com/watch?v=umjgFcbSzcc

The problem is that IPoIB doesn't have the ethernet header as part of the IB packet and its MAC address is 20bytes long, not 6bytes like ethernet.

According to the speaker, this is why you can't enslave an IB NIC to a vSwitch.

They said that they have written a shim layer driver between the TCP/IP stack and the native IPoIB. It makes the IPoIB packets look and act exactly like ethernet. As of the time they gave the talk it was for Vmware only. with other Virtualization standards to come.
 
Last edited:

cactus

Moderator
Jan 25, 2011
829
77
28
CA
Why don't you guys just run a little atom box with 2 NIC's in it that bridges the infiniband with the ethernet?
You could run vyatta or a similar router distro or even pfsense etc.
If one goes with the 4036(non E) switches that are popping up on ebay, this is exactly what I would do. It still does not help for VMs though.
 

cactus

Moderator
Jan 25, 2011
829
77
28
CA
I installed a couple linux distros on my workstation using VirtualBox and was only able to ge tinternet shared with them via NAT. It worked fine in both ubuntu and Fedora.

I watched a very interesting talk on this topic. Ethernet Tunneling Over IPoIB - YouTube

The problem is that IPoIB doesn't have the ethernet header as part of the IB packet and its MAC address is 20bytes long, not 6bytes like ethernet.

According to the speaker, this is why you can't enslave an IB NIC to a vSwitch.

They said that they have written a shim layer driver between the TCP/IP stack and the native IPoIB. It makes the IPoIB packets look and act exactly like ethernet. As of the time they gave the talk it was for Vmware only. with other Virtualization standards to come.
I have watched that video a few times and I believe Ali Ayoub is active on the Mellanox forum. It is about a year old, so I wish there was more progress on the Linux side. That is eIPoIB is was mentioned earlier in the thread. IIRC ESXi evolved from Linux and I have used the Mellanox stack on ESXi and it works great.

Interesting to note ifconfig shows 20 byte MAC, but the last 14 are zero.