Infiniband to Ethernet bridging question

[MaxCFM]

Ok this may be a dumb question but why can't you just hang the topspin 90 off the QDR switch as part of the IB fabric? Use the QDR switch for interconnectivity and the topspin as the gateway... I would think the subnet manager would be able to handle that. Another option might be to find a Xsigo Fabric Director (now Oracle Fabric Interconnect) used.

 

[renderfarmer]

Ok this may be a dumb question but why can't you just hang the topspin 90 off the QDR switch as part of the IB fabric? Use the QDR switch for interconnectivity and the topspin as the gateway... I would think the subnet manager would be able to handle that. Another option might be to find a Xsigo Fabric Director (now Oracle Fabric Interconnect) used.

That could work, but then you have a whole other switch taking up space/power/cooling just for bridging.

I've personally decided to switch over to 40/10GbE and call it a day.

 

[MaxCFM]

Fair enough. If you're looking to part with your topspin please keep me in mind

 

[renderfarmer]

Fair enough. If you're looking to part with your topspin please keep me in mind

I have a spair one with Eth module that I have to flash to the latest OS so it will support the bridge. Just need to find time to set up an FTP for the image to flash it. I'll ping you if I get it working. The other switch will be for sale once I get my 40GbE gear.

 

[mrkrad]

which card is 40gbe ? the 10gbe 10gbase-T is finally affordable. can they rock 40gbe over cheap wire? (0.5 to 5 feet of cat7)? SFP is too damn expensive.

With $100/port XS712T and XS708E 10gbase-t can bring a small group a single 10gb pipe to share rather than single gigabit directs.

 

[renderfarmer]

For my specific scenario, I only need 40GbE between my workstation and the file server; a pair of MCX313A-BCBT ($477ea) are what I'm leaning towards. Unfortunately there is 20m of distance to bridge so a MC2207310-020 FDR Optical cable ($456) is the best option. Not a cheap option but not astronomically expensive.

As for the rest of my farm; I'm seriously looking at migrating it to a SuperBlade. I have two networking options there:

1. 1GbE switch with three 10GbE SFP uplinks (dual 10GbE Card at the File Server side). Team both 1GBE ports on each blade for 2GbE aggregate.
2. They currently have promotional pricing on 10GbE swithes and mezzanine cards (50% off). Dual Port 10GbE mezzanine cards are $215ea.

Obviously these are not in the same pricing league as the kind of stuff we find on eBay but since I'm earning a living off of this gear it's easier to justify.

 

[cactus]

renderfarmer, are you doing 10GbE with your Mellanox cards?

 

[mrkrad]

benefit of the $300 intel x540-T2 is that you can use cheap cat6 cabling . beats the heck out of SFP+ or CX4 cabling that costs $50/foot or gbic's.

150 feet with a single cat6 wire at full power. 300 feet with cat6a shielded or cat7.

less than 15 feet runs in green mode which reduces the power per port by a alot.


the older 2006 based X520-T2 are sometimes cheaper. The broadcom 10gbase-t have 128meg per port so you don't need to reserve 1gb per port of system memory. they also do not send pause frame by default. Pretty hard to find them. HP nc530t .

X520's can be bought cheap

 

[renderfarmer]

renderfarmer, are you doing 10GbE with your Mellanox cards?

I have them set to "10Gb Max" when I install the drivers. I don't think that my infinihost III cards are capable of actual Ethernet.

 

[renderfarmer]

benefit of the $300 intel x540-T2 is that you can use cheap cat6 cabling . beats the heck out of SFP+ or CX4 cabling that costs $50/foot or gbic's.

I agree with your point on 10Base-T. But since there is no such thing as 40Base-T it's not really applicable.

As for wiring each of my nodes with 10GbE, if I go the blade route there are no cables required except for two 1m SFP uplinks which are $50ea.

 

[cactus]

I have them set to "10Gb Max" when I install the drivers. I don't think that my infinihost III cards are capable of actual Ethernet.

Yeah, no EN mode on pre-ConnectX VPI cards. So, IPoIB on SDR IB.

What 10GbE cards are you going with?

 

[renderfarmer]

What 10GbE cards are you going with?

If I go the SuperBlade route then I have only one option for the render nodes, the aforementioned Mezzanine cards:
Super Micro Computer, Inc. - Products | SuperBlade® | Networking | AOC-XEH-iN2 (Intel sillicone, dual ports, $215ea retail.)

To connect my file server to the blade switch I'd likely get a Mellanox ConnectX-3 EN MCX312A-XCBTdual port 10GbE card ($351 online) which is by far the cheapest dual port 10GbE card I've found. I'd normally be weary of such a bargain but since it's Mellanox I'll chance it.

If I were to stick with 1U's then depending on the switch I get (SFP or Base-T) I'd either go with that same card or Intel X540-T1 with Cat6 if that worked out better with the savings on cables versus SFP.

I'm all ears if you have any comments or suggestions.

 

[Jeggs101]

I always thought that SFP is better. Can go optical at some point if you want.

 

[mrkrad]

10gbe gbic's are $$$$.00 - no joke.

optical has 4 times less latency than wire , yes, but with large MTU that really isn't a problem and we can all agree 9000 MTU is a smart thing at 10gbe (some even go to 12000) less overhead.

i never understood why they could not do a start-stop bit scenario so MTU could be dynamically adjusted for the application since fragmentation is stupid expensive and leads to delayed ack buffer nightmares.

 

[renderfarmer]

I got my Mellanox 40GbE cards. I have one in my file server + a 1GbE line for FiOS, and the other 40GbE is in my workstation.

I tried Bridging for the 1GbE/40GbE connections but the 40GbE bandwidth went down to 1GbE...

I can't get Windows 2012 Routing service up and running so that I can enable NAT and let the internet connection be shared with the 40GbE connection.

I've found plenty of online tutorials on how to do it in Windows 2008 R2 but none so far for Windows 2012.

Can anyone point me in the right direction?

 

[mrkrad]

Routing and Remote Access in Windows Server 2012 - Part 1

I don't see how its different? Is RRAS/Routing not in your version of 2012 server?

These new 2920 switches (4 of them) i have sounds a lot like the IPoiB where you can use openflow to route traffic. Going to have to read up on openvswitch and openflow it seems like a traffic director which doesn't seem too far off.

A netgear XS7224S (24 ports SFP+ 10GB ethernet) with 4 shared 10GBASE-T showed up for demo. Stupid expensive for netgear ($10K). Going to see if they are up the pepsi challenge never the less.

 

[renderfarmer]

Thanks. I was missing the DirectAccess part...

Crap, now it's saying I have to have a domain running. this never ends.

 

[renderfarmer]

I just spent the entire day trying to set up some basic routing on my file server and I have nothing to show for it.

Can anyone please give me some advice as to what the best approach is to configure my file server for my network to function properly? This is a simple diagram illustrating my network:

What I'm trying to accomplish is:

1. Give the PC access to the FileServer @ 40GbE
2. Give the PC access to the internet @ 1GbE
3. Give the Render Slaves access to the file server @ 10GbE.
4. Give the PC access to the Render Slaves @ 10GbE

What I'd like to know specifically is what services should I run on the FileServer (Active Directory Domain, DNS, DHCP, Routing, Other) to make this work?

The only catch is that for the time being it has to work in Server 2008 R2 or Server 2012. I hope to move to CentOS in the future but that's a really big leap that I'm not ready to make yet.

If you need any additional specs or information just ask and I'll make it available.

Thanks, guys.

 

[cactus]

To me you have the file server doing too much, you should break this out to a SAN network and a GbE internet/management network. If you don't want to set up another GbE network, you need to have the fileserve act as a router, breaking your network into multiple subnets, or do L2 bridging, this should be the least complex of the two. A L2 bridge will have the fileserver acting as a switch with GbE and 40GbE links and a LAGG with the four 10GbE links. This will allow the sonicwall to do DNS, DHCP, and internet routing. I have not done bridging in windows before so you will have to look that up(maybe start here). I would start with getting the LAGG set up to your 10GbE switch(which I am assuming you have) then move to the bridging the GbE link to both the 40GbE and 10GbE. If you add another network, you can do a subnet for the 10GbE and a subnet for the two 40GbE nodes. When accessing the file server, just use the IPs on the corresponding networks.

The latter is what I have set up, I have GbE to everything, two QDR cards in my file server, and one in my test box and one in my desktop. The two QDR p2p links are each on their own subnets.

Hope this helps, looks like a wall of text to me right now.

 

[renderfarmer]

Thanks, Cactus! That is very helpful.

So I should specify that the 10GbE switch is a Supermicro SuperBlade switch which I have yet to purchase. I want to be sure I can get the bridging or routing working before I commit to spending thousands of dollars on this setup. The blades have a built in Chassis Management Module with iKVM and all of that. It connects to the LAN via a 1GbE connection.

I've amended the diagram to include the CMM connection and made everything that I have yet to purchase in dashed lines for clarity.

My immediate goal is to get a proof of concept working by successfully connecting the 40GbE connection and the 1GbE connection on the Server.

Bridging the NICs was the first thing I tried. In windows I opened the Network Connections page, I selected the 1GbE and 40GbE network interfaces>right click>Bridge Connections. That did bridge them however my network speed ground to a screeching halt. This may have everything to do with my IPV4 settings. Can anyone tell me what they should be in this specific instance for both the bridge and the 40GbE NIC on the PC side? I had them both set up as if they were directly connected to the SonicWall via a switch but that may be wrong:

IP: 10.10.10.XXX
Mask: 255.255.255.0
Gateway: 10.10.10.1
DNS: 71.250.0.12

If I can't get bridging to work properly I'll start subdividing my network into segments and try to figure out how to route traffic from one subnet to the other.

Thanks again.