Looking for advice on 10/40Gb networking setup

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

SRussell

Active Member
Oct 7, 2019
327
152
43
US
I finished reading the monster Brocade thread which I wish I had found STH a year ago.

I have 8x systems that I want to move to 10/40Gb networking.
1x JBOD 4U Supermicro x10 motherboard, 2x 10GbE-T, 2x 40GbE-QSFP+, FreeNAS or ZoL
1x JBOF 4U Supermicro x10 motherboard, 2x 10GbE-T, 2x 40GbE-QSFP+, FreeNAS or ZoL
6x Supermicro Xeon D-1540, 2x 10GbE-T, running ESXi

1x MacMini, 1x 10GbE-T
1x Synology, 2x 10GbE-T that is being decommissioned

I am not sure what is the 'best' route to setup connectivity. The Brocade thread pretty much deals with SFP+ and QSFP. My needs run a little more than beef king and a lot less than beef overload.
  • If I go with the ICX6610 I have exactly the number of ports I want/need. I would need to buy 6x dual SFP+ cards. But I am minus the Mac mini which I can live without 10Gb on.
  • I could go with the ICX6650 and have more than enough room to grow into. I will still need to buy 6x SFP+ cards. This will put me right around my max budget.
  • The Brocade ICX7750-48C is a good contender but I know nothing about the switch outside of the spec sheet. This gives me 48x 10GbE-T ports, 6x QSFP+, I do not need to buy any additional HBAs, but this unit is a bit outside of my budget and I would just need to save a few more dollars. Brocade ICX7750-48C 48x 10GBASE-T 6x 40GB QSFP+ Port Layer 3 switch SKJ | eBay
  • The Arista DCS-7050T-64-R appears to also be a good choice. It is just as power hungry as the ICX7750-48C. I have 2x less QSFP+ ports which I will not miss and it is inside my budget. Arista DCS-7050T-64-R 48x RJ45 1/10GBASE-T 4x QSFP+ Switch R-F Air 2x AC PSU HSS | eBay
  • Lastly, I could go with an inexpensive NetGear 10GbE-T switch and direct connect the JBOD and JBOF.
If you were in my space what choice would you pursue?
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
Why not combine an EMC converted SX6012 and a 10G switch (Netgear/Microtik)? Ideally one with 2+ SFP+ ports so you can LAG the uplink

The brocades are nice but if they don't fit your need then it does not make sense to get them
 

Mithril

Active Member
Sep 13, 2019
354
106
43
I'm kind of in the same boat. I don't currently have any 40GB, but my ideal switch would have 2+ QSFP for either 40GB later or uplink to a 40GB switch (yes, you could do a LAGG, but that's less ideal IMHO), unfortunately I'm also looking to keep the power usage down as I don't have a "proper" rackmount UPS (every time I see a good deal its far enough away the shipping is a killer :( ) so the ICX6xxx with enough 10 and 40GB are out, the ICX7xxx are largely out as well. I'm leaning to the SX6012, but the painpoint there is going from QSFP to LC for the machines that are (and will be) 10G but not in the rack. Optics are not bad used, but the MPO to LC fiber even at 1m or 2m seems outrageous to me.
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
There are QSFP to SFP splitter cables (DAC) at considerable lengths available - would eg 15m not be enough?
 

Mithril

Active Member
Sep 13, 2019
354
106
43
There are QSFP to SFP splitter cables (DAC) at considerable lengths available - would eg 15m not be enough?
These are machines in other rooms, for one I already have done the fiber run. They all have to go through-wall, and in two cases the "someday, I'd like to" plan is to upgrade to 40GB links (using QSFP BiDi adaptors most likely as long runs of MPO remain $$$ which the BiDi optics prices are dropping over time) to switches in those rooms.
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
Ok, so you need dedicated fibre runs - with which connectors ?
Whats forcing you to use MPO 2 LC?
 

Mithril

Active Member
Sep 13, 2019
354
106
43
Ok, so you need dedicated fibre runs - with which connectors ?
Whats forcing you to use MPO 2 LC?
Ok so I have switch now with SFP+ ports, machines in the same rack with SFP+, and remote machines with SFP+ ports. For machines in the same rack I'm using a mix of DACs and MMF, the remote machines are MMF as well, all the MMF is LC on both ends. I've got a ton of SFP+ MMF modules I purchased for really cheap. Regardless, even singlemode would be LC (single if BiDi, or dual otherwise).

The SX6012 (and its big brothers) are QSFP *only*, so I'd need some way to connect/breakout to the 10GB machines. I could use QSFP->SFP slot adapters, but at $10+ each and consuming a slot each that seems like a poor option for mass use. As far as I've seen the most common fiber connection for 40GB is MPO, with 4 active pairs (10GB each), so in order to split that there are MPO->LC fiber or cassettes.
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
Ok and how many hosts are we speaking of in total (how many qsfp and sfp+ attached) ? How many other rooms are there?

It would be no issue to connect to the ones in the rack with a splitter cable - 6012 can split up all ports, the larger ones are somewhat limited.

Now splitting over existing fibre runs is indeed more complicated, not sure its worth the trouble (depends on the specific circumstances).

And re keeping power to a minimum - thats due to your current UPS not providing enough backup power for the switch? How will that matter if you cant power the clients in the first place?
 

Terry Wallace

PsyOps SysOp
Aug 13, 2018
197
118
43
Central Time Zone

Mithril

Active Member
Sep 13, 2019
354
106
43
Ok and how many hosts are we speaking of in total (how many qsfp and sfp+ attached) ? How many other rooms are there?

It would be no issue to connect to the ones in the rack with a splitter cable - 6012 can split up all ports, the larger ones are somewhat limited.

Now splitting over existing fibre runs is indeed more complicated, not sure its worth the trouble (depends on the specific circumstances).

And re keeping power to a minimum - thats due to your current UPS not providing enough backup power for the switch? How will that matter if you cant power the clients in the first place?
This would end up being the main switch, so while not all clients would be on long running ups (or generator), it would plus firewall, main desktop, etc.

And yes, for whats in the rack I plan on doing DACs, they are around 20-35 for the reasonable ones, and in the past I've seen them at one of the PC recyclers near me for IIRC ~12 per (Shame I didn't pick up any last I saw some :()

Not just the current UPS, but also any future UPS, buying runtime gets expensive. The 6012 looks to run at or under 50w, and I really have not found anything comparable port count wise that has 2+ QSFP for under 100w, while being a "reasonable" price. (tl;dr 10+ SFP+ ports and 2+ QSFP, while doing decent L3, for 500 or less) About 2x a year we have a power outage long enough to need to run a generator, so I'm keeping that in mind too. I get that I'm looking for a "unicorn" switch, but thats my choice to make.
 

Mithril

Active Member
Sep 13, 2019
354
106
43
Buy one of these....
Quanta LB6M 24-Port 10GbE SFP+ & 4x 1GbE L2/L3 Switch W/ Dual AC Power Supply | eBay

convert it to Brocade... See these forums... https://www.servethehome.com/turbocharge-the-quanta-lb6m-with-brocade-turboiron/

and lag a (4) 10's into a 40 linked to your SX6012. Lots of ports.. lots of growth..easy to manage..

also..UPS's off ebay are cheap are you can replace the batteries yourself with some common sense. :)
Power hungry and no QSFP at all, which IMHO defeats the point of having a QSFP switch. At that point I'd just go with one of the SFP+ only options that uses less power than the LB6M, and put the 40GB networking into the "not todays problem" basket.

As far as UPSs, I've had bad luck with finding ones that don't want an arm and a leg for shipping due to distance
 

Mithril

Active Member
Sep 13, 2019
354
106
43
If 3ms would suffice (i doubt) here are some fairly cheap breakout cables MPO Cable - 1 xmpo on 4x LC/PC MM (Breakout) - OM 3 (NEW) - INCL VAT | eBay


still it might be better to get a 6018 and do qsfp->sfp adapters or get a second 6012 and place it in the other room...
Unfortunately shipping is almost as much as the cable.

qsfp->sfp end up costing about the same per 10G connection as going with the MPO->LC route and eat up a port per :(. Having another 6012 in another room means running MPO, or getting QSFP BiDi, and likely needing to go with either QSFP->SFP+ or AOC in order to each everything nicely. Plus dealing with the noise.
I'm really feeling like my best bet is to try to find a good deal on the MPO->LC, the LC-LC couplers are cheap and the QSFP optics can often be had for a reasonable price.
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
its 3 of the cables if you have not seen it? maybe that makes it worthwhile
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,729
3,081
113
33
fohdeesha.com
(tl;dr 10+ SFP+ ports and 2+ QSFP, while doing decent L3, for 500 or less) About 2x a year we have a power outage long enough to need to run a generator, so I'm keeping that in mind too. I get that I'm looking for a "unicorn" switch, but thats my choice to make.
The 24 port icx6610 meets all those requirements, 80w power draw, 2x qsfp, 16x 10gbe, a lot of L3, can be routinely had for $200. If you're hoping to find the same at even lower power draw you will probably be waiting a few years
 

Mithril

Active Member
Sep 13, 2019
354
106
43
The 24 port icx6610 meets all those requirements, 80w power draw, 2x qsfp, 16x 10gbe, a lot of L3, can be routinely had for $200. If you're hoping to find the same at even lower power draw you will probably be waiting a few years
Fair point, it does still run about 2x what the 6012 does idle (assuming you get the 24 port, as the 48 port is 110w?). I noticed the top post on that thread has been updated, and that there is a note that the fan's can't be modified? It would be nice to not have whatever I add be too much of a "screamer" for when I'm working on things in the garage. So a little more power use, maybe louder, but ~200 and I can just drop it in the rack and go now with 8(+8 more with DACs when needed), VS up to 40 10G ports less power use (especially with a popular fan mod), but ~80-100 extra for the same 8 ports now, plus additional extra for more DACs or MPO->LC in the future. OTOH, while nothing critical would be running on my current 1G+10G switch so it would be on the small UPS (just there for small power blips) going with the icx6610 would mean I wouldn't even need that one powered on at all... Hows the licence situation with the 6610?
 

itronin

Well-Known Member
Nov 24, 2018
1,233
793
113
Denver, Colorado
It would be nice to not have whatever I add be too much of a "screamer" for when I'm working on things in the garage.
I'll save you a little bit of reading. Sound is always subjective though. This is my experience:

ICX6610
Using two Rev A PSU, loud.
Using one Rev A PSU, one Rev B PSU, loud.
Using two Rev B PSU, not loud (about the same as a SM 836 with stock fans and default pwm profile).

Its hit or miss what kind of PSU's you'll get. If purchasing off eBay ask the seller. Pulling a PSU is easy.