A good 10Gb SFP+ Switch for SOHO

[Balteck]

hello everyone, I have 4 Esxi box (HP Proliant G6) that I've updated with dual 10Gb SFP+ network card using vsan/vmotion/vm traffic with many VLANs and a VMware distributed switch through the hosts.

Now I'm using a Dell X4012, but I wih to build a high availability solution with 2 10Gb switches (every server will have a DAC cable to one switch and one to the other).

I could buy another X4012, that in Europe I could pay around 900 euros new, but it lacks of 1Gb ports and it has only 12 ports.

So, I prefer to buy other 2 switches, I looked around and I found:

1) New Quanta LB6M for about 400 euros each, that I know it is possible to Turbocharge with Brocade firmware.
2) Refurbished Cisco UCS 6120xp fabric interconnect for about 380 euros each, that I know it is possible to convert in a Cisco nexus 5010p with 5.0.3 firmware
3) Refurbished Cisco Nexus 5010p for about 450 euros each (but it is unconfirmed when they will be available)
3) Refurbished Blade RackSwitch G8124 for about 600 euros each.

What is the best deal?

My target is to put 2 switches in HA and to forget them for many years till one of them will have a fault.
It is my first experience about HA: VMWare best practice recommends STP, Portfast, BPDU guard, LACP, Link State Tracking and I don't know if any of these switches have. (VDS Best Practices - Virtual and Physical Switch Parameters (Part 2 of 6) - VMware vSphere Blog)

Thank you in advance

 

[Balteck]

Nobody here using vmware vSan with redundant switch may help me?

I opened to many other solutions...

 

[StammesOpfer]

The power draw on the Quanta LB6M is over 100watts and it only has a couple 1G ports. The others I can't speak for. You already have one Dell X series switch why not run an X1052 that gives you 4x 10G ports and 48x 1G.

I think most people that do VSAN in the home lab are doing it with 3 nodes direct connected to each other or single switch.... Could also consider running the backup links over 1Gbe?

 

[Balteck]

Power draw is not important (Nexus is about 500W). What I'm really looking for is a sort of redundant for this little lab, but i have a low budget.
Quanta LB6M seems interesting because it has redundant power supply, but it is possible to use them also in redundant way, so that a switch fault does not block everything?
Also Quanta LB4M seems interesting for backup links over 1Gbe and total cost of 2 LB6M with 2 LB4M is about 1000 euros.

Another X4012 and 2 X1052 is about 2000 euros and they are only 1 power supply. (and I don't know if they can be used in HA)

UnixPlus offers also Brocade FastIron FLS648 at good price and LB6M can be transformed in a Brocade TurboIron. (and also I don't know if FastIron and TurboIron or VxWorks on LB4M and LB6M can be configured in HA).

I wish to buy something that have these features. Then I study manuals and I will configure them

My configuration is a simple LBT with NIOC, so no LACP/LAG or Etherchannel and I think I need to use STP or MST in this way:

 

[StammesOpfer]

I was suggesting that you may not need the x4012 that you could get away with just 2 x1052 or any other switch that has 4x 10G ports in addition to 24-48 1G ports. Do you really need the cross links between you 10G switches since all devices are directly connected to each one already? You might even be able to do a stack (HDMI stacking option) of 4+ Dell 5524/5548 switches, not ideal but cheap and allows for failure. The LB6M is a good switch if it meets your need and you don't mind power and noise. I would avoid the LB4m since it's feature set is still limited. I think you may be making this more complex than it needs to be though.

Also based on that diagram

 

[Balteck]

Maybe I'm a complex person and I don't know.

Anyway I cannot use 2 X1052 because I have 3 hosts with dual 10Gb NIC and quad 1Gb NIC, but I wish to update NAS at 10Gb and before the end of the year I will retire 2 used rack servers from my customers). In my diagram each line represents a physical cable: grey cat5e rj45 and green SFP+ Twinax DAC

So each switch uses 3 10Gb ports and 6 1gb ports for rack servers, other that 1 port for ILO, 1 port for backup NAS and 2 port for Firewall.
Only two X1052 are perfect now (if I use the last 10Gb port for uplink), but I cannot expand anymore, so it is not a good investment.
My goal is install, configure and forget. Changing firmware or rebooting switches is not possible if I don't have a redundant topology.

The best would be a switch with 8 SFP+ ports and 24 1Gb ports, but I don't know if it exists.

 

[Evan]

Switches with 1G or 10G coppper have usually a maximum 6 x SFP+ ports, new Cisco Catalyst 9300 seem to have 8 though.
Anyway what your looking for is difficult to find.

Closest solution if you really only need 1G copper is a 16,24,48 port SFP+ switch and use GLC-T type SFP for 1G rj45.

 

[Balteck]

I'm not looking for a switch like Catalyst 9300. It was a reply at @StammesOpfer that suggested to put everything on two switches stacked together in HA. But why not 2 pair of switches?

I mean a 12 SFP+ and up switch for 10GB connectivity (vSAN, vMotion and network for high inter-trafic VMs) and a 24 rj45 switch and up for 1gb connectivity (Management Network, VM Network, internet access).

I found a better schema to explain my goal to find the right switch. This is based on Cisco Nexus, but I hope that someone may suggest me a cheaper switch to do it with equivalent settings (VST, portfast, trunk, ecc) that have dual power supply.

 

[fohdeesha]

The switches in that example can be pretty dumb, there's nothing special going on with them. In regards to "VST", all the switch actually sees is normal vlan tagging. You simply need a switch that supports VLANs as well as STP/RSTP with the ability to define "portfast" on your hypervisor-connected ports, which is cisco's magical name for an "edge" port in STP. Any of the major manufacturers will let you define this along with BPDU guard etc (for example on brocade, it's "spanning-tree 802-1w admin-edge-port" on the ports connected to your devices).

You could do this with pretty much any layer2/layer3 switch out there, so just find one with the port count you need. The LB6M's flashed to brocade would certainly work, and if you're feeling adventurous you can use Brocade MRP for redundant links between the switches instead of STP/LACP, it's failover is much faster, on the order of 5-10ms if you crank the timers down. You can even run multiple independent MRP rings between multiple pairs of switches, over LACP trunks as well if you want load sharing across multiple interconnects. It can even be defined per-vlan as well ,if you need that granularity

If you want to keep it to one vendor, then a good matching 1gbe switch pair from brocade would be the FCX - Full l2/l3 featureset, mature platform with firmware as recent as a couple months ago, and hilariously cheap - $100 on ebay with dual power supplies and everything, around $150 for a chassis that includes a 2-port 10GBE module (which you could use as an uplink redundancy ring style to each of your 10gbe switches). They'll even do BGP, VRF's, GRE tunnels etc with a license, which I can probably send you if you need one. Everything else (ospf, ipv4/ipv6 routing etc) comes standard. They only draw 40 watts as well, granted I know you said power doesn't matter. You can think of the FCX as the "high end" version of the FLS you saw from UnixSurplus

I have a similar setup in my l2-only lab at home - LB6M pair running brocade in a redundant MRP ring with xen guests and storage uplinked to both, and then a pair of FCX's (PoE models, I have a lot of PoE devices) in their own ring with critical hosts uplinked to both of them, then finally both pairs ringed together, so all interconnects between any 2 switches is redundant, and I can lose any 1 switch entirely and important end devices will never notice. Works perfectly, if a bit overkill for home. If you don't mind slightly slower failover/convergence just replace "MRP" in all of the above with STP/RSTP, it's all the same stuff, and other vendors will have their own versions of proprietary fast l2 ring crap

 

[Balteck]

Thank you very much!!
This is the right choice: 2 LB6M for 10GB NICs and 2 FCX for 1gb NICs.

As you did, I will put Brocade firmware on LB6M and because I like complex behaviour, I will configure them with MRP.
I found some Brocade FCXs and Dell Brocade FCXs in Ebay. Are they the same?
I prefer the version E or I (not S) because they have better cooling and SFP+ modules instead of XFP, so I could use Twinax cable and not XFP and SFP+ transceivers with fiber cable. For what I read does not exist a XFP to SFP copper Twinax cable.
Where did you find them a around 100/150 USD. Maybe they are S version?

If I understood well, I can stack 2 FCXs in redundant way with LACP trunks on 1gb ports or with 10GB SFP+ module.
And the same on LB6M with TurboIron firmware. Is it right?

May I ask how you physically connect the pair of FCXs with the pair of LB6M?
Using 10GB module on FCXs or a LACP with the 4 1gb ports on LB6M?
Is it similar to my first schema pic (with green and grey cables)?

Thank you again for pointing me to the right direction.

It is a pity that LB4M and LB6M don't have this features with the stock firmware. They seems cheap and reliable.

 

[fohdeesha]

I realize I'm starting to sound like a Brocade salesmen - so just to be clear, ANY l2 or l3 switch that supports vlan's and STP will work in the above config, even an lb4m and lb6m with stock firmware. by no means do you have to go brocade

Don't bother with the Dell branded FCX's, grab a brocade original. I'd recommend the S versions, they're cheaper and more readily available. Both the S and I/E versions have 4x 10gbe ports - on the I/E's it's 4x sfp+ ports, on the S series, it's 2x XFP on the front, and 2x cx4 ports built in on the back. I would just use these cx4 10gbe ports on the rear to interconnect the two fcx's - all you need is a couple cheap cx4 cables. Then you have 2x 10gbe interconnects between your pair of FCX's, then each FCX still has 2x 10gbe XFP ports on the front - in my situation, each FCX has one port uplinked to LB6M #1, and the other port linked to LB6M #2.

As for DAC cables, just use actual fiber (I will never understand why this forum is so obsessed with DAC cables, lol). If you buy from ebay, optics and fiber are just as cheap. XFP and SFP+ 10gbe optics are 7 or 8 dollars, then a short run of om1 or om2 for another 10 bucks. Then you can easily go between XFP and SFP+....with fiber, like they were originally intended. Both chassis (lb6m's and FCX's) will take any vendor optics, regardless of firmware loaded

 

[fractal]

As for DAC cables, just use actual fiber (I will never understand why this forum is so obsessed with DAC cables, lol).

I could very well be wrong but a DAC cable seems a lot more rugged that the cheapo patch fibers I buy. That and they use less power and have lower latency.

 

[fohdeesha]

have a few hundred "cheapo fiber patch cables" in production for a few years now, have never had one fail. have had DAC cables refuse to link any longer because the teeny 30AWG twinax conductors got pinched in the wrong manner or the driver board in active cables burnt out. Not saying my experience is indicative of the technologies as a whole, but I think pretending either one is more reliable than the other is probably not very realistic

Power: sure, passive DAC saves you roughly half a watt per port - that will matter more to different people and different densities. For example, the 30x active SR optics in my house all combined still do not pull as much power as the little LED light bulb that illuminates the closet they're in. if you have 300x interconnects and are tight on power however, then it can add up

As for latency, the difference is negligible. You're talking 5 nanoseconds maybe, 5 millionths of a millisecond. The forwarding latency of modern silicon in switches is twice that on it's own. For the software stack of whatever's actually connected, add 3 zeros to that before they can even begin to measure the difference. Not even HFT financial cabs I've worked with in chicago bothered with such a minute difference, it's all SMF and MMF.

Don't get me wrong, DAC makes sense for home gamers and small setups etc, but a lot of people seem to think it's the ONLY way to interconnect 10gbe devices

 

[Jerry Renwick]

Third-party GLC-T SFP is a great budget for 1gb link. Best Price Cisco GLC-T 1000BASE-T SFP RJ45 Copper Transceiver Module | FS.COM

 

[Balteck]

concerning my self I have a lot of Cisco DAC cables that I'm using on Dell X4012 and they work very good. So if I can not buy other transceivers with fiber I'm happy, becuse I use money only for switches.

I realize I'm starting to sound like a Brocade salesmen - so just to be clear, ANY l2 or l3 switch that supports vlan's and STP will work in the above config, even an lb4m and lb6m with stock firmware. by no means do you have to go brocade

I think that often, what does the difference of equal hardware is the software. If Brocade firmware is always updated (I see last FastIron fw version released at 30-11-2017) and permits to stack other than play with STP,MSTP and MSP, why not sponsor it, if is it better?
I have a low budget, but I prefer to spend 100 euros more or to lose led port notifications if in exchange I will have a better and mature software that allows me to sleep peacefully. That's my 2 cents.

Anyway a Brocade FCX or a Dell Brocade FCX is the same and can I apply the last firmware?
Here in Italy I found some Dell branded FCX and I prefer to buy in Europe, because from USA I have customs and very high shipping costs.

 

[fohdeesha]

concerning my self I have a lot of Cisco DAC cables that I'm using on Dell X4012 and they work very good. So if I can not buy other transceivers with fiber I'm happy, because I use money only for switches

If the XFP and SFP+ versions of the FCX are the same price where you are, then definitely go with the one that lets you use cables you already have. However in my experience the SFP+ versions are nearly twice the cost of the XFP models even though they both have the same amount of 10gbe ports, so if you have to pay $150 extra then the $20 you saved by using DACs you already have is no longer much savings

I just confirmed that the Dell branded FCX does indeed run normal brocade firmware, so no worries there. It seems they slapped the dell logo on a lot of different vendors and re-sold them, including Juniper and Brocade.

Can I ask what kind of prices you're seeing in Italy? In the US, $100 is average for an FCX without 10gbe card, and around $150 for the FCX with 10gbe card.

If you are seeing prices much much higher than that, maybe look at getting one shipped from the US, or look at more readily available models - The Juniper EX4200 for instance, good switch with recent firmware, exact same port count and features as the FCX. They should also go for around $100 USD (however much more with the 10gbe card). If you need latest firmware, just PM me

 

[Balteck]

Here prices are higher. From 240 euros for a fcx624-e with 4 sfp+ module dual psu or from 400 for fcx648-e mono psu without modules.

Fcx s series seems not so common jn Europe (almost now)

I'm looking also for USA offers, but the big problem are the shipping and import costs.
Sellers ask from 150$ to 370$ only for shipping in Italy. So I'm looking for a complete product (dual psu with 10gb) because multiple shipping (fcx from one place, pay from another and module from another place) is very very expensive for the shipping.

Now I seems that I found 2 fcx648-I with 4 sfp+ and dual psu 350 USD on EBay.
If I can use my courier, I could spend about 70 euros each with 3/4 days delivery.

So I can get 2 complete ones with the European price of a base one.

Inviato dal mio HTC One M9 utilizzando Tapatalk

 

[fohdeesha]

Fcx s series seems not so common jn Europe (almost now)

see if juniper ex4200 switches are more available/cheaper. If they're cheaper I would just go with that, same feature set and good switches. Only 2x 10gbe ports max - but they have dedicated stacking connectors on the back, so you could stack them for high bandwidth interconnect from ex to ex, then use the 10gbe uplinks to the lb6m's